Which LLM should you use for Hermes Agent?
A large language model (LLM) is the central reasoning engine behind Hermes Agent. It processes instructions, evaluates intermediate results, decides on the next action and generates structured tool calls. The quality, context length and tool-calling reliability of the selected model directly affect how reliably and autonomously Hermes can complete complex tasks.
Key Takeaways
The key points for choosing an LLM for Hermes Agent:
- Minimum requirements: at least 64,000 tokens of context and reliable structured tool calling — Hermes rejects models with smaller context windows at startup.
- OpenRouter: one API key for many models and high flexibility, at the cost of an additional intermediary layer.
- Direct provider API (for example Anthropic): a clearer setup with more control over costs and data protection, but less flexibility.
- Running locally: Ollama as the easiest way to start, vLLM for GPU servers with many parallel requests, LM Studio for testing through a graphical interface.
- Local models: Qwen3-Coder 30B-A3B for code and terminal tasks, Llama 3.3 70B for language-heavy workflows — Llama 3.3 70B needs considerably more hardware.
What requirements must an LLM for Hermes Agent meet?
Not every chat model is automatically suitable for an autonomous agent. While a simple chatbot often handles only a single question-and-answer exchange, Hermes Agent uses LLMs to execute multi-step workflows. The model plans an action, calls a tool, receives a result, evaluates it, and then decides on the next step.
For these workflows, two properties are particularly important:
- a sufficiently large context window,
- reliable support for structured tool calling.
- Hermes-ready: root access, SSH-optimised, up and running in minutes
- Consistently fast: NVMe SSD and 1 Gbps, no throttling
- Secure: custom API integrations and Docker support
Size of the context window
The context window shows how much information a model can process at one time. It is measured in tokens. Tokens are small pieces of text. Depending on the language and content, one token can represent part of a word, a short word or a punctuation mark.
The context used by Hermes includes more than just your original task and previous responses. System instructions, descriptions of available tools, loaded skills, stored information, intermediate results and tool outputs also take up space in the context window. This means Hermes Agent needs a model with a context length of at least 64,000 tokens. Models with smaller context windows are rejected at startup because they do not provide enough space for multi-step tool workflows.
When choosing a model for Hermes Agent, you should not aim for the minimum limit of 64,000 tokens. A model with some additional capacity above this level is recommended. A model with 128,000 tokens or more provides even more room. Hermes can automatically compress long conversations, but this does not replace having a sufficiently large context window from the start. Compression helps keep long sessions stable, but it cannot make an unsuitable 8K or 32K model work for complex workflows.
If the available context window is too small, older information may be lost during the process or summarised too early. The model may then repeat steps it has already completed, assign a tool result incorrectly or lose track of the original goal.
Tool calling
Tool calling allows an LLM to do more than generate a written answer. It can also create a structured function call that tells Hermes which tool to use and what action to perform. For example, the model can decide to read a file, run a terminal command or start a web search.
A simplified tool call might look something like this internally:
{
"name": "web_search",
"arguments": {
"query": "latest Hermes Agent documentation"
}
}jsonHermes recognises this structured call, runs the requested tool and sends the result back to the model. The model then evaluates the result and decides whether any further actions are needed.
If a model only describes the required action in plain language instead of generating a structured tool call, the tool is not executed automatically. This means native or properly implemented tool calling is significantly more reliable than asking a model to generate JSON‑like text on its own. Claude supports defined tools through structured tool_use blocks, while OpenAI-compatible servers represent function calls using the tools, tool_choice, and tool_calls fields.
A large context window alone is not enough. A model can handle a 128,000-token context window and still not be suitable if it frequently generates incorrect function names, forgets required parameters or mixes tool calls with written answers.
Running Hermes Agent in the cloud: OpenRouter or a direct provider API?
For cloud deployments, there are two main options. You can connect Hermes Agent to different models through a model aggregator like OpenRouter or connect it directly to a model provider’s API, such as Anthropic’s.
OpenRouter: many models through a single API key
OpenRouter provides a single API for more than 400 models from different providers. You can filter supported models by context length, price, speed, tool-calling capability and other criteria.
The biggest advantage of using OpenRouter is flexibility. You only need one OpenRouter API key and can switch between models such as Claude, Qwen, Gemini, Llama, or DeepSeek. This allows you to use a powerful model for complex tasks and a more affordable model for simpler background tasks. OpenRouter can also route requests through different providers that offer the same model. You can sort options by price, throughput, or latency, choose preferred providers, exclude specific providers, and limit requests to endpoints with zero data retention.
A simple Hermes Agent configuration might look like this:
model:
provider: openrouter
default: anthropic/claude-sonnet-5
provider_routing:
sort: latency
require_parameters: true
data_collection: denyyamlrequire_parameters: true is particularly useful for agents. This setting tells OpenRouter to use only providers that support all parameters required by the request. These can include tool calling and structured outputs, for example.
You do not store the API key directly in this file. Instead, store it in ~/.hermes/.env:
OPENROUTER_API_KEY=your_OpenRouter_API_KeybashHermes separates secret credentials from normal configuration settings. API keys belong in the .env file, while the model, provider, endpoint, and routing rules are stored in the central config.yaml.
The downside of OpenRouter is the additional intermediary layer. Exact latency and behaviour can vary depending on the backend you choose. For sensitive data, you should also keep in mind that both OpenRouter and the model provider handling the request may be involved in processing the data.
Direct provider APIs: connecting models without an aggregator
Instead of connecting Hermes Agent through a model aggregator like OpenRouter, you can connect it directly to a model provider’s API. Examples of providers include Anthropic, OpenAI and Google. With this type of direct integration, Hermes sends requests straight to the selected provider. You need your own account with that provider, a suitable API key, and, if required, a billing setup.
You also configure this connection in the central config.yaml file. The provider name and model name you need to enter depend on the provider you choose. A simplified configuration might look like this:
model:
provider: anthropic
default: claude-sonnet-5yamlThe API key should also not be stored directly in the configuration file.
If you use a provider other than Anthropic, update the provider name, model ID, and environment variable name to match that provider.
The main benefit of a direct provider API is a clearer setup. You know exactly which company processes the model request, what pricing applies, and which data protection requirements and retention policies you need to consider. Because there is no additional intermediary or routing layer, a direct connection can also provide more consistent response times. However, it is not automatically faster than OpenRouter for every request. Actual latency still depends on the model you choose, the context length, the provider’s current load, and the location of its servers.
A downside of direct integration is less flexibility. If you want to switch to a model from another provider, you often need another account, a new API key, and an updated Hermes configuration. You also need to set up automatic switching between providers yourself using fallback rules.
This means direct provider APIs are a good fit for real-world applications where you have already chosen a specific model and want precise control over costs, data protection, support and how the system behaves technically. In contrast, OpenRouter is usually more practical if you want to compare models, switch between providers frequently or manage multiple providers through a single interface.
Running locally with Ollama, vLLM, and LM Studio
When running Hermes Agent locally with an LLM, the model runs on your own hardware or a server you manage yourself. Your prompts and model responses do not need to be sent to an external LLM provider. This can be particularly useful when working with internal documents, source code, customer data, or applications that need to run without sending data to external services.
Running an LLM locally does not automatically mean that all UK GDPR requirements are met. You still need to consider factors such as the lawful basis for processing, access rights, logging, retention periods, security measures, and any additional cloud services you use. Organisations must implement appropriate technical and organisational measures throughout the entire data processing lifecycle. For example, if Hermes uses an external web search while running with a local LLM, data can still leave the local system and may be processed by third-party services.
Ollama: the easiest way into local models
Ollama is a good choice for beginners. The tool downloads models, runs them locally, and provides an OpenAI-compatible endpoint. Hermes can access this endpoint at http://localhost:11434/v1.
A suitable model can be loaded as follows:
ollama pull qwen3-coder:30bbashThen, start Ollama with the required context window size:
OLLAMA_CONTEXT_LENGTH=65536 ollama servebashIn Hermes, enter the local endpoint:
model:
provider: custom
default: qwen3-coder:30b
base_url: http://localhost:11434/v1
context_length: 65536yamlThe value in the Hermes configuration tells Hermes how much context it can expect to use. It does not automatically increase the context window provided by the Ollama server. You must also set the context length on the server using OLLAMA_CONTEXT_LENGTH or an Ollama model file. This value cannot be increased later through the OpenAI-compatible chat endpoint. You can check the currently loaded context window with the following command:
ollama psbashOllama supports tool calling through its OpenAI-compatible API. However, the selected model itself must support a compatible tool calling format. Qwen-3, Llama-3.x, Mistral, and Hermes models are examples of model families that support this.
vLLM: powerful self-hosting for GPU servers
vLLM is aimed at professional users and people who are comfortable with technical setups. The software also provides an OpenAI-compatible API, but it is designed for high throughput, parallel requests and efficient use of GPU servers.
A Qwen model can be started as follows, for example:
vllm serve Qwen/Qwen3-Coder-Next \
--port 8000 \
--tensor-parallel-size 2 \
--max-model-len 65536 \
--enable-auto-tool-choice \
--tool-call-parser qwen3_coderbashThe parameters serve different purposes:
--max-model-len 65536sets a context window above the minimum required by Hermes Agent.--enable-auto-tool-choiceallows the model to decide whether it needs to use a tool.--tool-call-parser qwen3_coderconverts the model-specific tool calls into structured API calls.
For Llama 3.3, vLLM uses the llama3_json tool parser. You should also specify a suitable JSON chat template. This template defines how Hermes passes available tools, required parameters and previously executed tool calls to the model.
Why long contexts require a lot of VRAM
When running a model locally, not only the model weights need to fit into GPU memory. The server also needs memory for the KV cache. It stores information from previously processed tokens so the model can reuse them when generating the next parts of a response. The larger the context window, the more additional memory is needed.
Quantised models mainly reduce the memory needed for model weights. However, they do not automatically solve the memory demands of a large context window. This means on limited hardware it can make more sense to use a smaller model with reliable tool calling and a true 64K context window than a larger model whose context has to be partially offloaded to the CPU.
LM Studio: local models through a graphical interface
LM Studio provides a graphical interface for Windows, macOS and Linux. You can search for, download, configure, and run models through a local API server. The OpenAI-compatible endpoint runs at http://localhost:1234/v1 by default.
The server can be started from the Developer section of the app or from the command line:
lms server start
lms ls
lms load <QWEN3_CODER_MODEL_ID> --context-length 65536bashHermes has its own LM Studio option in the model assistant. Hermes preserves the context length of an already-loaded LM Studio model. For an unloaded model, LM Studio applies its own context setting unless context_length is explicitly configured in Hermes. You can also control and adjust the context length in the model settings. LM Studio supports tool and function calling through its OpenAI-compatible endpoints. Models with native tool calling generally provide more reliable results than models that rely on a general workaround format. For initial local tests, LM Studio’s graphical interface makes it especially easy to use. For a continuously running headless server with many parallel requests, however, vLLM is usually the better choice.
- NEW: Flexible scaling with VM-cloning, load balancing, new storage options and more
- Unlimited traffic and >99.99% availability
- 24/7 expert support and your own personal consultant
Which local models work well with Hermes Agent?
Hermes Agent supports different LLMs. The right choice depends on your specific use case.
Qwen3-Coder
Qwen3-Coder is Qwen’s model family specialised in programming and agent-based software tasks. It natively supports a context window of 262,144 tokens. On suitable servers with enough memory, Qwen3-Coder can be extended to up to one million tokens using YaRN. For local Hermes Agent installations, Qwen3-Coder 30B-A3B-Instruct is a strong option. The model has a total of 30.5 billion parameters, with 3.3 billion activated at a time. This architecture combines a relatively manageable model size with strong capabilities for code generation, tool calling and agent-based workflows.
Llama 3.3 70B
Llama 3.3 70B Instruct is the newer and more powerful alternative to Llama 3.1 70B. The model supports a 128,000-token context window and includes a format for structured tool calls. This means it generally meets Hermes Agent’s requirements, provided the local model server provides at least 64,000 tokens and uses the appropriate tool-calling template. Llama 3.3 is particularly suitable for Hermes Agent workflows involving general text processing, multilingual tasks, research and tasks that require strong language capabilities alongside tool calling. However, with 70 billion parameters, the model requires significant hardware resources.
LLMs for Hermes Agent compared
| Provider/model | Location | Minimum context length | Tool-calling stability | Cost |
|---|---|---|---|---|
| Anthropic direct / Claude Sonnet 5 or 4.6 | Anthropic Cloud | Hermes: 64,000; model: up to 1 million | Very high; native structured tools | High, usage-based token costs |
| OpenRouter / Claude Sonnet or another tool-enabled model | Cloud through aggregator | At least 64,000; depends on model | High to very high; also depends on the provider handling the request | Variable from low to high |
| Ollama or LM Studio / Qwen3-Coder 30B-A3B | Own computer or server | Hermes: at least 64,000; model natively 262,144 | High; optimised for tool calling and agent-based code workflows | No API costs; medium to high hardware costs |
| vLLM or LM Studio / Llama 3.3 70B | Own high-performance server | Hermes: at least 64,000; model natively 128K | High with a suitable chat template and server-side tool-call handling; for vLLM, use llama3_json. |
High RAM and VRAM requirements |
The tool calling stability rating is not a guarantee. It assumes that the model, context length, chat template, tool parser and server parameters are configured correctly and work together as expected.
Which setup fits your use case?
-
For absolute beginners: OpenRouter is the easiest way to get started with a cloud-based setup. A single API key lets you try different models without separate provider setups.
-
For complex cloud-based web automations: A direct Anthropic connection with a current Claude Sonnet model is a particularly reliable option. It offers a very large context window and stable tool calling.
-
For cloud systems used for real workloads with high availability: Use a direct provider API as the primary provider and configure OpenRouter as a fallback.
-
For local cron jobs and recurring scripts: Ollama with a suitable Qwen or Llama model is an easy way to start running models locally. The main advantages are local data processing and no token costs.
-
For code and terminal tasks: Qwen3-Coder 30B-A3B is a strong choice for local setups. The model offers a native 256K context window, reliable tool calling and a relatively efficient MoE architecture.
-
For a local multi-user server or production environment: vLLM is the most powerful self-hosting option in this comparison. It offers high throughput and good GPU utilisation but requires significantly more technical expertise.
-
For testing local models with a graphical interface: LM Studio is a good choice if you want to download and configure models without relying heavily on the command line.
-
For low-resource hardware: Llama 3.2 with 1B or 3B parameters can handle simple tasks. For autonomous web research, extensive file editing or long-running learning loops, a larger model is recommended.
Conclusion: which LLM is the best choice for Hermes Agent?
The best setup for Hermes Agent depends on what matters most to you, such as ease of use, data protection, cost, speed or maximum reliability. Regardless of the provider, the model should offer at least 64,000 tokens of effective context, structured tool calling and enough reasoning capability for multi-step decisions.
For a simple way to get started in the cloud, OpenRouter is particularly flexible. For demanding automations used in real-world operations, a direct Claude connection provides stable tool calling and a very large context window. For local experiments and simple cron jobs, Ollama is the easiest option, while vLLM is best suited for powerful GPU servers and workloads with many parallel requests.


