A large language model (LLM) is the central reasoning engine behind Hermes Agent. It processes in­struc­tions, evaluates in­ter­me­di­ate results, decides on the next action and generates struc­tured tool calls. The quality, context length and tool-calling re­li­ab­il­ity of the selected model directly affect how reliably and autonom­ously Hermes can complete complex tasks.

Key Takeaways

The key points for choosing an LLM for Hermes Agent:

  • Minimum re­quire­ments: at least 64,000 tokens of context and reliable struc­tured tool calling — Hermes rejects models with smaller context windows at startup.
  • Open­Router: one API key for many models and high flex­ib­il­ity, at the cost of an ad­di­tion­al in­ter­me­di­ary layer.
  • Direct provider API (for example Anthropic): a clearer setup with more control over costs and data pro­tec­tion, but less flex­ib­il­ity.
  • Running locally: Ollama as the easiest way to start, vLLM for GPU servers with many parallel requests, LM Studio for testing through a graphical interface.
  • Local models: Qwen3-Coder 30B-A3B for code and terminal tasks, Llama 3.3 70B for language-heavy workflows — Llama 3.3 70B needs con­sid­er­ably more hardware.

What re­quire­ments must an LLM for Hermes Agent meet?

Not every chat model is auto­mat­ic­ally suitable for an autonom­ous agent. While a simple chatbot often handles only a single question-and-answer exchange, Hermes Agent uses LLMs to execute multi-step workflows. The model plans an action, calls a tool, receives a result, evaluates it, and then decides on the next step.

For these workflows, two prop­er­ties are par­tic­u­larly important:

  1. a suf­fi­ciently large context window,
  2. reliable support for struc­tured tool calling.
Hermes Agent VPS hosting
Your VPS for Hermes Agent – powerful, secure, scalable
  • Hermes-ready: root access, SSH-optimised, up and running in minutes
  • Con­sist­ently fast: NVMe SSD and 1 Gbps, no throt­tling
  • Secure: custom API in­teg­ra­tions and Docker support

Size of the context window

The context window shows how much in­form­a­tion a model can process at one time. It is measured in tokens. Tokens are small pieces of text. Depending on the language and content, one token can represent part of a word, a short word or a punc­tu­ation mark.

The context used by Hermes includes more than just your original task and previous responses. System in­struc­tions, de­scrip­tions of available tools, loaded skills, stored in­form­a­tion, in­ter­me­di­ate results and tool outputs also take up space in the context window. This means Hermes Agent needs a model with a context length of at least 64,000 tokens. Models with smaller context windows are rejected at startup because they do not provide enough space for multi-step tool workflows.

When choosing a model for Hermes Agent, you should not aim for the minimum limit of 64,000 tokens. A model with some ad­di­tion­al capacity above this level is re­com­men­ded. A model with 128,000 tokens or more provides even more room. Hermes can auto­mat­ic­ally compress long con­ver­sa­tions, but this does not replace having a suf­fi­ciently large context window from the start. Com­pres­sion helps keep long sessions stable, but it cannot make an un­suit­able 8K or 32K model work for complex workflows.

If the available context window is too small, older in­form­a­tion may be lost during the process or sum­mar­ised too early. The model may then repeat steps it has already completed, assign a tool result in­cor­rectly or lose track of the original goal.

Tool calling

Tool calling allows an LLM to do more than generate a written answer. It can also create a struc­tured function call that tells Hermes which tool to use and what action to perform. For example, the model can decide to read a file, run a terminal command or start a web search.

A sim­pli­fied tool call might look something like this in­tern­ally:

{
  "name": "web_search",
  "arguments": {
    "query": "latest Hermes Agent documentation"
  }
}
json

Hermes re­cog­nises this struc­tured call, runs the requested tool and sends the result back to the model. The model then evaluates the result and decides whether any further actions are needed.

If a model only describes the required action in plain language instead of gen­er­at­ing a struc­tured tool call, the tool is not executed auto­mat­ic­ally. This means native or properly im­ple­men­ted tool calling is sig­ni­fic­antly more reliable than asking a model to generate JSON‑like text on its own. Claude supports defined tools through struc­tured tool_use blocks, while OpenAI-com­pat­ible servers represent function calls using the tools, tool_choice, and tool_calls fields.

A large context window alone is not enough. A model can handle a 128,000-token context window and still not be suitable if it fre­quently generates incorrect function names, forgets required para­met­ers or mixes tool calls with written answers.

Running Hermes Agent in the cloud: Open­Router or a direct provider API?

For cloud de­ploy­ments, there are two main options. You can connect Hermes Agent to different models through a model ag­greg­at­or like Open­Router or connect it directly to a model provider’s API, such as Anthropic’s.

Open­Router: many models through a single API key

Open­Router provides a single API for more than 400 models from different providers. You can filter supported models by context length, price, speed, tool-calling cap­ab­il­ity and other criteria.

The biggest advantage of using Open­Router is flex­ib­il­ity. You only need one Open­Router API key and can switch between models such as Claude, Qwen, Gemini, Llama, or DeepSeek. This allows you to use a powerful model for complex tasks and a more af­ford­able model for simpler back­ground tasks. Open­Router can also route requests through different providers that offer the same model. You can sort options by price, through­put, or latency, choose preferred providers, exclude specific providers, and limit requests to endpoints with zero data retention.

A simple Hermes Agent con­fig­ur­a­tion might look like this:

model:
  provider: openrouter
  default: anthropic/claude-sonnet-5

provider_routing:
  sort: latency
  require_parameters: true
  data_collection: deny
yaml

require_parameters: true is par­tic­u­larly useful for agents. This setting tells Open­Router to use only providers that support all para­met­ers required by the request. These can include tool calling and struc­tured outputs, for example.

You do not store the API key directly in this file. Instead, store it in ~/.hermes/.env:

OPENROUTER_API_KEY=your_OpenRouter_API_Key
bash
Note

Hermes separates secret cre­den­tials from normal con­fig­ur­a­tion settings. API keys belong in the .env file, while the model, provider, endpoint, and routing rules are stored in the central config.yaml.

The downside of Open­Router is the ad­di­tion­al in­ter­me­di­ary layer. Exact latency and behaviour can vary depending on the backend you choose. For sensitive data, you should also keep in mind that both Open­Router and the model provider handling the request may be involved in pro­cessing the data.

Direct provider APIs: con­nect­ing models without an ag­greg­at­or

Instead of con­nect­ing Hermes Agent through a model ag­greg­at­or like Open­Router, you can connect it directly to a model provider’s API. Examples of providers include Anthropic, OpenAI and Google. With this type of direct in­teg­ra­tion, Hermes sends requests straight to the selected provider. You need your own account with that provider, a suitable API key, and, if required, a billing setup.

You also configure this con­nec­tion in the central config.yaml file. The provider name and model name you need to enter depend on the provider you choose. A sim­pli­fied con­fig­ur­a­tion might look like this:

model:
  provider: anthropic
  default: claude-sonnet-5
yaml

The API key should also not be stored directly in the con­fig­ur­a­tion file.

Note

If you use a provider other than Anthropic, update the provider name, model ID, and en­vir­on­ment variable name to match that provider.

The main benefit of a direct provider API is a clearer setup. You know exactly which company processes the model request, what pricing applies, and which data pro­tec­tion re­quire­ments and retention policies you need to consider. Because there is no ad­di­tion­al in­ter­me­di­ary or routing layer, a direct con­nec­tion can also provide more con­sist­ent response times. However, it is not auto­mat­ic­ally faster than Open­Router for every request. Actual latency still depends on the model you choose, the context length, the provider’s current load, and the location of its servers.

A downside of direct in­teg­ra­tion is less flex­ib­il­ity. If you want to switch to a model from another provider, you often need another account, a new API key, and an updated Hermes con­fig­ur­a­tion. You also need to set up automatic switching between providers yourself using fallback rules.

This means direct provider APIs are a good fit for real-world ap­plic­a­tions where you have already chosen a specific model and want precise control over costs, data pro­tec­tion, support and how the system behaves tech­nic­ally. In contrast, Open­Router is usually more practical if you want to compare models, switch between providers fre­quently or manage multiple providers through a single interface.

Running locally with Ollama, vLLM, and LM Studio

When running Hermes Agent locally with an LLM, the model runs on your own hardware or a server you manage yourself. Your prompts and model responses do not need to be sent to an external LLM provider. This can be par­tic­u­larly useful when working with internal documents, source code, customer data, or ap­plic­a­tions that need to run without sending data to external services.

Note

Running an LLM locally does not auto­mat­ic­ally mean that all UK GDPR re­quire­ments are met. You still need to consider factors such as the lawful basis for pro­cessing, access rights, logging, retention periods, security measures, and any ad­di­tion­al cloud services you use. Or­gan­isa­tions must implement ap­pro­pri­ate technical and or­gan­isa­tion­al measures through­out the entire data pro­cessing lifecycle. For example, if Hermes uses an external web search while running with a local LLM, data can still leave the local system and may be processed by third-party services.

Ollama: the easiest way into local models

Ollama is a good choice for beginners. The tool downloads models, runs them locally, and provides an OpenAI-com­pat­ible endpoint. Hermes can access this endpoint at http://localhost:11434/v1.

A suitable model can be loaded as follows:

ollama pull qwen3-coder:30b
bash

Then, start Ollama with the required context window size:

OLLAMA_CONTEXT_LENGTH=65536 ollama serve
bash

In Hermes, enter the local endpoint:

model:
  provider: custom
  default: qwen3-coder:30b
  base_url: http://localhost:11434/v1
  context_length: 65536
yaml

The value in the Hermes con­fig­ur­a­tion tells Hermes how much context it can expect to use. It does not auto­mat­ic­ally increase the context window provided by the Ollama server. You must also set the context length on the server using OLLAMA_CONTEXT_LENGTH or an Ollama model file. This value cannot be increased later through the OpenAI-com­pat­ible chat endpoint. You can check the currently loaded context window with the following command:

ollama ps
bash

Ollama supports tool calling through its OpenAI-com­pat­ible API. However, the selected model itself must support a com­pat­ible tool calling format. Qwen-3, Llama-3.x, Mistral, and Hermes models are examples of model families that support this.

vLLM: powerful self-hosting for GPU servers

vLLM is aimed at pro­fes­sion­al users and people who are com­fort­able with technical setups. The software also provides an OpenAI-com­pat­ible API, but it is designed for high through­put, parallel requests and efficient use of GPU servers.

A Qwen model can be started as follows, for example:

vllm serve Qwen/Qwen3-Coder-Next \
  --port 8000 \
  --tensor-parallel-size 2 \
  --max-model-len 65536 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder
bash

The para­met­ers serve different purposes:

  • --max-model-len 65536 sets a context window above the minimum required by Hermes Agent.
  • --enable-auto-tool-choice allows the model to decide whether it needs to use a tool.
  • --tool-call-parser qwen3_coder converts the model-specific tool calls into struc­tured API calls.

For Llama 3.3, vLLM uses the llama3_json tool parser. You should also specify a suitable JSON chat template. This template defines how Hermes passes available tools, required para­met­ers and pre­vi­ously executed tool calls to the model.

Why long contexts require a lot of VRAM

When running a model locally, not only the model weights need to fit into GPU memory. The server also needs memory for the KV cache. It stores in­form­a­tion from pre­vi­ously processed tokens so the model can reuse them when gen­er­at­ing the next parts of a response. The larger the context window, the more ad­di­tion­al memory is needed.

Quantised models mainly reduce the memory needed for model weights. However, they do not auto­mat­ic­ally solve the memory demands of a large context window. This means on limited hardware it can make more sense to use a smaller model with reliable tool calling and a true 64K context window than a larger model whose context has to be partially offloaded to the CPU.

LM Studio: local models through a graphical interface

LM Studio provides a graphical interface for Windows, macOS and Linux. You can search for, download, configure, and run models through a local API server. The OpenAI-com­pat­ible endpoint runs at http://localhost:1234/v1 by default.

The server can be started from the Developer section of the app or from the command line:

lms server start
lms ls
lms load <QWEN3_CODER_MODEL_ID> --context-length 65536
bash

Hermes has its own LM Studio option in the model assistant. Hermes preserves the context length of an already-loaded LM Studio model. For an unloaded model, LM Studio applies its own context setting unless context_length is ex­pli­citly con­figured in Hermes. You can also control and adjust the context length in the model settings. LM Studio supports tool and function calling through its OpenAI-com­pat­ible endpoints. Models with native tool calling generally provide more reliable results than models that rely on a general work­around format. For initial local tests, LM Studio’s graphical interface makes it es­pe­cially easy to use. For a con­tinu­ously running headless server with many parallel requests, however, vLLM is usually the better choice.

VPS Hosting
VPS at an un­beat­able price – and better than ever
  • NEW: Flexible scaling with VM-cloning, load balancing, new storage options and more
  • Unlimited traffic and >99.99% avail­ab­il­ity
  • 24/7 expert support and your own personal con­sult­ant

Which local models work well with Hermes Agent?

Hermes Agent supports different LLMs. The right choice depends on your specific use case.

Qwen3-Coder

Qwen3-Coder is Qwen’s model family spe­cial­ised in pro­gram­ming and agent-based software tasks. It natively supports a context window of 262,144 tokens. On suitable servers with enough memory, Qwen3-Coder can be extended to up to one million tokens using YaRN. For local Hermes Agent in­stall­a­tions, Qwen3-Coder 30B-A3B-Instruct is a strong option. The model has a total of 30.5 billion para­met­ers, with 3.3 billion activated at a time. This ar­chi­tec­ture combines a re­l­at­ively man­age­able model size with strong cap­ab­il­it­ies for code gen­er­a­tion, tool calling and agent-based workflows.

Llama 3.3 70B

Llama 3.3 70B Instruct is the newer and more powerful al­tern­at­ive to Llama 3.1 70B. The model supports a 128,000-token context window and includes a format for struc­tured tool calls. This means it generally meets Hermes Agent’s re­quire­ments, provided the local model server provides at least 64,000 tokens and uses the ap­pro­pri­ate tool-calling template. Llama 3.3 is par­tic­u­larly suitable for Hermes Agent workflows involving general text pro­cessing, mul­ti­lin­gual tasks, research and tasks that require strong language cap­ab­il­it­ies alongside tool calling. However, with 70 billion para­met­ers, the model requires sig­ni­fic­ant hardware resources.

LLMs for Hermes Agent compared

Provider/model Location Minimum context length Tool-calling stability Cost
Anthropic direct / Claude Sonnet 5 or 4.6 Anthropic Cloud Hermes: 64,000; model: up to 1 million Very high; native struc­tured tools High, usage-based token costs
Open­Router / Claude Sonnet or another tool-enabled model Cloud through ag­greg­at­or At least 64,000; depends on model High to very high; also depends on the provider handling the request Variable from low to high
Ollama or LM Studio / Qwen3-Coder 30B-A3B Own computer or server Hermes: at least 64,000; model natively 262,144 High; optimised for tool calling and agent-based code workflows No API costs; medium to high hardware costs
vLLM or LM Studio / Llama 3.3 70B Own high-per­form­ance server Hermes: at least 64,000; model natively 128K High with a suitable chat template and server-side tool-call handling; for vLLM, use llama3_json. High RAM and VRAM re­quire­ments

The tool calling stability rating is not a guarantee. It assumes that the model, context length, chat template, tool parser and server para­met­ers are con­figured correctly and work together as expected.

Which setup fits your use case?

  • For absolute beginners: Open­Router is the easiest way to get started with a cloud-based setup. A single API key lets you try different models without separate provider setups.

  • For complex cloud-based web auto­ma­tions: A direct Anthropic con­nec­tion with a current Claude Sonnet model is a par­tic­u­larly reliable option. It offers a very large context window and stable tool calling.

  • For cloud systems used for real workloads with high avail­ab­il­ity: Use a direct provider API as the primary provider and configure Open­Router as a fallback.

  • For local cron jobs and recurring scripts: Ollama with a suitable Qwen or Llama model is an easy way to start running models locally. The main ad­vant­ages are local data pro­cessing and no token costs.

  • For code and terminal tasks: Qwen3-Coder 30B-A3B is a strong choice for local setups. The model offers a native 256K context window, reliable tool calling and a re­l­at­ively efficient MoE ar­chi­tec­ture.

  • For a local multi-user server or pro­duc­tion en­vir­on­ment: vLLM is the most powerful self-hosting option in this com­par­is­on. It offers high through­put and good GPU util­isa­tion but requires sig­ni­fic­antly more technical expertise.

  • For testing local models with a graphical interface: LM Studio is a good choice if you want to download and configure models without relying heavily on the command line.

  • For low-resource hardware: Llama 3.2 with 1B or 3B para­met­ers can handle simple tasks. For autonom­ous web research, extensive file editing or long-running learning loops, a larger model is re­com­men­ded.

Con­clu­sion: which LLM is the best choice for Hermes Agent?

The best setup for Hermes Agent depends on what matters most to you, such as ease of use, data pro­tec­tion, cost, speed or maximum re­li­ab­il­ity. Re­gard­less of the provider, the model should offer at least 64,000 tokens of effective context, struc­tured tool calling and enough reasoning cap­ab­il­ity for multi-step decisions.

For a simple way to get started in the cloud, Open­Router is par­tic­u­larly flexible. For demanding auto­ma­tions used in real-world op­er­a­tions, a direct Claude con­nec­tion provides stable tool calling and a very large context window. For local ex­per­i­ments and simple cron jobs, Ollama is the easiest option, while vLLM is best suited for powerful GPU servers and workloads with many parallel requests.

Reviewer

Go to Main Menu