6 Best Practices to Build Token-Efficient Multi-Agent Systems

Building an AI application that works well in a test environment is one thing. Keeping it affordable when hundreds or thousands of users start using it is another. The difference becomes even more important when you move from a single AI assistant to a multi-agent system. A single request may involve several AI agents, multiple model calls, tool usage and several rounds of information being passed from one step to another. Every one of those steps can add to your token usage and, ultimately, your AI bill.

The problem is that this cost is not always obvious when you are building the application. A workflow may look efficient during development because it handles only a few test requests. Once real users start interacting with it at scale, however, the same workflow can process far more information than expected. Repeated instructions, unnecessary conversation history, large tool descriptions and using an expensive model for simple tasks can all add cost without improving the final result.

This is why token efficiency needs to be considered while you design the multi-agent architecture, not after the bills start increasing. The goal is not to make your AI system do less. It is to make sure that every model call, every piece of information and every agent involved in the workflow has a clear purpose.

The idea is similar to good software engineering. You would not build an application without thinking about performance, security or how it will handle more users later. The same thinking should apply to AI costs. If you know where unnecessary token usage can occur, you can design the system to avoid it before it becomes an expensive problem.

In a multi-agent system, this means looking closely at how information moves between agents, how models are selected, how tools are made available and how much of the previous conversation needs to be carried forward.

Here are six practical approaches you can use to build a multi-agent system that remains efficient as it grows.

1. Retrieval over injection

One of the easiest ways for a multi-agent system to waste tokens is to give an agent more information than it actually needs.

Consider an insurance company with thousands of policy documents, claims records, product guides, and internal rules. A customer asks whether a particular policy covers water damage. The system does not need to send the entire document library to the model and ask it to find the answer. Most of that material has nothing to do with the question, but the model still has to process it.

 

 

A better approach is to search the company’s information first and pass only the relevant material to the model. This is commonly known as retrieval-augmented generation (RAG). The application retrieves information from a knowledge base or other data source and uses that information to ground the model’s response. Google Cloud’s guide to RAG explains how this approach can connect an LLM to private, specialised and frequently changing information without putting the entire source collection into the model’s context.

The token savings become more important when several agents are involved. Imagine that a workflow needs to check an insurance policy, assess the customer’s situation and prepare a response. If the first stage retrieves twenty documents and passes all of them to the next agent, and that agent passes the same twenty documents forward again, the system keeps paying to process information that may no longer be relevant.

Instead, each stage should receive the context required for the work it is doing. The initial retrieval step may need access to a large knowledge base. The policy-checking stage may only need two or three relevant sections. By the time the final response is being prepared, the required context may be smaller still.

There is another part of retrieval that is easy to overlook: how the source material is divided before it is searched. If a long policy document is broken into arbitrary pieces, the system may retrieve a paragraph without the definition, exception or condition that gives it meaning. Better chunking can therefore improve the quality of retrieval while also reducing the amount of supporting text that needs to be passed to the model. Google Cloud’s guidance on optimising RAG retrieval also highlights the importance of evaluating retrieval quality rather than assuming that a search system is working simply because it returns results.

When designing retrieval for a multi-agent system, a few practical rules are worth keeping in mind:

• Give each agent the context required for its task. More context is not automatically better context.
• Keep retrieval results focused. Returning a large number of loosely related documents can recreate the same token problem you were trying to solve.
• Retrieve again when the task changes. The information needed for research may be different from what is needed for verification or decision-making.
• Preserve meaningful sections. Documents should be divided in a way that keeps related information together instead of creating fragments that require additional context.

The goal is not to give an agent as little information as possible. It is to give it the right information at the right stage of the task.

2. Prompt caching

There is another source of waste that is easy to overlook because nothing appears to be wrong with the application itself. The same instructions and background information are often sent to an AI model again and again.

 

A customer service agent, for example, may receive the same company policies, response guidelines and product information every time it handles a customer request. In a multi-agent system, several agents may receive some of the same information during different stages of the workflow. There is little reason to treat information that has not changed as completely new every time.

Prompt caching allows an application to reuse information that the model has already processed. Instead of starting from the beginning with the same large set of instructions, the system can reuse the repeated part and process the new information separately.

This can make a significant difference in a high-volume application. For example, OpenAI’s current pricing information shows that cached input tokens can be up to 90% cheaper than regular input tokens on supported models. The exact saving depends on the model and how much of your prompt can actually be reused but the principle is important: if the same instructions are being sent thousands of times, there is a real cost advantage in designing the system to reuse them.

A simple example would be a multi-agent customer support system. You may have one agent responsible for understanding the customer’s problem, another responsible for checking company policy and a third agent responsible for preparing the response. The basic instructions for these agents may remain unchanged for thousands of requests.

You can structure the prompts so that this stable information appears in the part that can be reused, while the information that changes from one customer to another is kept separate. That makes caching much more useful.
There is another reason to think about caching at the architecture level. It is not enough to turn caching on and assume that every request will benefit from it. The way you structure the information matters. If you constantly change the part that you expected to reuse, the system may not be able to make effective use of the cache.

When you design a multi-agent application, consider caching information such as:

• Stable instructions that define how an agent should behave.
• Frequently reused reference material that does not change often.
• Examples that are consistently used to guide the model’s responses.
• Repeated tool information when the same tools are used frequently.

At the same time, information such as the customer’s latest message, current order details or live account information will normally change from request to request and should be treated differently.

Prompt caching is particularly useful when your system receives a large number of similar requests. The more often the same information is processed, the more opportunity you have to avoid doing the same work repeatedly.

3. Model cascading and routing

A multi-agent system also does not need to use the same model for every part of a workflow. Different agents may have very different responsibilities, and the model that is appropriate for one task may be unnecessary for another.
A request classification step, for example, may not require the same level of reasoning as an agent reviewing a complicated contract. If both are sent to the most capable model available, the simpler task is likely costing more than it needs to.

 

Model routing addresses this by choosing a model according to the requirements of the request. A smaller model can handle straightforward work, while a more capable model can be brought in when the task involves deeper reasoning, multiple sources or a higher level of accuracy. The same idea can be applied within a single multi-agent workflow. A customer service system might use a smaller model for categorising incoming requests and another model for cases involving policy exceptions or complicated complaints. The decision does not have to be based only on the type of agent. It can also consider the difficulty of the individual request.

This is where routing and cascading differ slightly. With routing, the system chooses a model before the request is processed. With cascading, the system can start with a less expensive model and escalate to a stronger one when the first response does not meet a defined quality threshold. IBM Research describes these approaches as ways of balancing model quality, cost and latency rather than relying on one model for every query.

The potential savings can be significant, but cost should not be the only thing being measured. A cheaper model that regularly produces poor answers may lead to retries, escalations or additional human review. In that situation, the lower model price does not necessarily translate into a lower cost for the complete workflow.

When setting up routing or cascading, it is useful to consider:

• Task difficulty: Does the request require simple classification or deeper reasoning?
• Accuracy requirements: How much does an incorrect answer matter for this particular task?
• Latency: Can the workflow afford an additional model call if escalation is required?
• Cost: Is the additional capability of the larger model actually valuable for this task?
• Quality thresholds: What needs to be true for the system to accept a response from a smaller model?

Routing should also be reviewed as the application develops. A model that made sense during the first version of a system may not remain the best option after prompts, retrieval, tools and evaluation methods have improved. Recent work from IBM Research continues to look at routing as a cost-and-quality problem, including research presented at NeurIPS 2025 on selecting models according to accuracy and cost.

The important point is that model choice does not have to be a decision made once for the entire application. It can be made according to the work being performed, allowing expensive models to be reserved for the situations where their additional capability actually matters.

4. Deferred tool loading

A multi-agent system becomes more useful as you connect it to more business tools. An agent might be able to search documents, check an order, update a CRM record, send an email, schedule a meeting, check inventory and create a support ticket. The problem starts when the AI is given information about every available tool every time it handles a request.

 

If a customer simply asks about a return policy, the agent does not need detailed information about the company’s delivery system, CRM update function or meeting scheduler. Sending those tool descriptions anyway gives the model more information to process without giving it anything useful for the current task.

Deferred tool loading takes a simpler approach. Instead of presenting every tool at the beginning, you make tools available when they are actually needed.

This becomes increasingly important as an agent’s capabilities grow. A small assistant with three tools may not have much overhead. An enterprise agent connected to dozens or hundreds of tools has a very different problem.

Anthropic’s engineering work on advanced tool use provides a useful real-world example. Its Tool Search approach allows Claude to find relevant tools when they are needed instead of loading the complete set of tool definitions into the model’s context from the beginning. Anthropic reported an 85% reduction in token usage in its internal testing for large tool libraries.

The idea is easier to understand with an online retailer. Imagine one agent can check an order, process a return, search products and contact a delivery partner. A customer asks about the return policy. The agent only needs information related to returns. If the customer later asks where an order is, the system can make the order-tracking capability available at that point.

This approach also provides a useful way to organise responsibilities among agents. A returns agent does not need every customer-service tool. A delivery agent does not need access to the finance system. Limiting what each agent can see can reduce unnecessary information and make the system easier to manage.

A sensible tool strategy can include:

• Grouping related tools so the system can find the right category without scanning everything.
• Giving specialised agents limited access to the tools they are actually expected to use.
• Keeping rarely used tools out of the initial context and making them available when required.
• Reviewing the tool library regularly so old integrations do not remain available simply because they were once useful.

This is especially important when you are building AI agent solutions that need to connect with several existing business systems. The goal is not to limit what your agents can eventually do. It is to ensure that agents are not carrying information about capabilities they do not need for the task at hand.

5. Memory compression

Memory is one of the most useful parts of a multi-agent system but it can also become one of its biggest sources of unnecessary token usage.

 

The problem is easy to understand. As a task continues, more information gets added to the conversation. If the system keeps sending the entire history to the model, the amount of information being processed grows every time another step is added.

In a multi-agent workflow, the problem can become larger because the history is not limited to the conversation between a user and one AI. It can include messages between agents, tool results, previous decisions and intermediate information collected during the task. You do not always need all of that information at every stage.

Consider an AI assistant helping a customer plan a two-week international trip. During the first few conversations, the customer explains their budget, travel dates, preferred hotel type and the cities they want to visit. Later, they ask whether they should book tomorrow’s train in advance.

The system does not need to send every conversation from the beginning to answer that question. It needs the important facts that still affect the decision. A useful memory design can therefore keep a summary of important information while storing the original conversation separately. The system can bring older details back when they become relevant instead of sending them with every request.

For a multi-agent system, you can think about memory at three levels.

1. Working memory contains information needed for the current step. If an agent is checking a customer’s order, it needs the order details and the relevant policy, not the complete history of every previous action.
2. Long-term memory contains information that may be useful later, such as a customer’s preferences or decisions made during an earlier task.
3. Task history keeps a record of what happened so that the system can trace the workflow when necessary without forcing every agent to read the entire history during normal operation.

You can also use techniques such as summarising older conversations, keeping only the most recent exchanges in full and storing important facts separately. The right approach depends on the application. A financial workflow may need more detailed records than a simple shopping assistant. LangChain’s guidance on memory for agents also distinguishes between short-term memory used during an active task and longer-term information that can be stored and retrieved when needed.

Recent research shows that this is more than a theoretical cost-saving idea. A 2026 Microsoft Research study on long-running AI agents found that context compression reduced peak memory usage by 26% to 54% across several tasks while largely maintaining performance. The researchers also reported that a smaller model used for the compression step could preserve more than 95% of the original accuracy.

The important distinction is between remembering information and sending information to the model. Your system can retain a complete history without forcing every agent to process that history on every step.
That distinction becomes especially valuable when your agents perform long-running tasks. As workflows become more complex, keeping useful memory separate from the active conversation can help control both cost and response time.

6. Constrained generation

The final approach is about controlling how much an AI agent produces. AI models are very good at generating detailed responses but detailed responses are not always useful. If an agent needs to extract a customer’s name, order number and delivery date, there is no reason for it to write a paragraph explaining what it found. You can ask the model to return only those fields in a format that the rest of your application can understand. This is often called structured or constrained generation. Instead of leaving the format completely open, you define what information should be returned and how it should be organised.

 

Consider an insurance claims workflow. One agent reads the customer’s claim, another checks the policy and a third updates the claims system. The first agent does not need to send a long written explanation to the next agent. It could return a small set of clearly defined fields such as claim type, policy number, incident date and estimated amount.

That makes the handoff easier and can reduce unnecessary output. The same principle applies when an agent needs to call another system. If the next step requires an order number and customer ID, ask for those values rather than asking the AI to describe the customer and leaving another system to extract the information. There is also a reliability benefit. Free-form AI responses can contain extra wording, inconsistent formats or missing information. Structured responses give the next part of the workflow a clearer result to work with.

Amazon Bedrock introduced structured outputs in February 2026, allowing developers to define JSON schemas that models must follow. AWS explains that this can reduce failed requests and retries in production workflows where applications need predictable machine-readable responses.

For a multi-agent system, that can make a meaningful difference because one agent’s output often becomes another agent’s input. If the first agent produces unnecessary paragraphs, the second agent has to process them. If the first agent returns only the information the second agent needs, the transfer becomes much cleaner.

When using structured responses, you can:

• Define only the fields the next step actually needs.
• Avoid asking the model for explanations when the application only needs data.
• Use consistent formats across agents.
• Validate important outputs before another agent or business system acts on them.

This does not mean every AI response should be reduced to a rigid format. Customer-facing conversations may still need natural language. The point is to use a structured format where one part of your system is passing information to another part of your system.

Building a Multi-Agent System That Stays Efficient as It Grows

Token efficiency should not be treated as something to fix after an AI application starts becoming expensive. By that point, the decisions that created the waste may already be built into the way agents communicate, retrieve information, use tools and manage their context.

The six approaches discussed here work together.

• Retrieval helps you avoid sending unnecessary information.
• Prompt caching prevents repeated information from being processed from scratch.
• Model routing helps you match the model to the task.
• Deferred tool loading keeps unused capabilities out of the active context.
• Memory compression prevents long-running workflows from carrying their entire history.
• Structured output keeps agent-to-agent communication focused on the information that actually needs to move forward.

They do not all need to be implemented at once. A customer-support application with a large knowledge base may benefit most from retrieval and caching, while a long-running research workflow may get more value from routing, memory management and deferred tool loading. The right combination depends on how your agents work, what information they handle and where the system is spending most of its tokens.

The important thing is to make those decisions while the architecture is still being shaped. Once unnecessary context, excessive model calls or poorly controlled agent loops become part of a production workflow, fixing them can require changes across the application rather than a simple prompt adjustment.

This is where architecture and engineering need to work together. Building an efficient multi-agent system is not only about selecting a capable model. It involves designing how agents communicate, controlling their access to tools and data, managing memory, choosing models according to the task, and putting the right monitoring in place as the system grows.

If your team is building or scaling a multi-agent application, Unthinkable’s AI agent development services can help with the engineering behind those decisions, from agent architecture and integrations to deployment, testing and optimisation.

The goal is not simply to make an AI system use fewer tokens. It is to build an architecture where every model call, piece of context and agent action has a clear reason for being there. That is what makes a multi-agent system easier to scale without allowing its cost and complexity to grow at the same rate.