What a 40+ tool agentic loop taught us about production AI

Most AI features in production are a prompt and a response. The interesting problems start when the model is allowed to act — call tools, write to memory, speak, and come back later.

On a recent platform we gave the model a loop of more than 40 tools. The lessons had almost nothing to do with prompting, and everything to do with architecture: an agent loop is a system you operate, not a feature you switch on.

What the loop actually decides

A few choices set the cost, the speed and the shelf life of the whole thing.

• Prompt caching and extended thinking set your latency and your bill — measure them before you tune wording
• The reflection pipeline decides whether long-term memory improves with use or quietly rots
• The loop needs a hard cap on turns — without one, a single bad judgement becomes an infinite loop and a growing bill
• Without an eval set you are tuning blind: fix a suite of real conversations, score every prompt or tool change against it, and treat a regression like a failing test
• Tool errors are the error budget: a model that cannot tell a failed call from an empty result will confidently lie

Latency comes from the whole turn, not the prompt

A turn is not one request. It is prompt assembly, one or more model calls, every tool round trip, and the synthesis at the end — and the tail is what users feel.

Measure p95 rather than p50, because the slow turn is the one people remember. Prompt caching pays off precisely where the context is large and stable; extended thinking buys accuracy and spends milliseconds. Tools want timeouts, retries with jitter, and a circuit breaker, or one slow dependency becomes every user waiting.

Memory is a data-modeling problem

We built memory as a knowledge graph with semantic vector recall, not a pile of embeddings.

The difference shows up weeks into a relationship: a graph can answer how two facts connect, a similarity search can only find things that sound alike. Keeping it current is a scheduled job, not a prompt.

Memory in tiers, not one bucket

A single flat context is the expensive way to forget things. We split memory the way a machine does — a fast working set, an episodic layer, and a cheap archive underneath — and moved entries between them on rules rather than on vibes.

• Separate stores for the session, the user profile and long-term knowledge, every entry carrying a timestamp, a source and a priority
• Retrieval by meaning: embed the query, search by similarity, filter on metadata — never hand back everything ever written
• Only the top few entries reach the prompt; the rest stay out, because a context stuffed with noise makes the answer worse and the turn slower
• Entries age: what goes unused is compressed or dropped, or the store grows until it is both expensive and useless

The security surface nobody budgets for

Once a model can call tools, every tool is an API you have shipped to an over-eager caller.

• The tool suite runs as its own authenticated server, on a standard protocol
• Prompt injection is a real attack, not a thought experiment: anything the model reads — a document, a page, a tool response — is data, never instruction
• Every tool gets the least privilege it can do its job with, and its own limits: how many calls, how much data, which resources
• Every call is logged — you want the audit trail the first time the model surprises you

If your agentic roadmap has no security line item, it is not a production roadmap yet.