A user typed the four characters "test" into a fresh conversation, and that turn cost us $0.61.
Not because of the answer — the answer was a sentence. The cost was everything that travelled with those four characters: the standing machinery a capable assistant carries into every turn — the shared platform instructions, the agent's identity, and above all the schemas of every tool it might use, two hundred and three of them, about a quarter of a million characters of JSON — written into the prompt cache at cache-write rates because a fresh conversation starts cold. Measured properly, our cold prompt floor stood at 122,000 to 163,000 tokens depending on the depth level, of which the tool block alone was fifty to sixty-five thousand.
This is the tax we had not priced in when we sketched an agent product: everything that renders before the user's first word is paid for on every turn, for the life of the product. Warm turns pay it at the cache-read discount; every new conversation, level switch and cache expiry pays it in full. A capability you add "for free" — one more tool, one more instruction block — is a standing charge against every user's every message. So this phase's work was to start treating the prompt as what it economically is: a budget, with line items, measurements, and an owner.
Measure with the meter that bills you
Rule one, before any optimisation: no estimates. Token counts are tokenizer-specific and intuition about them is reliably wrong, so every figure in this phase comes from the model API's own token-counting endpoint, run against the real built prompt and the real tool definitions — the same arithmetic that prices the request. It costs nothing to run and it converts an argument about what "feels heavy" into a sorted table of what is. The first table's headline was blunt: the single largest line item was not prose at all but the tool schemas — the API contracts themselves, which no amount of description-tightening touches (we measured trimming descriptions: under 8% of the block; the weight is the schemas).
Two hundred tools, most of them asleep
The assistant's breadth is the product — a conversation can turn to dashboards, documents, media generation, any of a long tail of connectors — and breadth was exactly what the budget was buying at full price on every "hello". The fix is the model API's deferred-loading mechanism, which decomposes "the model can use this tool" into two properties we had been conflating: registered and rendered. A deferred tool is registered — fully callable — but not rendered into the context; it costs approximately nothing until needed. A small resident search tool lets the model discover deferred schemas by name pattern, server-side, mid-turn; and the discovered schemas are appended after the cached prefix, so discovery never invalidates the cache it just benefited from.
The policy question — which tools sleep? — matters more than the mechanism, and ours is three explicit buckets with a deliberate bias. Resident: every interactive panel (a consent card must never depend on a search), the daily drivers (files, mail, calendar, memory, web), and the gateway tools that skills route through. Deferred: the long tail — the creative studio's dozens of tools, dashboard and dataset authoring, document tooling, the low-frequency connectors — all of them families that a governing skill or prompt names literally, so a regex search finds them the moment a flow needs them. And the bucket that keeps the system honest: anything unclassified stays resident and fails a CI test, so a new tool cannot silently join the deferred tail (a behaviour risk) or silently bloat the floor (a cost regression) — someone has to file it, on purpose, in a reviewed change.
Two scope decisions bound the risk. Deferral applies to interactive chat only — an unattended run gets the full rendered set, because a scheduled dashboard refresh must never fail on a discovery miss with nobody watching. And a single environment flag reverts everything to fully-rendered, because a cost optimisation that cannot be turned off in one move during an incident is itself a risk.
Measured effect, same real-token methodology: the cold prompt fell just over 40% at every depth level — 122k to 72.5k tokens at the fast level, 163k to 97.4k at the deeper ones — with 134 of the 203 floor tools deferred, and warm-cache reads scaling down identically. The four-character test message's write cost dropped proportionally. Nothing about capability changed; the toolbox simply stopped charging rent on drawers nobody opened.
Keeping it warm while someone is around
The most common avoidable cold write was mundane: a user reads a reply, thinks for eight minutes, answers — and the cache TTL lapsed in the gap, so their next turn re-writes the whole prefix. The fix is a demand-gated keep-alive: after each interactive turn, a debounced timer fires a one-token request with a byte-identical prefix just inside the TTL — a pure cache read that refreshes every entry's clock. The economics are lopsided in the right direction (a fraction-of-a-cent tick against a ~$0.30 rewrite, roughly twelve to one), and the gating is the design: a real turn resets the budget, a tick cap stops warming about nine minutes after the last human action — matching the pod's own idle lifetime — and the spend lands on the platform's internal ledger, never a user's. No cron, no round-the-clock warming, no spend without plausible demand. Warming one user's prefix also keeps the shared head warm for every same-model user, a free externality of the layout above.
The test that makes editing the prompt a decision
The quietest artifact of the phase may outlast the rest: a golden test that fails whenever the shared prompt's rendered bytes change. It sounds bureaucratic and it is the opposite — it is the budget acquiring an enforcement mechanism. The shared block is identical for every agent and cached across every conversation; a casual edit to it is simultaneously a cost change for every user's every turn and a cache invalidation for the whole fleet at deploy. The test converts "someone tweaked a sentence" into a visible diff someone must deliberately accept — the same move as the unclassified-tool check, applied to prose. A budget without a gate is a graph you look at sadly once a quarter. Both of ours now have gates.
What transfers
Treat the standing prompt as a budget line, owned and measured — with the model API's own counting endpoint, never estimates, sorted into a table before anything is trimmed. Split registered from rendered and defer your long tail behind discovery, with the interactive-only scope and a kill switch. Lay the request out coldest-first and spend cache breakpoints like the scarce resource they are; keep every volatile byte behind the last one. Warm on demand with a cap, never on a schedule. And put a failing test on anything whose cost is paid by everyone — because the prompt only stays a budget while changing it feels like spending.
Postscript, August 2026. A month later a fuller audit re-measured everything and delivered two corrections we value more than the original wins. First: the per-turn spend was hiding in plain sight — the small uncached dynamic block, ~7,600 tokens, costs more per turn than the entire ~68,000-token cached prompt behind it, because uncached tokens bill at ten times the cache-read rate. A third of per-turn prompt spend from five percent of the tokens; every future optimisation now sorts by cached-versus-uncached first and size second. Second, a documentation claim died on contact with the code: we had described the tools array as broadly shareable across users, but connector tool families render conditionally on connection state — so the array is effectively per-user, and cross-user cache sharing is far narrower than the design notes claimed. The audit caught our own doc contradicting our own code, which is the argument for re-running audits at all. The one-hour cache TTL shipped once billing normalisation dissolved its fairness blocker, putting the keep-alive warmer into honourable dormancy — it re-arms automatically if the TTL ever drops back, which is what a good mechanism does when its moment passes: waits, instead of leaving.