The compounding cost of LLM context.

It's gone past two when the fare gets in at the Bonani Chairmanbari signal, asking to be taken from Tongi to Diabari by way of his cousin's place on Airport Road. Eleven kilometres as the crow flies, and neither I nor the crow is taking the route the crow had in mind. There is a wedding letting out near Banani 11 spilling Civic Type Rs onto the service road, the Mohakhali flyover is in a mood I cannot describe in print, and the cousin is the kind of cousin who answers on the fifth call. The meter sits on the dashboard between us doing the thing meters do, which is not lying.

By the time we are crawling under the rail crossing at Khilkhet, the fare has stopped looking at the meter. This is the moment, every time. The eye gives up around fare 280 taka because the numbers have stopped being clicks of one taka apiece and started being something more like weather. We sit at the Diabari roundabout for eleven minutes with the engine on, and that eleven minutes is the part of the ride he will be angry about at the gate, even though he agreed to it one red light at a time.

This is roughly what an agentic LLM call costs after the eighth turn of a long conversation. The bill at the gate is the part where the engineering team starts asking, with the same surprised look as my fare in the back, where exactly that came from.

Every call you make to a model in agent mode concatenates the running history into the prompt before the model sees the new message. The user turn from twelve steps ago is in there in full, the tool result from step six is in there, the system prompt is in there. The model does not retrieve from memory the way the cha-er-dokan boy retrieves your usual order. It rereads the whole conversation, every token, on every turn, and the pricing follows exactly that rereading.

The math is the meter on the dashboard. If turn N adds k new tokens, and you make N calls in a session, the total input tokens billed add up as N·k + (N-1)·k + (N-2)·k + ... which is roughly k·N²/2. Half of N squared. The shape looks linear from inside any single turn, and looks like a wall once you plot the cumulative bill at month's end. Thirty turns cost about thirty units, sixty turns one hundred and twenty, ninety turns closer to two hundred and seventy. The fare cap goes up in chunks the rider does not see coming.

I am borrowing the meter image, but the math is older than any of us. Andrej Karpathy has been pointing at the quadratic cost of transformer attention, quadratic in the length of the sequence, for years in his "Let's build GPT" lecture, reminding people that context is not a free RAM you happen to be sitting on. It is a recompute every forward pass. The invoice is linear in input tokens per call, multiplied by a call count that grows with the conversation. A linear price per token becomes a quadratic monthly invoice.

A team I sat with in February had built a competent agentic assistant on a frontier model. Internal tool, around four hundred users, sessions running fifteen to forty agent turns. Cost per call looked flat, a p50 sitting around forty milliseconds. Their monthly bill had gone from roughly four thousand US dollars in November to a little over twelve thousand in January, with no change in the user base that anyone could find. The PM kept asking the engineering lead if they had pushed a new model version. They had not. The bill was the same code against the same users, with sessions that had gradually grown longer because users had learned the tool was useful enough to keep talking to.

Two things you can do about it, in roughly the order I would recommend them.

The first is prompt caching, which most providers now offer and which Anthropic documents directly in their page about prompt caching. You mark the static portion of your prompt as cacheable. The provider stores the keyed prefix on their side for a short TTL. The next call within that TTL that starts with the same prefix bills the cached portion at roughly a tenth of the normal input rate. This does not change the quadratic shape of the conversation; it drops the constant in front of the N² by five to ten on the part of the prompt that does not change. If your system prompt is twelve thousand tokens of safety policy and your conversation is another six, caching is a bigger lever on the bill than message compression.

The second is to actually reduce the running context, which teams skip because it requires a product decision and not a config change. Summarisation of older turns into a short rolling memory is the textbook move, with the asterisk that the summariser is itself a model call that costs money and can hallucinate. Truncation of older tool outputs (keep the citation, drop the full document body) usually saves more and is safer. The vLLM team's PagedAttention paper, mostly about KV cache management on the inference side, has the cleanest description of how dramatically the working set shrinks once you stop pretending every tool output deserves to live in context until the session ends.

The reason teams stop noticing is structural. Observability at the level of a single call is what every agentic platform ships first, because that is what the trace UI surfaces naturally. Each row is a span with a duration and a token count. The user reads left to right, turn by turn, and the visual rhythm is "this turn was fine, this turn was fine," all the way down. Nobody scrolls to the bottom and asks for a sum. The sum is at the bottom of the receipt at the end of the ride, by which time the fare is at the gate and the cousin still has not picked up.

The fix is to elevate the number at the session level into a panel next to the one for each call, with the same prominence and a colour that gets your attention. Cumulative input tokens per session, cumulative per user per week, and a red light that turns on when a session has drifted far enough from the cached prefix that you are paying full rate again. Most teams I know running agentic workloads at scale built this panel themselves in the third month, after the third surprise on the invoice.

The price per token of frontier models has been falling for eighteen months, and the token count per session has been rising slightly faster. If your team has not put effort into context discipline, you are probably paying more per session now than you were a year ago even though the unit price is lower. That is the same paradox the fare in the back experiences when his cousin moves to Diabari and the rate per kilometre is unchanged from the year he lived in Mohakhali, where the number that moved was the distance and not the price.

I drop him at the gate at 02:47 and the cousin finally comes downstairs in a lungi, wearing the face of a man who has been asleep for two hours and would like everyone to know it. The fare looks at the meter, then at me, then at the meter again, and pays without complaint, because somewhere back around Khilkhet he had already done the arithmetic himself, tallying the eleven minutes at the Diabari roundabout against the wedding traffic at Banani 11 and the level crossing that is always down when you are already late.

That reconciliation is the entire difference between a fare who pays and a fare who argues at the gate, and it costs him nothing except attention spent during the ride rather than at the end of it. The meter never paused for any of those things. It has no opinion about the wedding. It knows how far we went, and it will tell you that at the gate whether or not you were watching on the way.