Six Billion Requests Later: What a Full Year of LLM Serving Actually Looks Like
Most of what we "know" about LLM serving workloads comes from snapshots. An hour here, a week there, a handful of models, usually sampled and anonymized until the interesting parts are gone. That is fine for a quick sanity check, but it is a shaky foundation if you are trying to plan capacity, tune a cache, or design a router that will still make sense six months from now.
A new preprint, A Year in LLM Serving: Workload Evolution, Caching and Load-Balancing (Nixon et al., arXiv:2608.13573), goes the other way. The authors sat on a full year of production traffic from a serverless inference platform, kept it unsampled and at request level, and then actually looked at how the workload behaves over time. The scale is the headline: 6.12 billion requests, 314,970 users, 9,174 models, and 875,921 serving instances, spanning April 2025 to April 2026. Roughly 35.8 trillion input tokens and 2.5 trillion output tokens passed through it. They also say they will release the trace, which is the part that actually matters for the rest of us.
The provider is kept anonymous in the paper as "CompanyX." If you are curious, two of the authors are affiliated with a serverless inference company, so read into that what you will, but I will stick to what the paper claims.
Here is what stood out to me, and why I think it should change how you reason about serving infrastructure.
The workload never sits still
The first and most quietly important finding is that there is no such thing as "the workload." Daily request volume climbed through mid 2025, peaked around late summer and fall, then fell off into early 2026. Token volume followed a similar shape but not an identical one, which means two days with the same request count can represent very different amounts of actual serving work. If you size a fleet on request rate alone, you will get this wrong.
The model mix churns too. Active models per day grew from under 100 early on to more than 400 at the peak, driven largely by user deployed private models coming and going. The set of dominant models turned over completely across the year: early traffic was DeepSeek heavy, and later months saw newer models like Qwen3-32B take a growing share. On top of all that, the familiar weekly and daily rhythm persists, with UTC afternoons and weekdays busiest.
The practical takeaway is blunt. A one week measurement window is not enough to plan capacity for a year. The thing you tuned for is not the thing you will be serving.
Prompt heavy now, and getting more so
Requests lean input heavy. Most inputs land somewhere between hundreds and tens of thousands of tokens with a long tail of very long prompts, while outputs cluster tighter, mostly in the tens to low thousands. Most requests generate fewer output tokens than they consume.
The interesting part is the trend. Over the year, input lengths stayed broadly stable while output lengths drifted down, with the median output falling from a few hundred tokens to under a hundred near the end. Newer cohorts of users also showed up with longer, more variable prompts than earlier ones. So the balance of pressure is slowly shifting from decode toward prefill, which changes what you should be optimizing.
This is where the three "representative" models the authors track earn their keep. MiniMax-M2.5 is input heavy with short outputs, the classic coding and agentic profile. DeepSeek-R1 is the opposite, generating long reasoning outputs. DeepSeek-V3.2 sits near the global average as a chat and role play workload. Same platform, wildly different cost structures. A model's request count tells you almost nothing about the load it imposes.
Latency: decode dominates the clock, but prefill is where caching pays
End to end latency is dominated by decode. Time to first token is usually only a fraction of total request duration, and duration tracks output length. TTFT, meanwhile, tracks input length.
That split is the whole reason prefix caching is interesting. Caching does not touch decode, so it cannot fix your long tail of slow reasoning requests. What it does is cut prefill, and the benefit shows up most on long input requests where recomputing the prefix would have been genuinely expensive. On short prompts, even a near perfect cache hit saves almost nothing worth measuring. Keep that in mind before you attribute a latency win to your cache.
Prefix caching is bimodal, recency driven, and allergic to clever algorithms
The cache section is the most immediately useful part of the paper for anyone running inference infrastructure.
First, per request cache hit ratios are bimodal. A request tends to either inherit almost the entire previous turn's prefix or start basically from scratch. Partial hits are rare. This matches intuition about multi turn sessions: you either continue a conversation or you begin a new one.
Second, reuse is intensely concentrated in time. For repeat requests from the same user and model pair, roughly half arrive within 0.1 seconds, around 80 percent within 10 seconds, and 99 percent of reuse happens within 15 minutes of the previous request. In cache terms, a short time to live captures most of the value. You do not need to hoard KV state for hours.
Third, and this is the one worth arguing about at your next design review: simple eviction policies win. Replaying the trace through a simulator, FIFO and LRU matched or beat more sophisticated state of the art algorithms borrowed from web and storage caching. ARC in particular did noticeably worse at intermediate cache sizes. The workload has such a strong recency bias, prefixes get reused right after they are created and then decay fast, that fancy admission and eviction logic mostly gets in its own way. There is still a gap to the offline Belady optimum, so there is real room for LLM specific cache designs, but the lesson for today is that reaching for a complicated policy is probably premature.
Bursty, but predictably bursty
Most models are bursty, with inter arrival times uneven enough to sit above a coefficient of variation of 1. But the burstiness is persistent and positively autocorrelated, meaning busy periods follow busy periods and quiet stretches tend to stay quiet. Traffic sits in a state rather than flickering randomly.
For low traffic models this is an opportunity, not a problem. If a sparse model's active and idle windows are predictable and do not overlap with its neighbors, you can colocate or multiplex several of them on shared GPUs instead of paying for each one as an always on service. With more than 5,000 private models in this trace, most of them long tail, that is not a rounding error.
The tension nobody escapes: caching versus load balancing
Here is the part that reframes a problem you have probably already felt. Prefix caching wants you to send a user's requests back to the same instance, so the KV state is already warm. Load balancing wants to spread requests across instances so no single replica melts. These two goals actively fight.
The paper shows it directly. Under low load, a user's traffic naturally concentrates on one instance and cache hits pile up there. Under high load, the router spreads that user across many replicas to avoid overload. Hits can stay high, but only by duplicating the same prefix across multiple instance local caches, which burns capacity and evicts other users' prefixes.
They test four routing policies across the spectrum: round robin and load first ignore the cache, sticky pins each user to one instance, and cache first routes to whatever replica holds the most reusable prefix. Cache aware routing converts bigger caches into real hit ratio gains and keeps sessions from scattering, while the cache blind policies leave that capacity on the table no matter how large the cache gets. Sticky preserves locality but only at user granularity, so a heavy user is stuck with one replica's cache instead of the whole fleet's.
The surprising result is that locality is cheap. Cache first paid only about 5 to 7 percent load imbalance, largely because the workload is full of single turn requests the router can freely send to the least loaded instance to offset the sticky multi turn traffic. Workloads with fewer one off requests would have less slack.
The authors also make a good case against the obvious alternative of shuffling KV state around the network. A 100K token MiniMax-M2.5 context is roughly 27 GB of KV state, which is about 2.7 seconds to move over a 10 GB/s link one direction, ignoring contention. When most reuse happens within seconds, that transfer can cost more than it saves. Keeping the reuse in local GPU memory and tolerating a little imbalance often beats moving the data.
Why this matters if you run any of this
Strip away the numbers and the design implications are clear enough to act on.
Plan capacity on token work, not request counts, and measure over long enough windows to catch the drift. Expect your model mix to turn over. Treat prefill and decode as separate optimization targets, because different models are bound by different phases. Before you adopt an exotic cache policy, check whether plain LRU or FIFO already gets you most of the way, because for this workload it does. Lean on the persistence of bursts to colocate sparse models. And design your router knowing that cache locality and load balance are a genuine tradeoff, not a free lunch, but also that a modest tolerance for imbalance buys you a lot of reuse.
The honest caveats
This is one platform, a serverless multi model provider, so the shape of the traffic reflects its particular user base of agents, routers, and API integrations more than it reflects a single first party chat product. The cached token logging only covers the final two months, so the production cache numbers rest on a shorter window than the year long trends. Sessions had to be reconstructed rather than logged, since the trace has no prompts or explicit session IDs, using a heuristic that pairs each request to a recent earlier one within the same user. And it is a v1 preprint with a trace release that is promised but, as of writing, still pending. None of that undercuts the findings, but it is worth knowing what you are standing on.
Even with those caveats, this is the most complete public picture of production LLM serving I have seen. When the trace actually drops, it should let a lot of us stop evaluating serving systems against synthetic arrivals and made up prompts, and start testing them against something that behaves like the real thing.
Paper: A Year in LLM Serving: Workload Evolution, Caching and Load-Balancing, Nixon, Durbin, Standhartinger, Gunawi, and Yang, 2026. arXiv:2608.13573.