Cloudflare has detailed how it is reducing the GPU memory required to serve Moonshot AI’s Kimi K2.6 and Z.ai’s GLM 5.2 through Workers AI. The company is combining lower-precision KV caches, compressed model weights, and cache-integrity checks to handle more requests on shared hardware.
The results need careful interpretation. Cloudflare’s August 3 engineering disclosure does not show that every request becomes faster after quantization. In one Kimi test, the lower-precision cache was slightly slower at the same concurrency level. Its advantage appeared when the reduced memory footprint allowed the deployment to accommodate twice as many concurrent requests.
That distinction matters for developers evaluating AI inference performance. Per-request speed, total throughput, GPU capacity, and cost per token measure different things.
Why Memory Becomes the Inference Bottleneck
Kimi K2.6 and GLM 5.2 are large, long-context mixture-of-experts models. Although this architecture activates only part of a model for each token, operators still need substantial GPU memory for the model’s weights and its growing KV cache.
The KV cache stores attention information for tokens the model has already processed. This allows the model to continue a conversation or analyze a long prompt without recomputing the entire context for each new token. As more requests and longer contexts are admitted, the cache can consume the GPU memory left after loading the model.
InfoSeely’s AI glossary provides plain-language explanations of inference, mixture-of-experts models, memory caches, tokens, and model weights.
Cloudflare had previously described separating inference into prefill and decode stages. Prefill processes the input and is generally limited by computation, while decode generates output tokens and is more sensitive to memory bandwidth. Running the stages in separate pools lets Cloudflare optimize each one differently.
FP8 Lets Kimi Fit More Concurrent Requests
A KV cache is commonly stored using the 16-bit BF16 format. Cloudflare instead tested an eight-bit FP8 format for the Kimi K2.6 decode stage, cutting the cache’s memory requirement roughly in half.
Cloudflare reports that the amount of cached context its deployment could keep in memory increased from about 686,000 tokens to 1.37 million tokens. This figure describes the deployment’s cache capacity and should not be confused with the maximum context window exposed to one Kimi request.
There is a performance cost at equal concurrency. On Cloudflare’s disaggregated H200 deployment, BF16 delivered 1,558 tokens per second with 32 concurrent requests, compared with 1,489 tokens per second for FP8. The additional conversion work required by the FP8 attention kernel made BF16 a few percentage points faster in that comparison.
The outcome changed when Cloudflare increased concurrency. BF16 ran out of memory beyond 32 requests, while FP8 handled 64 concurrent requests and reached 2,192 tokens per second. That was approximately 41% above BF16’s highest reported throughput.
Cloudflare also says this configuration reduced its cost per token by roughly 30%. This is a company-reported infrastructure result, not an announcement that Workers AI customers will receive a 30% price reduction.
Cloudflare continues using BF16 during prefill because that stage is compute-bound and does not receive the same benefit from the smaller cache. FP8 is applied where additional memory capacity matters most: decode.
INT4 Reduces GLM 5.2’s Weight Footprint
For GLM 5.2, Cloudflare focused on the model weights rather than only the KV cache. It compressed the weights from FP8 to four-bit integers, or INT4.
Across the eight-GPU configuration, weight memory fell from approximately 88GB to 52GB per GPU. That reduction freed about 36GB on each device for runtime data. Based on Cloudflare’s measurements, the reclaimed capacity was sufficient to store KV-cache state for roughly 1.18 million tokens.
Smaller weights also improved decode throughput because less data had to move through GPU memory for every generated token. At one concurrent request, Cloudflare measured 92 tokens per second with INT4, compared with 60 using FP8, a reported improvement of 55%. The 64-request run produced a smaller relative benefit: INT4 generated 261 more tokens each second than FP8, an improvement of approximately 15.6% in Cloudflare’s benchmark.
No single precision format delivered the best result throughout the request-processing pipeline. INT4 performed worse while GLM processed input tokens. Cloudflare recorded 8,660 tokens per second during prefill, about 15% below the 10,160 tokens per second achieved with FP8. The storage advantage does not carry cleanly into prefill. GLM’s four-bit weight data must first be unpacked into a numerical form suitable for the GPU’s arithmetic units, creating work that the FP8 run avoids.
Cloudflare therefore uses INT4 for decode and FP8 for prefill instead of treating one format as universally faster.
Accuracy Results Still Require Attribution
The published benchmark scores moved slightly in both directions after compression. Cloudflare treats those variations as too small to indicate a practical quality regression, but that interpretation has not yet been independently validated.
The scores are close but not numerically identical. For example, Kimi’s MMLU-Pro result changed from 80.29 with a BF16 cache to 79.29 with FP8. Other tests moved slightly in the opposite direction. Cloudflare describes the overall results as indistinguishable.
For GLM 5.2, the company says INT4 remained within 0.8 points of FP8 across the benchmarks it ran. These findings are useful production evidence, but independent researchers have not yet reproduced the full configurations. The use of an internal benchmark also limits how completely outside teams can audit the evaluation.
Developers considering similar compression should test their own prompts, tool calls, output formats, and failure cases rather than assuming aggregate benchmark similarity guarantees identical application behavior.
Cache Integrity Checks Address Shared-Memory Risk
Fitting more requests into GPU memory also creates a larger shared-cache environment. Paged attention, continuous batching, and cache reuse depend on accurately tracking which physical cache pages belong to each request.
Cloudflare developed an integrity-checking system that assigns a changing tag to every physical cache page. Before supported decode operations read cached information, the server checks that the page and tag match what the request expects. If they do not match, the affected request is stopped instead of receiving information from an incorrect page.
Cloudflare evaluated the safeguard with two servers handling prefill and two handling decode. As simultaneous requests increased from one to eight, throughput declined by 0.38% to 0.79%, while p95 latency rose by 0.42% to 0.80%. These company-reported results apply only to that test configuration and may differ with other models or workloads.
What Developers Should Take From the Results
The main lesson is not simply that lower numerical precision makes inference faster. It is that operators can assign different formats to different stages according to the bottleneck.
FP8 KV caching may sacrifice a small amount of same-concurrency throughput while admitting more simultaneous requests. INT4 weights may accelerate memory-bound decoding while slowing compute-bound prefill. Separating the stages allows an operator to capture the useful side of both trade-offs.
Cloudflare says its tests and production traffic use SGLang, an open-source inference-serving framework that supports prefill-decode disaggregation, continuous batching, paged attention, and multiple quantization formats.
Developers accessing the models through Workers AI should also note that Kimi K2.6 and GLM 5.2 now require the Workers Paid plan. The plan starts at $5 per month and includes a daily allocation of 10,000 free Neurons, according to Cloudflare. Model usage beyond that allocation is billed under the applicable pricing.
What Cloudflare Has Not Yet Demonstrated
Cloudflare has not disclosed the exact percentage of its fleet currently using each optimization. It says FP8 KV caches are expanding across more deployments, NVFP4 weights are being evaluated on Nvidia Blackwell hardware, and integrity checks are enabled per deployment while the company works toward wider use.
The published benchmarks also cannot predict performance for every prompt length, concurrency pattern, hardware configuration, or application. Independent reproduction, broader latency distributions, and workload-specific quality testing would provide stronger evidence.
For now, the results show how careful memory management can raise the useful capacity of expensive GPU infrastructure. They do not establish that one quantization format is the fastest choice for every model, inference stage, or developer workload.
