The Cache Was 681,280 Tokens and the Window Was 1,048,576, in a Bite

Cloudflare has published the clearest account yet of a bug class that every hosted inference platform is exposed to, and it is about Cloudflare’s own stack.

• Between 27 August and 9 September, 155 responses from GLM 5.3 on Workers AI carried text from other customers’ requests. Every one came from a prompt longer than 681,280 tokens.
• Two faults stacked. The prefill KV cache pool held 681,280 tokens under a 1,048,576 token context window, and SGLang admitted a request its own scheduler had already marked rejected.
• The rejection was replaced by a one-token placeholder but still reported itself unfinished, so it passed admission. Decode reserved eight cache pages, one was written, and attention read all eight.
• The pool is sized from leftover GPU memory at startup rather than from a config value, so tuning quantization, tensor parallel width or CUDA graph sizes quietly resizes it.
• Fixes are in SGLang v0.5.20 through v0.5.22. Cloudflare’s pre-launch test checked status codes and throughput, not whether the answers were coherent.

Read the full analysis for what to check on your own prefill nodes.