Cloudflare Fixes GLM-5.3 KV Cache Leak Exposing 155 Responses

Cloudflare has disclosed a significant tenant isolation failure in its Workers AI platform, where the GLM-5.3 model (@cf/zai-org/glm-5.3) leaked data between customer sessions. The incident, which occurred between August 27 and September 9, 2026, affected 155 responses, all triggered by prompts exceeding 681,280 tokens. The company has since remediated the issue and restored the model, confirming that no other hosted models were impacted.

The vulnerability was reported by security researcher Gadi Evron from Knostic via Cloudflare's HackerOne program on September 8. Cloudflare's investigation revealed a combination of a configuration oversight and a bug in the open-source SGLang inference engine, which powers the platform's disaggregated prefill/decode architecture. The company found no evidence of deliberate exploitation, noting that the affected requests were consistent with ordinary long-context usage patterns.

Root Cause: A Perfect Storm of Misconfiguration and Engine Bug

The exposure stemmed from two overlapping issues. First, Cloudflare deployed GLM 5.3 with a 1,048,576-token context window, but the KV cache pool on its prefill GPUs could only accommodate 681,280 tokens. The company was aware of this mismatch before launch but assumed the engine would return an error for oversize prompts. Testing validated HTTP response codes and performance, but not output coherence—a critical gap.

Second, SGLang's handling of rejected requests in disaggregated mode was flawed. When a prompt exceeded the pool, the scheduler marked it as rejected and replaced it with a one-token placeholder. However, instead of aborting, the prefill node generated a KV cache for the placeholder and transferred it to the decode node. The decode node, believing the full prompt had been processed, attended over stale cache pages containing other users' data, producing responses with cross-tenant content.

Immediate Remediation and Timeline

Cloudflare acted swiftly after the report. The model was set to private on September 9 at 14:00 UTC, and the root cause was identified within 35 minutes. By 14:52, all prefill nodes were rolled to a corrected configuration, and by 19:04, a full deployment with a Cloudflare-authored engine patch was validated. The model was restored at 19:40 UTC the same day.

• Configuration fix: The KV pool was resized to hold approximately 2.04 million tokens, eliminating the rejection path for valid requests.
• Engine patch: Rejected requests are now retired at the admission queue before any KV transfer occurs, preventing the bug from being triggered.
• Upstream collaboration: Cloudflare worked with SGLang maintainers at RadixArk to merge multiple fixes, including #36988 and #38935.

Ongoing Hardening and Recommendations

Beyond the immediate fix, Cloudflare has implemented broader hardening measures across all its SGLang deployments. These include deferred KV release on both prefill and decode roles, bounded Mooncake transfer batches, and improved load balancer behavior to avoid unnecessary aborts. The company also introduced a KV checksum validation feature (--disaggregation-enable-kv-checksum) to detect corruption at the handoff.

For operators running SGLang in disaggregated mode, Cloudflare recommends: measuring KV pool size against context length on every node, upgrading to v0.5.22 or later, enabling deferred release, and turning on corruption detection where supported. The company emphasizes that configuration checks alone are insufficient—the engine-side fix is the critical backstop.

Long-Term Vision: Integrity by Construction

Cloudflare is pursuing two long-term strategies to prevent similar incidents. The first, "Integrity by construction," involves tagging every physical KV page with a unique identifier that changes on reallocation. Each request records expected tags, and the mapping is verified before decode reads the cache. A prototype showed under 1% performance overhead but was disabled due to reliability issues. The company is now working with SGLang to build this into the engine.

The second approach applies formal verification methods to RDMA-based KV transfer protocols, similar to Cloudflare's work on Topaz and Meerkat. The goal is to mathematically prove the safety of the distributed cache system, identifying and fixing any divergences from desired properties.

Coordinated Disclosure and Industry Impact

This incident highlights the growing complexity of securing multi-tenant AI infrastructure. As models scale to million-token contexts, the interplay between inference engines, distributed memory management, and configuration becomes increasingly critical. Cloudflare's transparent disclosure and collaboration with the SGLang team set a benchmark for handling such vulnerabilities.

The researcher, Gadi Evron, provided a reproducible script that enabled rapid verification and remediation. Cloudflare's response—from incident declaration to model restoration in under seven hours—demonstrates the importance of robust security processes in AI platforms. For enterprises relying on Workers AI, this serves as a reminder to review their own deployment configurations and stay updated with engine patches.