DeepSeek Open-Sources DeepInfer: The Engine Behind Its 155T Daily Tokens

DeepSeek Pulls Back the Curtain on DeepInfer

DeepSeek has released DeepInfer, the high-performance inference framework that powers its large-scale production services, as an open-source project under the Apache License 2.0. The announcement, made on October 21, 2026, gives developers and enterprises a rare look at the software stack that serves an astonishing 155 trillion tokens per day across a single cluster.

The framework, which has already amassed over 10,800 GitHub stars, is not a stripped-down demo—it includes the core features DeepSeek uses in production, albeit with some key components held back. This release marks a significant step in DeepSeek's strategy to share its infrastructure while maintaining a competitive edge.

What Exactly Is DeepInfer?

DeepInfer is an inference engine—the software layer that runs trained AI models, processes incoming requests, and generates responses. Its efficiency directly determines compute costs and energy consumption for AI services. DeepSeek's engineers built DeepInfer primarily on vLLM, drawing inspiration from SGLang and TensorRT-LLM, but heavily customized it with proprietary optimizations.

At launch, DeepInfer supports DeepSeek-V4.1-Flash, the company's latest model, with code for future models promised to land in the repository. The framework runs on NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs, reflecting DeepSeek's dual-platform strategy.

Key Features: Optimized Kernels and DSpark

The framework bundles several optimized kernels that DeepSeek has previously open-sourced, including FlashMLA, DeepGEMM, DeepEP, and TileKernels. These are complemented by custom TileLang kernels for model execution. The result is a highly tuned runtime for both prefill and decode phases.

One of the standout features is DSpark, DeepSeek's confidence-scheduled speculative decoding. This technique uses draft tokens to accelerate decoding, with confidence estimates determining when to verify predictions. Developers can control the maximum draft length via the DEEPINFER_USE_NEXTN_DECODE environment variable, defaulting to 5 tokens.

Other notable features include:

• Parallelism: Attention DP / balanced CP, plus MoE EP with EPLB load balancing
• SWA bounded replay: Reduces decoder prefill work by replaying only a bounded suffix of the context
• Asynchronous scheduling: Overlaps CPU batch scheduling with device execution, enabled by default
• Batch-invariant inference: Ensures identical results across different batch compositions (NVIDIA only)
• Deterministic sampling: Reproducible outputs with fixed seeds (NVIDIA only)
• Structured outputs: Constrains generation to regex, JSON schema, or EBNF grammars

What's Missing? The Proprietary Parts

DeepInfer is not a complete inference service out of the box. DeepSeek explicitly notes that the release excludes:

• A KV cache system for managing key-value caches at scale
• VE/Prefill/Decode disaggregation (though most required engine functionality is implemented)
• Load balancing across data-parallel replicas for prefill and decode

These components are critical for multi-tenant, production-grade serving. By withholding them, DeepSeek retains a competitive advantage in its own cloud offerings while still providing a valuable foundation for developers.

Production Insights and Performance

The release includes performance benchmarks measured on 4 SM100 GPUs for prefill and 64 SM100 GPUs for decode, each within one scale-up domain. The company reports US$38 per GPU-hour revenue at valley pricing, offering a glimpse into the economics of large-scale AI inference.

In production, DeepSeek runs decode with expert parallelism (EP) from 64 to 128 within a supernode, while prefill uses the smallest EP that memory allows—typically EP2 to EP8. This asymmetric configuration optimizes throughput and latency.

Also Open-Sourced: deepseek-recipe

Alongside DeepInfer, DeepSeek released deepseek-recipe, a production-grade token encoding and decoding component. It converts between mainstream LLM API protocols and model token sequences, ensuring third-party API providers can replicate the exact behavior of the official DeepSeek API. This move simplifies the deployment of compatible services.

Why This Matters

DeepInfer's open-sourcing is a strategic play. By sharing its inference stack, DeepSeek enables the broader ecosystem to optimize for its models, potentially increasing adoption. At the same time, the company keeps its most advanced serving capabilities proprietary, protecting its commercial interests.

The move also pressures competitors like vLLM and SGLang, which serve as the industry standard for open-source inference. DeepInfer's production-proven performance and DeepSeek's rapid release cadence could make it a preferred choice for deploying DeepSeek models specifically.

Getting Started

To use DeepInfer, developers need Linux, GCC 13+, Python 3.12, and either CUDA 13 (NVIDIA) or CANN 9.2 (Ascend). The framework is built from source, and the default checkpoint deepseek-ai/DeepSeek-V4.1-Flash-1017 is downloaded from Hugging Face on first run. A simple python examples/offline_inference.py gets you started.

DeepSeek has committed to open-sourcing inference code for future models in this repository, signaling that DeepInfer will evolve alongside its model releases. For now, it's a powerful tool for anyone looking to run DeepSeek-V4.1-Flash at scale with cutting-edge optimizations.