Inside vLLM: Anatomy of a High-Throughput LLM Inference System (2025) In this post, I'll gradually introduce all of the core system components and advanced features that make up a modern high-through
By Coderz Club · 2026-08-07 · Tags: go
Inside vLLM: Anatomy of a High-Throughput LLM Inference System (2025)
In this post, I'll gradually introduce all of the core system components and advanced features that make up a modern high-throughput LLM inference system. In particular I'll be doing a breakdown of how vLLM [1] works.This post is the first in a series. It starts broad and then layers in detail (following an inverse-pyramid approach) so you can form an accurate high-level mental model of the complete system without drowning in minutiae.Later posts will dive into specific subsystems.This post is structured into five parts:LLM engine & engine core: fundamentals of vLLM (scheduling, paged attention, continuous batching, etc.) Advanced features: chunked prefill, prefix caching, guided & speculative decoding, disaggregated P/DScaling up: from single-GPU to multi-GPU executionServing layer: distributed / concurrent web scaffoldingBenchmarks and auto-tuning: measuring latency and throughput 📝NotesAnalysis is based on commit 42172ad (August 9th, 2025).Target audience: anyone curious about how state-of-the-art LLM engines work, as well as those interested in contributing to vLLM, SGLang, etc.I'll focus on the V1 engine. I also explored V0 (now deprecated), which was valuable for understanding how the project evolved, and many concepts still carry over.The first section on LLM Engine / Engine Core might be a bit overwhelming/dry - but the rest of the blog has plenty examples and visuals. :)LLM Engine & Engine CoreThe LLM engine is the fundamental building block of vLLM. On its own, it already enables high-throughput inference - but only in an offline setting. You can't serve it to customers over the web yet.We'll use the following offline inference snippet as our running example (adapted from basic.py).from vllm import LLM, SamplingParams prompts = [ "Hello, my name is", "The president of the United States is", ] sampling_params = SamplingParams(temperature=0.8, top_p=0.95) def main(): llm = LLM(model="TinyLlama/TinyLlama-1.1B-Chat-v1.0") outputs = llm.generate(prompts, sampling_params) if __name__ == "__main__": main()📝Environment vars:VLLM_USE_V1="1" # we're using engine V1VLLM_ENABLE_V1_MULTIPROCESSING="0" # we're running in a single processThis configuration is:offline (no web/distributed system scaffolding)synchronous (all execution happens in a single blocking process)single-GPU (no data/model/pipeline/expert parallelism; DP/TP/PP/EP = 1)using standard transformer [2] (supporting hybrid models like Jamba requires a more complex hybrid KV-cache memory allocator)From here, we'll gradually build up to an online, async, multi-GPU, multi-node inference system - but still serving a standard transformer.In this example we do two things, we:Instantiate an engineCall generate on it to sample from the given promptsLet's start analyzing the constructor.LLM Engine constructorThe main components of the engine are:vLLM config (contains all of the knobs for configuring model, cache, parallelism, etc.)processor (turns raw inputs → EngineCoreRequests via validation, tokenization, and processing)engine core client (in our running example we're using InprocClient which is basically == EngineCore; we'll gradually build up to DPLBAsyncMPClient which allows serving at scale)output processor (converts raw EngineCoreOutputs → RequestOutput that the user sees)📝Note:With the V0 engine being deprecated, class names and details may shift. I'll emphasize the core ideas rather than exact signatures. I'll abstract away some but not all of those details.Engine core itself is made up of several sub components:Model Executor (drives forward passes on the model, we're currently dealing with UniProcExecutor which has a single Worker process on a single GPU). We'll gradually build up to MultiProcExecutor which supports multiple GPUsStructured Output Manager (used for guided decoding - we'll cover this later)Scheduler (decides which requests go into the next engine step) - it further contains:policy setting - it can be either FCFS (first come first served) or priority (higher priority requests are served first)waiting and running queuesKV cache manager - the heart of paged attention [3]The KV-cache manager maintains a free_block_queue - a pool of available KV-cache blocks (often on the order of hundreds of thousands, depending on VRAM size and block size). During paged attention, the blocks serve as the indexing structure that map tokens to their computed KV cache blocks.Core components described in this section and their relationshipsBlock size for a standard transformer layer (non-MLA [4]) is computed as follows: 2 (key/value) * block_size (default=16) * num_kv_heads * head_size * dtype_num_bytes (e.g. 2 for bf16)During model executor construction, a Worker object is created, and three key procedures are executed. (Later, with MultiProcExecutor, these same procedures run independently on each worker process across different GPUs.)Init device:Assign a CUDA device (e.g. "cuda:
In this post, I'll gradually introduce all of the core system components and advanced features that make up a modern high-throughput LLM inference system. In particular I'll be doing a breakdown of how vLLM [1] works.This post is the first in a series. It starts broad and then layers in detail (following an inverse-pyramid approach) so you can form an accurate high-level mental model of the complete system without drowning in minutiae.Later posts will dive into specific subsystems.This post is structured into five parts:LLM engine & engine core: fundamentals of vLLM (scheduling, paged attention, continuous batching, etc.) Advanced features: chunked prefill, prefix caching, guided & speculative decoding, disaggregated P/DScaling up: from single-GPU to multi-GPU executionServing layer: distributed / concurrent web scaffoldingBenchmarks and auto-tuning: measuring latency and throughput 📝NotesAnalysis is based on commit 42172ad (August 9th, 2025).Target audience: anyone curious about how state-of-the-art LLM engines work, as well as those interested in contributing to vLLM, SGLang, etc.I'll focus on the V1 engine. I also explored V0 (now deprecated), which was valuable for understanding how the project evolved, and many concepts still carry over.The first section on LLM Engine / Engine Core might be a bit overwhelming/dry - but the rest of the blog has plenty examples and visuals. :)LLM Engine & Engine CoreThe LLM engine is the fundamental building block of vLLM. On its own, it already enables high-throughput inference - but only in an offline setting. You can't serve it to customers over the web yet.We'll use the following offline inference snippet as our running example (adapted from basic.py).from vllm import LLM, SamplingParams prompts = [ "Hello, my name is", "The president of the United States is", ] sampling_params = SamplingParams(temperature=0.8, top_p=0.95) def main(): llm = LLM(model="TinyLlama/TinyLlama-1.1B-Chat-v1.0") outputs = llm.generate(prompts, sampling_params) if __name__ == "__main__": main()📝Environment vars:VLLM_USE_V1="1" # we're using engine V1VLLM_ENABLE_V1_MULTIPROCESSING="0" # we're running in a single processThis configuration is:offline (no web/distributed system scaffolding)synchronous (all execution happens in a single blocking process)single-GPU (no data/model/pipeline/expert parallelism; DP/TP/PP/EP = 1)using standard transformer [2] (supporting hybrid models like Jamba requires a more complex hybrid KV-cache memory allocator)From here, we'll gradually build up to an online, async, multi-GPU, multi-node inference system - but still serving a standard transformer.In this example we do two things, we:Instantiate an engineCall generate on it to sample from the given promptsLet's start analyzing the constructor.LLM Engine constructorThe main components of the engine are:vLLM config (contains all of the knobs for configuring model, cache, parallelism, etc.)processor (turns raw inputs → EngineCoreRequests via validation, tokenization, and processing)engine core client (in our running example we're using InprocClient which is basically == EngineCore; we'll gradually build up to DPLBAsyncMPClient which allows serving at scale)output processor (converts raw EngineCoreOutputs → RequestOutput that the user sees)📝Note:With the V0 engine being deprecated, class names and details may shift. I'll emphasize the core ideas rather than exact signatures. I'll abstract away some but not all of those details.Engine core itself is made up of several sub components:Model Executor (drives forward passes on the model, we're currently dealing with UniProcExecutor which has a single Worker process on a single GPU). We'll gradually build up to MultiProcExecutor which supports multiple GPUsStructured Output Manager (used for guided decoding - we'll cover this later)Scheduler (decides which requests go into the next engine step) - it further contains:policy setting - it can be either FCFS (first come first served) or priority (higher priority requests are served first)waiting and running queuesKV cache manager - the heart of paged attention [3]The KV-cache manager maintains a free_block_queue - a pool of available KV-cache blocks (often on the order of hundreds of thousands, depending on VRAM size and block size). During paged attention, the blocks serve as the indexing structure that map tokens to their computed KV cache blocks.Core components described in this section and their relationshipsBlock size for a standard transformer layer (non-MLA [4]) is computed as follows: 2 (key/value) * block_size (default=16) * num_kv_heads * head_size * dtype_num_bytes (e.g. 2 for bf16)During model executor construction, a Worker object is created, and three key procedures are executed. (Later, with MultiProcExecutor, these same procedures run independently on each worker process across different GPUs.)Init device:Assign a CUDA device (e.g. "cuda: