Kubernetes LLM Inference with llm-d: Deploy & Autoscale

TL;DR Standard Kubernetes round-robin load balancing destroys LLM performance. llm-d - a CNCF Sandbox project from Red Hat, Google Cloud, IBM Research, CoreWeave, and NVIDIA - fixes this with disaggregated prefill/decode serving, KV cache-aware routing, and SLO-aware autoscaling. Real benchmarks show 3x lower TTFT on LLaMA 4 Scout, 50% higher QPS (S2), and 2x baseline QPS (S3) on Llama 3.1 70B. This guide walks you through the architecture, the exact deployment commands, and the two autoscaling paths.

---

What Is Kubernetes LLM Inference?

Kubernetes LLM inference is the practice of serving large language models at scale using Kubernetes as the orchestration layer - managing GPU pods, routing requests, and scaling capacity in response to traffic.

It sounds straightforward. It isn't.

Standard Kubernetes was built for short-lived, uniform HTTP requests. LLM requests are the opposite: slow, expensive, and wildly variable in shape. A RAG query might send 20,000 input tokens and get back 100. A reasoning task does the reverse. Round-robin load balancing treats both identically - and that's where performance collapses.

Why standard scale-out falls short for LLMs

Three properties of LLM workloads break naive Kubernetes scaling:

1. Requests are expensive and non-uniform. Input/output token counts vary by orders of magnitude across workloads. Overloaded replicas develop longer inter-token latency (ITL), which attracts more load, which worsens ITL - a feedback loop that kills SLOs.

2. Cache locality matters enormously. vLLM implements automatic prefix caching. If a request lands on a replica that already holds the relevant KV cache entries, it skips a huge chunk of prefill computation. (vLLM's memory management is what makes this possible - see PagedAttention for Kubernetes serving.) Round-robin routing ignores this entirely, burning GPU cycles on redundant computation.

3. Prefill and decode compete for the same GPU. The prefill phase (processing the prompt) is compute-bound. The decode phase (generating tokens) is memory-bandwidth-bound. Running both on the same GPU means each phase degrades the other - especially under high concurrency.

The result: GPU utilization is pegged near 100% but throughput is mediocre, tail latencies spike, and you can't tell from CPU/GPU metrics alone whether you're saturated or just busy.

---

What Is llm-d?

llm-d (pronounced "LLM-dee") is a Kubernetes-native, open-source distributed inference serving stack. It sits above model servers like vLLM and provides the orchestration layer that standard Kubernetes lacks for LLM workloads.

It was announced at Red Hat Summit in May 2025, founded by Red Hat, Google Cloud, IBM Research, CoreWeave, and NVIDIA. On March 24, 2026, it was accepted into the CNCF Sandbox - the same governance model that produced Kubernetes and Prometheus. As of June 2026, the project is at v0.8.0 with 3,500+ GitHub stars and 552 forks.

What llm-d is not

It's not a model server. vLLM (or SGLang, or TensorRT-LLM) still handles the actual inference. (If you're still choosing an engine, compare vLLM as the inference engine in Kubernetes against the alternatives.) llm-d is the orchestration and routing layer on top - the part that decides which pod handles which request, when to scale, and how to move KV cache between nodes.

The design principles

llm-d is built around three commitments:

• Operationalizability - modular, resilient, native Kubernetes CRDs (InferencePool, InferenceObjective)
• Flexibility - validated on NVIDIA GPUs, AMD MI300X, Google TPUs, and Intel XPUs
• Performance - disaggregation + prefix-aware routing to maximize tokens/dollar while meeting SLOs

---

How llm-d Works: The Core Architecture

llm-d is built on three open-source foundations - vLLM, Kubernetes, and the Inference Gateway (IGW) - plus four key innovations on top.

The three foundations

Component	Role
vLLM	Leading open-source LLM inference engine; handles model execution
Kubernetes	Container orchestration; manages GPU pods, scaling, scheduling
Inference Gateway (IGW)	Kubernetes Gateway API extension; adds model routing, serving priority, smart load balancing

IGW is an official Kubernetes project (part of kubernetes-sigs). It integrates with Envoy, making it portable across any Kubernetes cluster.

Innovation 1: Disaggregated serving

This is llm-d's core architectural bet. Instead of running prefill and decode on the same GPU, disaggregated serving splits them onto independent worker pools.

• Prefill workers are compute-optimized. They process input prompts, build the initial KV cache, and are highly parallelizable.
• Decode workers are memory-bandwidth-optimized. They generate output tokens autoregressively using the KV cache.

Each pool scales independently. You can run 8 prefill instances at TP=1 alongside 2 decode instances at TP=4 - matching the resource profile of each phase rather than compromising both.

For medium-to-large models with long input sequences (think 10k+ tokens), P/D disaggregation delivers up to 70% higher tokens/sec vs. standard vLLM on NVIDIA B200s (AWS benchmark, GPT-OSS-120B).

Innovation 2: KV cache routing

The KV Cache Manager maintains a global, near-real-time view of which KV cache blocks live on which pods. The Inference Gateway uses this to route requests to pods that already hold relevant cached context - maximizing cache hits and skipping redundant prefill computation.

The routing pipeline works like this:

1. Incoming request arrives at the IGW
2. The External Processing Pod (EPP) scores candidate backends using the KV cache indexer
3. The indexer checks the kvblock.Index (an in-memory LRU cache) for consecutive matching blocks
4. The request routes to the pod with the highest prefix cache hit sequence
5. If no warm pod exists, the cold request is spread evenly to balance prefill load

In production testing, this achieved an 87.4% cache hit rate with sub-400ms response times for warm cache hits (vs. 2,850ms cold).

Innovation 3: NIXL transport

When a request is disaggregated, the KV cache built by the prefill worker must transfer to the decode worker. llm-d uses NIXL (NVIDIA's high-performance transport library) for this point-to-point transfer over InfiniBand, RDMA, or standard datacenter networking.

In v0.5, llm-d integrated the UCCL (Unified Collective Communication Library) backend into NIXL. Under network congestion, UCCL showed 2.4x greater resilience than UCX - latency degraded only 7.1% vs. 17.1% under heavy cross-traffic on a 200 Gb/s cluster.

Innovation 4: Advanced KV cache management

Beyond routing, llm-d v0.5+ introduced a three-tier memory hierarchy for KV cache storage: GPU → CPU → Disk.

This decouples cache capacity from GPU HBM. A shared filesystem acts as a persistent global KV store - new nodes hydrate immediately from the shared tier, bypassing the warm-up phase. At 250 concurrent users on 4× H100s, storage-backed KV offloading delivered a 13.9x throughput improvement vs. GPU-only configurations that collapsed once HBM was saturated.

LoRA prefix caching is also supported. The scheduler routes based on specific LoRA adapter cache locality, preventing the "thundering herd" problem where every replica loads every adapter simultaneously.

---

Step-by-Step: Deploy llm-d on Kubernetes

The fastest path to production is the Optimized Baseline guide - prefix-cache and load-aware routing out of the box, no disaggregation complexity required. Start here, then layer in P/D disaggregation once you've validated the baseline.

Prerequisites

Requirement	Version
Kubernetes	v1.29+
NVIDIA GPU Operator	Latest
Gateway API CRDs	v1.3.0+
Gateway API Inference Extension CRDs	v1.3.0+
cert-manager	Any recent
Hugging Face token	For model downloads

Supported accelerators: NVIDIA GPUs, AMD MI300X, Google TPU v6e/v7, Intel XPU, CPU.

Step 1: Clone the repo and set environment variables

export branch="main"
git clone https://github.com/llm-d/llm-d.git && cd llm-d && git checkout ${branch}

export REPO_ROOT=$(realpath $(git rev-parse --show-toplevel))
source ${REPO_ROOT}/guides/env.sh
export GUIDE_NAME="optimized-baseline"
export NAMESPACE=llm-d-optimized-baseline

Step 2: Install Gateway API Inference Extension CRDs

kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/releases/download/${GAIE_VERSION}/v1-manifests.yaml

Step 3: Create namespace and HuggingFace token secret

kubectl create namespace ${NAMESPACE} --dry-run=client -o yaml | kubectl apply -f -

export HF_TOKEN=<your_token>
kubectl create secret generic llm-d-hf-token \
  --from-literal="HF_TOKEN=${HF_TOKEN}" \
  --namespace "${NAMESPACE}" \
  --dry-run=client -o yaml | kubectl apply -f -

Step 4: Deploy the llm-d Router (Standalone Mode)

helm install ${GUIDE_NAME} \
  ${ROUTER_STANDALONE_CHART} \
  -f ${REPO_ROOT}/guides/recipes/router/base.values.yaml \
  -f ${REPO_ROOT}/guides/${GUIDE_NAME}/router/${GUIDE_NAME}.values.yaml \
  -n ${NAMESPACE} --version ${ROUTER_CHART_VERSION}

For Gateway Mode (with Istio, GKE, or agentgateway):

export PROVIDER_NAME=gke  # options: none, gke, agentgateway, istio
helm install ${GUIDE_NAME} \
  ${ROUTER_GATEWAY_CHART} \
  -f ${REPO_ROOT}/guides/recipes/router/base.values.yaml \
  -f ${REPO_ROOT}/guides/${GUIDE_NAME}/router/${GUIDE_NAME}.values.yaml \
  --set provider.name=${PROVIDER_NAME} \
  --set httpRoute.create=true \
  --set httpRoute.inferenceGatewayName=llm-d-inference-gateway \
  -n ${NAMESPACE} --version ${ROUTER_CHART_VERSION}

Step 5: Deploy the model server

export ACCELERATOR_TYPE=gpu   # gpu | amd | xpu | tpu/v6 | tpu/v7 | cpu
export MODEL_SERVER=vllm      # vllm | sglang | trtllm
export INFRA_PROVIDER=base    # base | gke (GPU only)

kubectl apply -n ${NAMESPACE} \
  -k ${REPO_ROOT}/guides/${GUIDE_NAME}/modelserver/${ACCELERATOR_TYPE}/${MODEL_SERVER}/${INFRA_PROVIDER}/

Step 6: Validate with a test request

export IP=$(kubectl get service ${GUIDE_NAME}-epp -n ${NAMESPACE} \
  -o jsonpath='{.spec.clusterIP}')

kubectl run curl-debug --rm -it \
  --image=cfmanteiga/alpine-bash-curl-jq \
  --namespace="${NAMESPACE}" \
  --env="IP=$IP" -- /bin/bash

# Inside the pod:
curl -X POST http://${IP}/v1/completions \
  -H 'Content-Type: application/json' \
  -d '{"model": "Qwen/Qwen3-32B", "prompt": "How are you today?"}' | jq

Deploying P/D disaggregation mode

For large models (gpt-oss-120b, DeepSeek-R1) with long input sequences, switch to the disaggregation guide:

export GUIDE_NAME="pd-disaggregation"
export NAMESPACE="llm-d-pd-disaggregation"
export MODEL_NAME="openai/gpt-oss-120b"

# Deploy router (same helm pattern as above)
helm install ${GUIDE_NAME} \
  ${ROUTER_STANDALONE_CHART} \
  -f ${REPO_ROOT}/guides/recipes/router/base.values.yaml \
  -f ${REPO_ROOT}/guides/${GUIDE_NAME}/router/${GUIDE_NAME}.values.yaml \
  -n ${NAMESPACE} --version ${ROUTER_CHART_VERSION}

# Deploy model server (8 TP=1 prefill + 2 TP=4 decode)
export INFRA_PROVIDER=base  # base | coreweave | gke | aws
kubectl apply -n ${NAMESPACE} \
  -k ${REPO_ROOT}/guides/${GUIDE_NAME}/modelserver/gpu/vllm/${INFRA_PROVIDER}

---

Autoscaling LLM Inference with llm-d

GPU utilization is a terrible autoscaling signal for LLMs. It's pegged near 100% during active batching regardless of whether you're at 10% or 100% of actual capacity. By the time CPU/memory metrics reflect saturation, latency has already spiked.

llm-d uses inference-native signals: queue depth, in-flight request counts, and KV cache pressure. Two paths are available.

Path 1: HPA + EPP Metrics (recommended for homogeneous hardware)

The Endpoint Picker (EPP) emits two key metrics:

• llm_d_epp_flow_control_queue_size - requests buffered waiting for a backend. High queue = replicas are saturated. Scale out before users feel it.
• inference_objective_running_requests - concurrent requests being processed. Useful for capacity planning.

Enable Flow Control in your EndpointPickerConfig:

apiVersion: config.x-k8s.io/v1alpha1
kind: EndpointPickerConfig
featureGates:
  - "flowControl"

Then create the HPA:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: qwen-qwen3-32b-hpa
  namespace: default
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: qwen-qwen3-32b
  minReplicas: 1
  maxReplicas: 3
  metrics:
    - type: External
      external:
        metric:
          name: epp_queue_size
        target:
          type: Value
          value: "250"
    - type: External
      external:
        metric:
          name: epp_running_requests
        target:
          type: AverageValue
          averageValue: "250"
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 0
      policies:
        - type: Percent
          value: 100
          periodSeconds: 15
    scaleDown:
      stabilizationWindowSeconds: 300

Scale-to-zero is supported. When epp_queue_size > 0, the EPP flow control layer queues incoming requests while the autoscaler provisions pods. Users see a latency spike (pod startup time) but no 5xx errors. Use KEDA if your cluster doesn't support the HPAScaleToZero alpha feature gate.

Path 2: HPA + WVA (Workload Variant Autoscaler - for heterogeneous hardware)

The Workload Variant Autoscaler (WVA) is designed for operators running multiple model variants across different GPU types (A100s, H100s, L4s) with different cost profiles.

WVA continuously monitors KV cache utilization, queue depth, and performance budgets. It calculates the optimal replica count per variant and emits a wva_desired_replicas external metric. The HPA acts on this metric. Critically, WVA preferentially adds capacity on the cheapest available variant and removes it from the most expensive - cost-aware scaling without violating latency SLOs. (Cost-aware autoscaling is only half the picture; the rest is the underlying self-hosting economics and infrastructure.)

Install WVA:

# Install WVA CRDs
kubectl apply -k github.com/llm-d/llm-d-workload-variant-autoscaler/config/base/crd?ref=release-0.8

# Install WVA controller
kubectl apply -k ${REPO_ROOT}/guides/workload-autoscaling/wva-config/platform/${PLATFORM} \
  -n ${WVA_NAMESPACE}

Enable autoscaling for your deployment:

kubectl apply -k optimized-baseline-autoscaling -n ${NAMESPACE}

The HPA will read wva_desired_replicas and scale accordingly. WVA discovers managed deployments via the llm-d.ai/managed: "true" annotation.

Choosing the right autoscaling path

	HPA + EPP Metrics	HPA + WVA
Best for	Homogeneous hardware, single model	Multi-variant, heterogeneous GPU fleet
Scaling signal	Queue depth, running requests	KV cache utilization, queue depth, cost budgets
Cost optimization	None	Prefers cheaper hardware variants
Extra components	None (standard HPA)	WVA controller required
Scale to zero	Supported	Supported

---

Performance Benchmarks

These numbers come from production deployments and partner benchmarks published by Red Hat, Tesla, AWS, and Google. All are reproducible using the llmdbenchmark CLI included in the llm-d repo.

Benchmark 1: Prefix-cache-aware routing (KV cache routing)

Setup: 2× NVIDIA 8xH100 nodes, LMbenchmark long-input/short-output configuration, comparing llm-d vs. baseline Kubernetes round-robin.

Scenario	Model	Config	ISL / OSL	Latency SLO	Result
S1	LLaMA 4 Scout FP8	TP2, 2 replicas	20,000 / 100	None	3x lower mean TTFT at 4 QPS
S2	LLaMA 4 Scout FP8	TP2, 4 replicas	12,000 / 100	P95 TTFT ≤ 2s	~50% higher QPS while meeting SLO
S3	Llama 3.1 70B FP16	TP2, 4 replicas	8,000 / 100	P95 TTFT ≤ 2s	2x baseline QPS under SLO constraints

Benchmark 2: Inference scheduling at scale (Qwen3-32B)

Setup: 8× vLLM pods, 16× NVIDIA H100 GPUs (TP=2), shared-prefix synthetic workload.

• Throughput: 4,500–11,000 output tokens/sec
• P50 TTFT: 136–157ms
• vs. baseline Kubernetes: 109% higher throughput, 99% lower TTFT at peak QPS

The baseline Kubernetes service degrades rapidly under load. llm-d maintains near-zero TTFT and scales to ~120k tokens/sec.

Benchmark 3: Wide Expert-Parallelism (NVIDIA B200)

Setup: 16× prefill GPUs / 16× decode GPUs (EP=16, DP=16, TP=1), random 1k/1k workload.

• Total throughput: ~50,000 output tokens/sec
• Per decode GPU: ~3,100 output tokens/sec

Benchmark 4: P/D disaggregation vs. aggregated (gpt-oss-120b)

Setup: 16× H200 GPUs on CoreWeave with InfiniBand, rate=45 QPS, 20:1 ISL:OSL.

Metric	Aggregated	llm-d P/D	Δ
E2E Latency (Mean)	6.7s	3.5s	-47%
E2E Latency (P95)	10.2s	5.1s	-50%
ITL (Mean)	25ms	8ms	-67%
ITL (P95)	197ms	67ms	-66%

Note: TTFT is higher in disaggregated mode because fewer resources are allocated to prefill processing. The trade-off is dramatically better ITL and end-to-end latency.

Benchmark 5: Hierarchical KV offloading

Setup: 4× NVIDIA H100, Llama-3.1-70B, 16K token requests, IBM Storage Scale.

• GPU-only: performance collapses once HBM is saturated
• Storage-backed: sustains ~185,000 tokens/sec at 250 concurrent users
• 13.9x throughput improvement at peak concurrency

---

llm-d vs. Standard vLLM Scale-Out

Capability	Standard vLLM Scale-Out	llm-d
Load balancing	Round-robin	KV cache-aware, prefix-aware, load-aware
Prefill/decode	Co-located on same GPU	Disaggregated onto independent pools
KV cache reuse	Per-replica only	Global indexing, cross-replica reuse
Autoscaling signal	CPU/GPU utilization (lagging)	Queue depth, KV cache pressure (proactive)
Scale to zero	Manual / KEDA only	Native via EPP flow control + KEDA
Multi-accelerator	NVIDIA-focused	NVIDIA, AMD, Google TPU, Intel XPU
LoRA routing	None	LoRA-precise prefix caching
Network resilience	UCX	UCCL (2.4x more resilient under congestion)
Governance	Apache 2.0	CNCF Sandbox (Apache 2.0)
Validated platforms	Generic K8s	GKE, AKS, CoreWeave CKS, OpenShift

The short version: standard vLLM scale-out works fine for low-concurrency or uniform workloads. Once you're running multi-turn agentic workloads, RAG pipelines, or large models at high QPS, the performance gap becomes significant and measurable. (For how this fits broader enterprise deployment patterns, the cost math is worth a read.)

---

Key Takeaways

The 5 things to remember from this guide:

1. Round-robin routing is the enemy of LLM performance. KV cache locality can mean the difference between 340ms and 2,850ms TTFT for the same request.

2. Disaggregated serving is not optional at scale. For models like gpt-oss-120b with long input sequences, P/D disaggregation cuts mean E2E latency by 47% and ITL by 67%.

3. Start with the Optimized Baseline. It's prefix-cache and load-aware routing out of the box - no disaggregation complexity. Benchmark it first, then layer in P/D.

4. GPU utilization is a useless autoscaling signal. Use llm_d_epp_flow_control_queue_size and inference_objective_running_requests instead. Scale before latency spikes, not after.

5. llm-d is CNCF-governed and hardware-agnostic. It runs on NVIDIA, AMD MI300X, Google TPU, and Intel XPU. The same deployment config works on GKE, AKS, CoreWeave, and OpenShift.

---

FAQ

What is Kubernetes LLM inference and why does it need a specialized framework?

Kubernetes LLM inference is the practice of running large language model serving workloads on Kubernetes clusters. It needs a specialized framework because LLM requests are slow, expensive, and non-uniform - properties that break standard Kubernetes round-robin load balancing and CPU/memory-based autoscaling. Frameworks like llm-d add KV cache-aware routing, disaggregated serving, and inference-native autoscaling signals that standard Kubernetes lacks.

What is llm-d and who maintains it?

llm-d (also written as llmd) is a Kubernetes-native distributed LLM inference serving stack. It was founded by Red Hat, Google Cloud, IBM Research, CoreWeave, and NVIDIA, announced at Red Hat Summit in May 2025, and accepted into the CNCF Sandbox on March 24, 2026. The project is licensed under Apache 2.0 and is actively maintained at github.com/llm-d/llm-d.

How does KV cache routing work in llm-d?

The KV Cache Manager maintains a global index of KV cache block locations across all decode pods. When a request arrives, the External Processing Pod (EPP) scores candidate backends by finding the pod with the longest consecutive sequence of matching KV cache blocks for that prompt prefix. Requests route to the warmest pod, skipping redundant prefill computation. In testing, this achieved an 87.4% cache hit rate with sub-400ms response times for warm hits.

When should I use disaggregated serving vs. the optimized baseline?

Use the Optimized Baseline for most workloads - it delivers significant gains with minimal operational complexity. Switch to P/D disaggregation when you're running medium-to-large models (70B+), input sequences longer than ~5,000 tokens, or sparse MoE architectures like DeepSeek-R1. For short prompts (200 ISL / 200 OSL), the KV transfer overhead can actually hurt performance.

How does LLM autoscaling on Kubernetes differ from standard HPA?

Standard HPA scales on CPU/memory, which are lagging indicators for LLM workloads - GPU utilization is pegged near 100% during active batching regardless of actual load. llm-d's autoscaling uses queue depth (llm_d_epp_flow_control_queue_size) and running request counts from the EPP as proactive signals. The Workload Variant Autoscaler (WVA) goes further, optimizing replica allocation across heterogeneous GPU types based on cost and KV cache pressure.

Does llm-d support scale-to-zero for GPU pods?

Yes. The EPP flow control layer queues incoming requests when a deployment is at zero replicas. As soon as the autoscaler provisions a pod, the EPP dispatches the queued requests. Users see a latency spike equal to pod startup time but no errors. Enable it with minReplicas: 0 on the HPA (requires the HPAScaleToZero alpha feature gate) or use KEDA as a stable alternative.

---

Useful Sources

• llm-d GitHub Repository - official source for guides, Helm charts, and release notes
• llm-d Announcement Blog - original architecture deep-dive from May 2025
• llm-d v0.5: Sustaining Performance at Scale - hierarchical KV offloading, UCCL, WVA, and scale-to-zero benchmarks
• Red Hat Developer: llm-d Kubernetes-Native Distributed Inferencing - benchmark data for S1/S2/S3 scenarios
• Red Hat Developer: Master KV Cache Aware Routing with llm-d - 87.4% cache hit rate case study
• Red Hat Blog: llm-d on CoreWeave and AKS - production deployment details
• AWS Blog: Disaggregated Inference on AWS Powered by llm-d - 70% higher tokens/sec on NVIDIA B200
• CNCF: Welcome llm-d to the CNCF - CNCF Sandbox acceptance announcement
• vLLM Documentation: llm-d Integration - official vLLM integration guide
• Solo.io: llm-d Distributed Inference Serving on Kubernetes - kgateway and routing architecture

---

Ready to Deploy?

The Optimized Baseline guide takes under 30 minutes on a GPU-enabled Kubernetes cluster. Clone the repo, set your HF_TOKEN, and run the four commands above. The benchmark CLI (llmdbenchmark) is included - you'll have reproducible performance numbers before end of day.

Start here: llm-d Quickstart Guide

Join the community: llm-d Slack - bi-weekly contributor standups every other Wednesday at 12:30 PM ET.

The GPU bill doesn't care about round-robin routing. Your users do.