Deploy LLM on Kubernetes: GPU Inference Step by Step
To deploy an LLM on Kubernetes, you need four things: nodes with GPUs and the NVIDIA device plugin, a container running an inference server, persistent storage for model weights, and a Service that exposes the server's OpenAI-compatible API. The manifest is short. Most of the work, and most failed rollouts, come from model loading time, shared memory, GPU scheduling and scaling, so this post spends its time there.
The example uses vLLM, which publishes an official vllm/vllm-openai image. The same structure applies to other inference servers.
Step 1: GPU nodes
Kubernetes sees a GPU only after the NVIDIA driver and container toolkit are installed on the node and the NVIDIA device plugin advertises it. The easiest route on most clusters is the NVIDIA GPU Operator, which installs the driver, toolkit and plugin together. Managed Kubernetes services such as EKS, AKS and GKE also have GPU node pools with the drivers prepared.
Verify before going further:
kubectl describe node <gpu-node> | grep -A2 nvidia.com/gpu
You should see nvidia.com/gpu: 1 (or more) under both capacity and allocatable. Taint the GPU node pool so that only GPU workloads land on it, otherwise ordinary pods will occupy the expensive machines:
kubectl taint nodes <gpu-node> nvidia.com/gpu=present:NoSchedule
Kubernetes schedules GPUs as whole units by default. Two pods cannot share one GPU unless you configure time-slicing or, on supported cards, MIG partitions. For a first deployment, give each model server its own GPU.
Step 2: Storage for model weights
A 70B model at 4-bit is roughly 42 GB. If every pod downloads it at start-up, scaling is slow and your egress bill grows. Put the weights on a PersistentVolumeClaim and download once:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: model-cache
spec:
accessModes: ["ReadWriteMany"]
resources:
requests:
storage: 200Gi
ReadWriteMany lets several replicas read the same volume, but it needs a storage class that supports it (NFS, EFS, Azure Files and similar). With ReadWriteOnce you are limited to pods on one node.
Step 3: The Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
name: llm
spec:
replicas: 1
selector:
matchLabels: { app: llm }
template:
metadata:
labels: { app: llm }
spec:
tolerations:
- key: nvidia.com/gpu
operator: Exists
containers:
- name: vllm
image: vllm/vllm-openai:latest # pin an exact version in production
args: ["--model", "Qwen/Qwen3-8B", "--max-model-len", "8192"]
env:
- name: HF_HOME
value: /models
ports:
- containerPort: 8000
resources:
limits:
nvidia.com/gpu: 1
volumeMounts:
- { name: models, mountPath: /models }
- { name: shm, mountPath: /dev/shm }
startupProbe:
httpGet: { path: /health, port: 8000 }
periodSeconds: 10
failureThreshold: 90
readinessProbe:
httpGet: { path: /health, port: 8000 }
volumes:
- name: models
persistentVolumeClaim: { claimName: model-cache }
- name: shm
emptyDir: { medium: Memory, sizeLimit: 8Gi }
---
apiVersion: v1
kind: Service
metadata:
name: llm
spec:
selector: { app: llm }
ports:
- port: 80
targetPort: 8000
Four details in this manifest matter more than the rest.
Pin the image tag. latest can change between two restarts of the same pod, and an engine upgrade can change memory use or flags. Pin a version and upgrade deliberately.
Mount shared memory. Inference servers use /dev/shm for communication between worker processes, and the container default is small. The emptyDir with medium: Memory fixes the "bus error" and hang that many first deployments hit.
Use a startup probe with a long budget. Loading a large model takes minutes. A readiness probe alone will mark the pod unhealthy and restart it before it finishes, creating a loop. The probe above allows up to 15 minutes (90 failures at 10-second intervals); size it to your model.
Set a context limit. --max-model-len caps the KV cache. Without it a server may refuse to start because the model's full context does not fit in GPU memory. The sizing method is in our GPU memory for LLM guide.
For secrets, such as a Hugging Face token for gated models, use a Kubernetes Secret and an env reference instead of putting the token in the manifest. See the vLLM Kubernetes docs for the maintained example.
Step 4: Test it
kubectl port-forward svc/llm 8000:80
curl http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"model":"Qwen/Qwen3-8B","messages":[{"role":"user","content":"Say hello"}]}'
Any OpenAI client works against this endpoint by changing the base URL, as described in the OpenAI-compatible API guide.
Scaling
CPU-based autoscaling does nothing useful for GPU inference, because the CPU is mostly idle while the GPU saturates. Scale on a signal that reflects load: queued requests, or time to first token. The inference server exposes Prometheus metrics for these, which a Prometheus adapter or KEDA can feed to a HorizontalPodAutoscaler.
Know the limits. A new replica needs a free GPU, and a node scale-up plus model load can take several minutes, so autoscaling responds to sustained growth, not to a spike over a few seconds. Keep a minimum replica count that covers your normal peak.
What to watch once it runs is in the LLM observability guide.
What this approach costs you
You now own GPU drivers, node pools, storage classes, upgrades and failures. That is reasonable when you already run Kubernetes and have people who do it well. If you do not, the plumbing can become the project. If you are weighing it against a packaged option, Ollama in production covers the lighter end, and AI Refinery, a self-hosted LLM platform, covers the other: it launches inference servers for catalog models with one click inside your own infrastructure, with access control and an audit log, so you do not write these manifests. It does not replace a cluster for teams that want full control of every setting. See the platform overview for what it does.
Image tags, flags and chart names change between releases. Check each project's current docs before copying anything here.