vLLM/Recipes
DeepSeek

deepseek-ai/DeepSeek-V3

DeepSeek-V3 is a 671B-parameter Mixture-of-Experts model with native FP8 weights and strong reasoning, coding, and math capabilities.

Frontier open-weights MoE with native FP8 and FP4 variants

View on HuggingFace
moe671B / 37B163,840 ctxvLLM 0.12.0+text
Guide

Overview

DeepSeek-V3 is a 671B-parameter Mixture-of-Experts model (37B activated per token) shipped with native FP8 weights. It shares its architecture with DeepSeek-R1, so the same launch recipes apply to both models. DeepSeek publishes a refreshed checkpoint as DeepSeek-V3-0324, available here as the V3 0324 variant. For Blackwell GPUs, NVIDIA publishes an FP4 quantized variant (nvidia/DeepSeek-V3-FP4 / nvidia/DeepSeek-R1-FP4) that runs on fewer GPUs.

Prerequisites

  • Hardware (FP8): 8x H200 (CUDA) or 8x MI300X / MI325X / MI355X (ROCm)
  • Hardware (FP4): 4x B200 GPUs
  • vLLM: Current stable release

CUDA

uv venv
source .venv/bin/activate
uv pip install -U vllm --torch-backend auto

ROCm (MI300X, MI325X, MI355X)

Requires Python 3.12, ROCm 7.2.1, and glibc >= 2.35.

uv venv --python 3.12
source .venv/bin/activate
uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/

Client Usage

8xH200 / 8xMI300X (FP8)

Tensor Parallel + Expert Parallel (TP8+EP) — CUDA:

vllm serve deepseek-ai/DeepSeek-V3 \
  --trust-remote-code \
  --tensor-parallel-size 8 \
  --enable-expert-parallel

Tensor Parallel + Expert Parallel (TP8+EP) — ROCm:

export VLLM_ROCM_USE_AITER=1

vllm serve deepseek-ai/DeepSeek-V3 \
  --trust-remote-code \
  --tensor-parallel-size 8 \
  --enable-expert-parallel

Data Parallel + Expert Parallel (DP8+EP) — CUDA:

vllm serve deepseek-ai/DeepSeek-V3 \
  --trust-remote-code \
  --data-parallel-size 8 \
  --enable-expert-parallel

8xMI350X / 8xMI355X (DPA+TP)

Each GPU runs its own attention replica with a private KV cache, while the MoE experts stay tensor-sharded across all eight GPUs. Aggregate KV-cache capacity is roughly 8x that of a single TP8 replica, useful for workloads benefiting from larger KV capacities.

Engine (all eight ranks in one process, listening on 8100):

export VLLM_ROCM_USE_AITER=1
export VLLM_ENGINE_READY_TIMEOUT_S=1800

vllm serve deepseek-ai/DeepSeek-V3-0324 \
  --trust-remote-code \
  --port 8100 \
  --data-parallel-size 8 \
  --tensor-parallel-size 1

Router (client-facing on 8000, spreads requests across the eight DP ranks):

uv pip install vllm-router

vllm-router \
  --host 0.0.0.0 \
  --port 8000 \
  --policy consistent_hash \
  --intra-node-data-parallel-size 8 \
  --worker-urls http://localhost:8100 \
  --worker-startup-timeout-secs 1800

Notes:

  • --intra-node-data-parallel-size must match the engine's --data-parallel-size. A mismatch silently under-uses ranks.
  • consistent_hash is the policy that makes this worthwhile for chat: it pins a conversation to the rank that already holds its prefix, so the larger aggregate cache actually gets hit. round_robin scatters turns of the same conversation across ranks and throws the prefix away.
  • Start the engine first, then the router; --worker-startup-timeout-secs 1800 covers the engine's weight load either way.
  • Send client traffic to the router on port 8000, never to 8100 directly.

4xB200 (FP4)

On vLLM v0.28.0 and later, the FlashInfer MoE kernels are selected automatically for both FP4 and FP8 on Blackwell — no environment variables are needed. To pin the backend explicitly, pass --moe-backend flashinfer_trtllm.

Tensor Parallel + Expert Parallel (TP4+EP):

CUDA_VISIBLE_DEVICES=0,1,2,3 vllm serve nvidia/DeepSeek-V3-FP4 \
  --trust-remote-code \
  --tensor-parallel-size 4 \
  --enable-expert-parallel

Data Parallel + Expert Parallel (DP4+EP):

CUDA_VISIBLE_DEVICES=0,1,2,3 vllm serve nvidia/DeepSeek-V3-FP4 \
  --trust-remote-code \
  --data-parallel-size 4 \
  --enable-expert-parallel

Benchmarking

For benchmarking, prefix caching is disabled by default in vLLM — no extra server flag is needed.

# Prompt-heavy benchmark (8k/1k)
vllm bench serve \
  --model deepseek-ai/DeepSeek-V3 \
  --dataset-name random \
  --random-input-len 8000 \
  --random-output-len 1000 \
  --request-rate 10000 \
  --num-prompts 16 \
  --ignore-eos

Test different workloads by adjusting input/output lengths:

  • Prompt-heavy: 8000 input / 1000 output
  • Decode-heavy: 1000 input / 8000 output
  • Balanced: 1000 input / 1000 output

Troubleshooting

References