How to Deploy Hy4 preview Locally

MoE: all 770B parameters must stay resident in memory, while speed is set by the 49B active per token.

Total = weights + KV cache (FP16) + 1GB runtime overhead. Weights marked GGUF are measured file sizes from the repository; the rest are estimated from bytes per parameter. Contexts beyond this model's 1M limit show a dash. Each tier takes whichever file of that bit width the repository has, preferring unsloth dynamic quants, so the exact file name varies by model.

Fits when weights + KV cache + runtime overhead is at or below usable memory, single card. "Offload" on MoE models means it runs with expert weights in system RAM (only attention layers and KV cache stay in VRAM); that speed assumes 70 GB/s RAM bandwidth. The context column is the largest tier that fits.

Expert weights live in system RAM (needs at least 415 GB); VRAM holds only attention layers, shared experts and the KV cache, at least 18 GB at Q4. Speed is bound by RAM bandwidth (estimated at 70 GB/s) and is far slower than a full-VRAM setup.

Device	VRAMGB	Used price	RAM neededGB	Est. t/s
Radeon RX 7900 XT	20	$690	415	3.2
Tesla V100 32GB	32	$739	415	3.4
Radeon RX 7900 XTX	24	$1,000	415	3.3
Instinct MI100	32	$1,084	415	3.3
GeForce RTX 3090	24	$1,550	415	3.4
RTX PRO 4000 Blackwell	24	$2,950	415	3.3
GeForce RTX 4090	24	$3,200	415	3.4
Radeon PRO W7900	48	$3,495	415	3.2
RTX PRO 4500 Blackwell	32	$4,300	415	3.4
Instinct MI210	64	$5,046	415	3.4
MacBook Pro M4 Max 128GB	128	$5,400	415	3.1
GeForce RTX 5090	32	$5,925	415	3.5
A100 40GB PCIe	40	$6,000	415	3.5
MacBook Pro M5 Max 128GB	128	$7,250	415	3.1
RTX PRO 6000 Blackwell	96	$16,985	415	3.5
GeForce RTX 2080 Ti 22GB	22	—	415	3.2
Arc Pro B60	24	—	415	2.9
GeForce RTX 5090 D V2	24	—	415	3.4
Arc Pro B65	32	—	415	3
Arc Pro B70	32	—	415	3
GeForce RTX 4080 SUPER 32GB	32	—	415	3.3
GeForce RTX 5090 D	32	—	415	3.5
Instinct MI50 32GB	32	—	415	3.3
Radeon AI PRO R9700	32	—	415	3.1
Radeon PRO W7800	32	—	415	3.1
CMP 170HX 40GB	40	—	415	3.5
GeForce RTX 4090 48GB	48	—	415	3.4
RTX PRO 5000 Blackwell 48GB	48	—	415	3.4
CMP 170HX 64GB	64	—	415	3.5
RTX PRO 5000 Blackwell 72GB	72	—	415	3.4
RTX PRO 6000D	84	—	415	3.5
DGX Spark 128GB	128	—	415	2.8
Mac Studio M4 Max 128GB	128	—	415	3.1
Mac Studio M5 Max 128GB	128	—	415	3.1
Ryzen AI Max+ 395 128GB	128	—	415	2.6

./build-cuda/bin/llama-cli -m Hy4-preview-Q4_K_M.gguf -ngl 99 -c 8192 --jinja -st -f prompt.txt

docker run --gpus all \
  -p 8000:8000 \
  --ipc=host \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:hy4-preview tencent/Hy4-preview-FP8 \
    --tensor-parallel-size 8 \
    --speculative-config '{"num_speculative_tokens":3,"method":"mtp"}' \
    --attention-backend FLASHMLA_SPARSE \
    --tool-call-parser hy_v4 \
    --reasoning-parser hy_v4 \
    --enable-auto-tool-choice \
    --port 8000 \
    --served-model-name hy4-preview

File names follow that week's model-library snapshot; the definitions are in the methodology. Methodology