How to Deploy Hy4 preview Locally
Hy4 preview is a MoE model with 770B total parameters and 49B active per token. At Q4 it needs 431GB of VRAM to sit entirely on the GPU; with experts offloaded to system RAM it needs only 18GB of VRAM plus 415GB of RAM. The cheapest card that runs it today is the Mac Studio M3 Ultra 512GB at about $19,315, at roughly 15 tokens/s. In offload mode the cheapest card that runs it is the GeForce RTX 2080 Ti 22GB, at roughly 3 tokens/s.
Hy4 preview details
How much VRAM does Hy4 preview need for local deployment?
MoE: all 770B parameters must stay resident in memory, while speed is set by the 49B active per token.
| Quantization | WeightsGB | Total at 8KGB | Total at 32KGB | Total at 128KGB |
|---|---|---|---|---|
| 1-bit | 235.4GGUF | 182.4 | 184.5 | 192.7 |
| 2-bit | 286.1est. | 287.8 | 289.9 | 298.1 |
| 3-bit | 369est. | 370.6 | 372.7 | 380.9 |
| 4-bit | 467.3GGUF | 430.9 | 432.9 | 441.2 |
| 5-bit | 519.6est. | 521.2 | 523.3 | 531.5 |
| 8-bit | 798.2est. | 799.8 | 801.9 | 810.1 |
| FP16 | 1,505.9est. | 1,507.6 | 1,509.7 | 1,517.9 |
Total = weights + KV cache (FP16) + 1GB runtime overhead. Weights marked GGUF are measured file sizes from the repository; the rest are estimated from bytes per parameter. Contexts beyond this model's 1M limit show a dash. Each tier takes whichever file of that bit width the repository has, preferring unsloth dynamic quants, so the exact file name varies by model.
Which GPUs can run Hy4 preview?
Xianyu
| Device | VRAMGB | Price | Est. t/s | Suggested context |
|---|---|---|---|---|
| Mac Studio M3 Ultra 512GB | 512 | $19,3156 listings | 15 | 128K |
Fits when weights + KV cache + runtime overhead is at or below usable memory, single card. "Offload" on MoE models means it runs with expert weights in system RAM (only attention layers and KV cache stay in VRAM); that speed assumes 70 GB/s RAM bandwidth. The context column is the largest tier that fits.
Recommended GPUs for Hy4 preview with CPU/GPU offloading
Xianyu
Expert weights live in system RAM (needs at least 415 GB); VRAM holds only attention layers, shared experts and the KV cache, at least 18 GB at Q4. Speed is bound by RAM bandwidth (estimated at 70 GB/s) and is far slower than a full-VRAM setup.
Deploy Hy4 preview locally with Ollama, llama.cpp or vLLM
The Ollama library has no official tag for it yet.
Follow the deployment guide to prepare the required runtime and hardware.
./build-cuda/bin/llama-cli -m Hy4-preview-Q4_K_M.gguf -ngl 99 -c 8192 --jinja -st -f prompt.txt
Follow the deployment guide to prepare the required runtime and hardware.
docker run --gpus all \
-p 8000:8000 \
--ipc=host \
-v ~/.cache/huggingface:/root/.cache/huggingface \
vllm/vllm-openai:hy4-preview tencent/Hy4-preview-FP8 \
--tensor-parallel-size 8 \
--speculative-config '{"num_speculative_tokens":3,"method":"mtp"}' \
--attention-backend FLASHMLA_SPARSE \
--tool-call-parser hy_v4 \
--reasoning-parser hy_v4 \
--enable-auto-tool-choice \
--port 8000 \
--served-model-name hy4-previewFile names follow that week's model-library snapshot; the definitions are in the methodology. Methodology
FAQ
How much VRAM does Hy4 preview need?
At Q4_K_M with an 8K context Hy4 preview needs about 431GB of VRAM; 520GB leaves comfortable headroom.
What is the cheapest GPU that runs Hy4 preview?
The Mac Studio M3 Ultra 512GB: 512GB of VRAM, currently about $19,315, at roughly 15 tokens/s.
Can you run Hy4 preview with Ollama?
Not yet: the Ollama library has no official tag for it. You can load the GGUF with llama.cpp instead.