How to Deploy Kimi K3 Locally

• Min VRAM at 4-bit1,511 GB · 8K
• Parameters2,800B
• Active params104B
• Max context1M

Architecture

MoE

Experts offloaded

32 GB VRAM + 1,480 GB RAM

License

Kimi K3 License

Launch date

Jul 27, 2026

Vendor

Moonshot AI

Hugging Face repo

moonshotai/Kimi-K3

GGUF repo

unsloth/Kimi-K3-GGUF

NVFP4 repo (Blackwell)

RedHatAI/Kimi-K3-NVFP4

Layers

93

Hidden size

7,168

KV heads

96

Head dim

192

Full calculation rules are on the Methodology page.

Mac Studios link up over Thunderbolt 5 and run with MLX or exo; vLLM doesn't support them. Full calculation rules are on the Methodology page.

Detailed calculation rules are on the Methodology page.

As an eBay Partner, VRAMGlass may be compensated if you buy through the eBay links on this page. The price you pay stays the same.

1. 1

   Quantization

   The official weights had their experts adapted to 4-bit (MXFP4) during training, and the rest is BF16. So Unsloth's 8-bit (UD-Q8_K_XL, about 1,561 GB) keeps them as-is and is lossless, and 4-bit, at about 1,509 GB, is also close to the original. Going lower is lossy: Unsloth measured that 2-bit (about 861 GB) picks the same top token as the original about 90% of the time, and the smallest, 1-bit, about 79%.

2. 2

   Thinking can only be tuned

   Thinking is always on; you can only choose low, high or max with reasoning_effort, and the default is max. It was trained to keep its thinking history, so in multi-turn conversations and tool calls, send back the whole response from the API unchanged, including reasoning_content and tool_calls. Sampling parameters: temperature 1.0 and top_p 0.95 for general tasks, and top_p 1.0 for agent tasks.

3. 3

   Framework support

   Mainline llama.cpp merged the text part of K3 on August 15, 2026 (PR #26185); for images, you need Unsloth's fork. vLLM needs the dedicated kimi-k3 image, which only comes as a CUDA 13 build and requires host driver r580 or later, and the recipe lists eight GB300s as the minimum hardware. vLLM also warns that the model occasionally outputs tool calls in a format the parser doesn't recognize, and recommends adding format validation and retries.

4. 4

   One more license clause

   The Kimi K3 License adds one clause to the K2 series' modified MIT: “model as a service” businesses with revenue above a threshold must sign a separate agreement with Moonshot AI before commercial use. Products with over 100 million monthly active users or very high monthly revenue still have to display “Kimi K3” in their interface. Neither clause applies if you only use it inside your company or access it through Moonshot AI's official or certified inference providers.

5. 5

   Long context

   69 of its 93 layers use linear attention, with a fixed-size state that doesn't grow with context; only the cache in 24 layers grows. So at 4-bit, going from 32K to 1M adds only about 28 GB, practically negligible at this scale.

llama-server -hf unsloth/Kimi-K3-GGUF:UD-Q4_K_XL -c 8192

llama-server -hf unsloth/Kimi-K3-GGUF:UD-Q4_K_XL -c 8192 -ngl 99 \
  --cpu-moe

vllm serve moonshotai/Kimi-K3 \
  --trust-remote-code \
  --gpu-memory-utilization 0.95 \
  --tensor-parallel-size 8 \
  --load-format fastsafetensors \
  --no-enable-flashinfer-autotune \
  --max-model-len 1048576 \
  --kv-cache-dtype fp8 \
  --attention-config '{"use_prefill_query_quantization":true,"mla_prefill_backend":"TOKENSPEED_MLA"}' \
  --enable-prefix-caching \
  --attention-backend TOKENSPEED_MLA \
  --prefix-match-unit 128 \
  --enable-auto-tool-choice \
  --tool-call-parser kimi_k3 \
  --reasoning-parser kimi_k3

File names follow that week's model-library snapshot; the definitions are in the methodology. Methodology

Model	Total params	Active params	4-bit size
Kimi Linear 48B A3BMoE	49.1B	3B	30.1 GB
Kimi K2MoE	1,000B	32B	621.3 GB
Kimi K2.5MoE	1,000B	32B	621.2 GB
Kimi K2.6MoE	1,000B	32B	583.7 GB
Kimi K2.7 CodeMoE	1,000B	32B	583.7 GB

About Kimi