How to Deploy Kimi K2.6 Locally

• Min VRAM at 4-bit586 GB · 8K
• Parameters1,000B
• Active params32B
• Max context256K

Architecture

MoE

Experts offloaded

10 GB VRAM + 578 GB RAM

License

Modified MIT

Launch date

Apr 20, 2026

Vendor

Moonshot AI

Hugging Face repo

moonshotai/Kimi-K2.6

GGUF repo

unsloth/Kimi-K2.6-GGUF

NVFP4 repo (Blackwell)

nvidia/Kimi-K2.6-NVFP4

MLX repo (Apple)

mlx-community/Kimi-K2.6-mlx-DQ3_K_M-q8

Layers

61

Hidden size

7,168

KV heads

64

Head dim

192

Full calculation rules are on the Methodology page.

Only common GPUs and systems are listed, one set for models under 50B and another for larger ones, sorted by price from low to high.

Device	Cards	Total price	Total VRAM	Speed t/s	Total power
Mac Studio M3 Ultra 512GB512 GB per unit	2 units	$45,800	1,024 GB	27.6	960 W
A100 80GB PCIe80 GB per card	8 cards	$151,992	640 GB	55.4	2,400 W
RTX PRO 600096 GB per card	8 cards	$180,000	768 GB	52.3	4,800 W

Mac Studios link up over Thunderbolt 5 and run with MLX or exo; vLLM doesn't support them. Full calculation rules are on the Methodology page.

Detailed calculation rules are on the Methodology page.

As an eBay Partner, VRAMGlass may be compensated if you buy through the eBay links on this page. The price you pay stays the same.

1. 1

   Only three quantization levels

   Unsloth has released only three: 2-bit (about 341 GB), 4-bit (about 584 GB) and 8-bit (about 595 GB). The 8-bit keeps the 4-bit experts as-is and uses BF16 for the rest, so it counts as lossless; 4-bit just compresses the rest further to 8-bit, and its difference from 8-bit is negligible (perplexity 1.8420 vs. 1.8419). Unsloth recommends 2-bit as the balance point between size and quality.

2. 2

   Thinking, instant and preserve_thinking

   Sampling parameters carry over from K2.5: temperature 1.0 for thinking mode and 0.6 for instant mode, with top_p 0.95. Switching to instant mode works the same way, by turning off thinking under chat_template_kwargs. The new preserve_thinking option is off by default; turning it on keeps all thinking content across multi-turn conversations, and Moonshot recommends enabling it only for coding agents running in thinking mode.

3. 3

   Images

   Starting with K2.6, Unsloth's GGUFs include the projector file (mmproj) for images, so adding --mmproj in llama.cpp lets you input images and use its vision capabilities locally too.

4. 4

   vLLM requirements

   The vLLM recipe requires 0.25 or later. The official original (INT4) was validated on eight H200s, and the recipe puts the requirement at about 640 GB of VRAM in total; on B300 you can switch to the NVFP4 version, where four cards are enough.

5. 5

   Context

   Native context is 256K, and it's light on VRAM: at 4-bit, going from 8K to 256K adds only about 18 GB. Unsloth recommends a default of 98,304 (about 96K).

llama-server -hf unsloth/Kimi-K2.6-GGUF:UD-Q4_K_XL -c 8192

llama-server -hf unsloth/Kimi-K2.6-GGUF:UD-Q4_K_XL -c 8192 -ngl 99 \
  --cpu-moe

vllm serve moonshotai/Kimi-K2.6 \
  --trust-remote-code \
  --tensor-parallel-size 8 \
  --tool-call-parser kimi_k2 \
  --enable-auto-tool-choice \
  --reasoning-parser kimi_k2 \
  --speculative-config '{"model":"lightseekorg/kimi-k2.6-eagle3-mla","method":"eagle3","num_speculative_tokens":3}' \
  --mm-encoder-tp-mode data

mlx_lm.server --model mlx-community/Kimi-K2.6-mlx-DQ3_K_M-q8

File names follow that week's model-library snapshot; the definitions are in the methodology. Methodology

Model	Total params	Active params	4-bit size
Kimi Linear 48B A3BMoE	49.1B	3B	30.1 GB
Kimi K2MoE	1,000B	32B	621.3 GB
Kimi K2.5MoE	1,000B	32B	621.2 GB
Kimi K2.7 CodeMoE	1,000B	32B	583.7 GB
Kimi K3MoE	2,800B	104B	1,508.7 GB

About Kimi