VRAMGlass contains affiliate links. As an eBay Partner, we may be compensated if you make a purchase.As an eBay Partner, we may be compensated.Learn more

GPUs tracked57Price history49 daysPrice points (24h)43,400 pointsBiggest 24h dropGeForce RTX 3060 12GB-11.1%Biggest 24h riseCMP 170HX 40GB+19.8%Updated
$ USD

How to Deploy Kimi K2 Locally

Kimi K2 is Moonshot AI's 1T-parameter MoE, and the version on this site is 0905 from September 2025: instant chat only, with no thinking, and text only. At 4-bit it is about 621 GB, far beyond any single GPU or 128GB machine. For a new deployment, look at the later versions instead: for coding and agents, use K2.7 Code, whose official weights are already 4-bit, so quantization adds almost no extra loss. K2's official weights are FP8, so squeezing them to 4-bit is genuinely lossy.

Kimi K2 details

  • Min VRAM at 4-bit623 GB · 8K
  • Parameters1,000B
  • Active params32B
  • Max context256K
Architecture
MoE
Experts offloaded
10 GB VRAM + 615 GB RAM
License
Modified MIT
Launch date
Sep 3, 2025
Vendor
Moonshot AI
Hugging Face repo
moonshotai/Kimi-K2-Instruct-0905
GGUF repo
unsloth/Kimi-K2-Instruct-0905-GGUF
MLX repo (Apple)
mlx-community/Kimi-K2-Instruct-0905-mlx-3bit
Revision
0905
Layers
61
Hidden size
7,168
KV heads
64
Head dim
192

How much VRAM does Kimi K2 need for local deployment?

QuantizationWeightsGB8K contextGB32K contextGB128K contextGB256K context (max)GB
1-bit304.8GGUF306.4308.2315.1324.3
2-bit383.6GGUF385.2387393.9403.1
3-bit452.4GGUF454455.8462.7471.9
4-bit621.3GGUF622.9624.7631.6640.8
5-bit728.7GGUF730.3732.1739748.2
8-bit1,091.1GGUF1,092.71,094.51,101.41,110.6
BF16/FP162,053.2GGUF2,054.82,056.62,063.52,072.7

Full calculation rules are on the Methodology page.

Can a single GPU run Kimi K2?

No. Single GPUs and 128GB machines are nowhere close: even at 2-bit, 1T parameters come to about 384 GB. The only machine that holds 2-bit is the 512GB Mac Studio M3 Ultra, and it's pressed right up against the three-quarters of memory macOS lets the GPU use by default, so raise that limit first. The more common route is a server with 384GB of RAM and one RTX PRO 6000, with most experts in system memory, which runs 2-bit at around 40 tokens/s. Also note that K2's official weights are 8-bit, so both 2-bit and 4-bit are genuinely lossy, unlike later versions.

eBay
DevicePriceSpeed t/sMax context
Mac Studio M3 Ultra 512GBWhole system · 512 GB$22,90013 listings40.5256K

Only common GPUs and systems are listed, one set for models under 50B and another for larger ones, sorted by price from low to high.

How many GPUs do you need to run Kimi K2?

You need eight 96GB RTX PRO 6000 cards: about 69 tokens/s at 2-bit and about 50 at 4-bit. K2's 2-bit is noticeably bigger than K2.6's, so with 32K of context it just misses on four 96GB cards and on eight 48GB cards, leaving no smaller option. Four networked DGX Sparks can hold 2-bit, but only manage about 14 tokens/s. If four 96GB cards are what you have, K2.6 is the better pick: same 1T architecture, its 2-bit fits on four cards, and its instant mode is the closest match to how K2 is used.

eBay
DeviceCardsTotal priceTotal VRAMSpeed t/sTotal power
Mac Studio M3 Ultra 512GB512 GB per unit2 units$45,8001,024 GB26.2960 W
RTX PRO 600096 GB per card8 cards$180,000768 GB504,800 W

Mac Studios link up over Thunderbolt 5 and run with MLX or exo; vLLM doesn't support them. Full calculation rules are on the Methodology page.

What PC or server build do you need to run Kimi K2 locally?

For expert offloading, the only setup that works is a 12-channel DDR5 server with one RTX PRO 6000: 2-bit with 384GB of RAM does about 41 tokens/s, and 4-bit, which needs every slot filled for 768GB, about 26. KTransformers, recommended on the model card, is built specifically for this kind of inference split between CPU and GPU.

eBay

Tesla V100 32GB + Dual Xeon E5 8-GPU server

Budget
Generation speed5.9t/s
Memory bandwidth137GB/s
GPU
Tesla V100 32GB2 / 61 expert layers in 32 GB VRAM
$895
RAM
768 GB DDR4-213324 × 32 GB sticks
$3,912
CPU
2 × Intel Xeon E5-2680 v428 cores / 56 threads · 8-channel in total
$38
Barebone
Supermicro SYS-4028GR-TR8 GPU slots · 24 DIMM slots
$1,300
PSU
Built-in 4 × 1,600 W
Included
Reference build price$6,145

GeForce RTX 5090 + Dual EPYC DDR5

Flagship
Generation speed30.1t/s
Memory bandwidth922GB/s
GPU
GeForce RTX 50902 / 61 expert layers in 32 GB VRAM
$8,395
RAM
768 GB DDR5-480024 × 32 GB sticks
$29,328
CPU
2 × AMD EPYC 935464 cores / 128 threads · 24-channel in total
$3,386
Board
Gigabyte MZ73-LM024 DIMM slots
$1,300
PSU
Seasonic PRIME PX-16001,600 W
$516
Reference build price$42,925

4 × RTX PRO 6000 + Dual EPYC DDR5

Giant models
Generation speed40.8t/s
Memory bandwidth922GB/s
GPU
4 × RTX PRO 600039 / 61 expert layers in 384 GB VRAM
$90,000
RAM
384 GB DDR5-480024 × 16 GB sticks
$14,328
CPU
2 × AMD EPYC 935464 cores / 128 threads · 24-channel in total
$3,386
Board
Gigabyte MZ73-LM024 DIMM slots
$1,300
PSU
2 × Seasonic PRIME PX-22004,400 W
$1,320
Reference build price$110,334

Detailed calculation rules are on the Methodology page.

As an eBay Partner, VRAMGlass may be compensated if you buy through the eBay links on this page. The price you pay stays the same.

What to know before running Kimi K2 locally

  1. Which version to deploy now

    For coding and agents, go straight to K2.7 Code: the same 1T architecture, with official weights that are natively 4-bit (INT4), so quantization adds almost no extra loss. For instant answers without thinking, the instant mode of K2.6 is the closest to how K2 is used. The main reason to keep K2 is that your existing workflows and prompts were tuned for it.

  2. Where to get GGUFs

    The llama-server command on this page uses Unsloth's 0905 GGUF (unsloth/Kimi-K2-Instruct-0905-GGUF), available from 1-bit to 8-bit, and llama.cpp runs it directly. Moonshot's recommended engines are vLLM, SGLang, KTransformers and TensorRT-LLM.

  3. What 0905 improves over the original

    In Moonshot's tests, SWE-bench Verified for coding rose from 65.8 on the 0711 version to 69.2, and Terminal-Bench for terminal tasks from 37.5 to 44.5; front-end code is more polished and practical, and context doubled from 128K to 256K.

  4. Sampling parameters

    Moonshot recommends temperature 0.6; for the system prompt, the default line given on the model card is fine.

  5. License

    A modified MIT license with just one extra clause: commercial products with more than 100 million monthly active users or very high monthly revenue must prominently display “Kimi K2” in their interface. K2.5 through K2.7 Code keep the same terms, with the displayed name changed to each version's own.

Launch commands

llama.cpp

  • NVIDIA
  • AMD
  • Mac
  • CPU
All on GPU
Terminal
llama-server -hf unsloth/Kimi-K2-Instruct-0905-GGUF:Q4_K_M -c 8192
MoE offloading

Short on VRAM? Add --cpu-moe to keep the routed experts in system RAM (about 615 GB needed).

Terminal
llama-server -hf unsloth/Kimi-K2-Instruct-0905-GGUF:Q4_K_M -c 8192 \
  -ngl 99 --cpu-moe

vLLM

  • Linux
  • NVIDIA

Follow the deployment guide to prepare the required runtime and hardware.

Terminal
vllm serve moonshotai/Kimi-K2-Instruct-0905 \
  --port 8000 \
  --served-model-name kimi-k2 \
  --trust-remote-code \
  --tensor-parallel-size 16 \
  --enable-auto-tool-choice \
  --tool-call-parser kimi_k2

MLX

  • Apple Silicon

On a Mac, pip install mlx-lm first; this command downloads the MLX weights from Hugging Face and starts an OpenAI-compatible server.

Terminal
mlx_lm.server --model mlx-community/Kimi-K2-Instruct-0905-mlx-3bit

Ollama

  • Windows
  • macOS
  • Linux

The Ollama library has no official tag for it yet.

File names follow that week's model-library snapshot; the definitions are in the methodology. Methodology

Other models in the Kimi family

ModelTotal paramsActive params4-bit size
Kimi Linear 48B A3BMoE49.1B3B30.1 GB
Kimi K2.5MoE1,000B32B621.2 GB
Kimi K2.6MoE1,000B32B583.7 GB
Kimi K2.7 CodeMoE1,000B32B583.7 GB
Kimi K3MoE2,800B104B1,508.7 GB

About Kimi