How to Deploy Mistral Nemo Locally
Mistral NeMo is a 12B model that Mistral AI and NVIDIA released together in July 2024, positioned by Mistral as a drop-in replacement for Mistral 7B. Short chats are light, and at 4-bit a 12GB card is enough; but its context is heavy on VRAM, adding about 5 GB for every 32K, and the full 128K needs about 30 GB.
Mistral Nemo details
- Min VRAM at 4-bit10 GB · 8K
- Parameters12.3B
- Max context128K
- Architecture
- Dense
- License
- Apache-2.0
- Launch date
- Jul 17, 2024
- Vendor
- Mistral AI
- Hugging Face repo
- mistralai/Mistral-Nemo-Instruct-2407
- MLX repo (Apple)
- mlx-community/Mistral-Nemo-Instruct-2407-4bit
- Ollama tag
- mistral-nemo:12b
- Revision
- 2407
- Layers
- 40
- Hidden size
- 5,120
- KV heads
- 8
- Head dim
- 128
How much VRAM does Mistral Nemo need for local deployment?
| Quantization | WeightsGB | 8K contextGB | 32K contextGB | 128K context (max)GB |
|---|---|---|---|---|
| 1-bit | 3.1est. | 5.5 | 9.5 | 25.6 |
| 2-bit | 5.4GGUF | 7.8 | 11.8 | 27.9 |
| 3-bit | 7.1GGUF | 9.5 | 13.5 | 29.6 |
| 4-bit | 7.5GGUF | 9.9 | 13.9 | 30 |
| 5-bit | 8.7GGUF | 11.1 | 15.1 | 31.2 |
| 8-bit | 13GGUF | 15.4 | 19.4 | 35.5 |
| BF16/FP16 | 24.5GGUF | 26.9 | 30.9 | 47 |
Full calculation rules are on the Methodology page.
Can a single GPU run Mistral Nemo?
One card is plenty: a 12GB card runs 4-bit with 8K of context (about 9.9 GB). Context is where the memory goes, roughly 5 GB more for every extra 32K, so on a 16GB card stay at 4-bit; at 8-bit there's little room left for context. For a balance of answer quality and length, a 24GB RTX 3090 at 8-bit reaches about 69K and still feels quick, while the full 128K takes at least a 32GB RTX 5090. One thing in its favor: its newer tokenizer encodes source code about 30% more compactly than earlier Mistral models, so a given window holds more of a codebase.
| Device | Price | Speed t/s | Max context |
|---|---|---|---|
 Tesla V100 16GB16 GB | $37024 listings | 98.2 | 63K |
| GeForce RTX 5060 Ti 16GB16 GB | $86958 listings | 52.5 | 63K |
| Tesla V100 32GB32 GB | $89518 listings | 98.2 | 128K |
| Radeon RX 7900 XTX24 GB | $1,25029 listings | 103.8 | 115K |
| GeForce RTX 309024 GB | $1,799107 listings | 101.6 | 115K |
| GeForce RTX 409024 GB | $3,75868 listings | 108.2 | 115K |
| Ryzen AI Max+ 395 128GBWhole system · 128 GB | $4,46420 listings | 31 | 128K |
| RTX A600048 GB | $4,89916 listings | 85.6 | 128K |
| Mac Studio M5 Ultra 96GBWhole system · 96 GB | $7,27412 listings | 127.7 | 128K |
| MacBook Pro M5 Max 128GBWhole system · 128 GB | $7,54920 listings | 70.1 | 128K |
| DGX Spark 128GBWhole system · 128 GB | $8,24022 listings | 33 | 128K |
| GeForce RTX 509032 GB | $8,395123 listings | 172.1 | 128K |
| RTX PRO 600096 GB | $22,50011 listings | 172.1 | 128K |
| Device | Price | Speed t/s | Max context |
|---|---|---|---|
| Tesla V100 16GB16 GB | $37024 listings | 75.6 | 51K |
| GeForce RTX 5060 Ti 16GB16 GB | $86958 listings | 39.8 | 51K |
| Tesla V100 32GB32 GB | $89518 listings | 75.6 | 128K |
| Radeon RX 7900 XTX24 GB | $1,25029 listings | 80.1 | 102K |
| GeForce RTX 309024 GB | $1,799107 listings | 78.3 | 102K |
| GeForce RTX 409024 GB | $3,75868 listings | 83.6 | 102K |
| Ryzen AI Max+ 395 128GBWhole system · 128 GB | $4,46420 listings | 23.3 | 128K |
| RTX A600048 GB | $4,89916 listings | 65.6 | 128K |
| Mac Studio M5 Ultra 96GBWhole system · 96 GB | $7,27412 listings | 99.4 | 128K |
| MacBook Pro M5 Max 128GBWhole system · 128 GB | $7,54920 listings | 53.4 | 128K |
| DGX Spark 128GBWhole system · 128 GB | $8,24022 listings | 24.8 | 128K |
| GeForce RTX 509032 GB | $8,395123 listings | 136.3 | 128K |
| RTX PRO 600096 GB | $22,50011 listings | 136.3 | 128K |
| Device | Price | Speed t/s | Max context |
|---|---|---|---|
| Tesla V100 16GB16 GB | $37024 listings | 47.2 | 18K |
| GeForce RTX 5060 Ti 16GB16 GB | $86958 listings | 24.3 | 18K |
| Tesla V100 32GB32 GB | $89518 listings | 47.2 | 120K |
| Radeon RX 7900 XTX24 GB | $1,25029 listings | 50.1 | 69K |
| GeForce RTX 309024 GB | $1,799107 listings | 48.9 | 69K |
| GeForce RTX 409024 GB | $3,75868 listings | 52.4 | 69K |
| Ryzen AI Max+ 395 128GBWhole system · 128 GB | $4,46420 listings | 14.1 | 128K |
| RTX A600048 GB | $4,89916 listings | 40.7 | 128K |
| Mac Studio M5 Ultra 96GBWhole system · 96 GB | $7,27412 listings | 62.9 | 128K |
| MacBook Pro M5 Max 128GBWhole system · 128 GB | $7,54920 listings | 32.9 | 128K |
| DGX Spark 128GBWhole system · 128 GB | $8,24022 listings | 15 | 128K |
| GeForce RTX 509032 GB | $8,395123 listings | 88.2 | 120K |
| RTX PRO 600096 GB | $22,50011 listings | 88.2 | 128K |
Only common GPUs and systems are listed, one set for models under 50B and another for larger ones, sorted by price from low to high.
What to know before running Mistral Nemo locally
Turn the temperature down
Mistral specifically notes that NeMo needs a lower
temperaturethan earlier Mistral models and recommendstemperature0.3. Many chat front ends default to 0.7 or higher, so remember to change it by hand.Which quantization to pick
Mistral says the model was trained with quantization in mind, so FP8 inference loses no accuracy. The GGUFs are about 7.5 GB at 4-bit, 8.7 GB at 5-bit and 13.0 GB at 8-bit: use 4-bit or 5-bit on a 12GB card, and step up to 8-bit at 16GB or more.
Fewer tokens for Chinese
NeMo switched to the new Tekken tokenizer, which Mistral says compresses Chinese and code about 30% more efficiently than before, meaning the same 128K of context holds more Chinese text.
Which framework to use
llama.cpp and vLLM support it directly, and the Ollama tag is mistral-nemo:12b. It uses a standard architecture, so a toolchain that already runs Mistral 7B needs little or no change.
Launch commands
Ollama
- Windows
- macOS
- Linux
ollama run mistral-nemo:12bllama.cpp
- NVIDIA
- AMD
- Mac
- CPU
llama-server -hf bartowski/Mistral-Nemo-Instruct-2407-GGUF:Q4_K_M \ -c 8192
vLLM
- Linux
- NVIDIA
- AMD
vllm serve mistralai/Mistral-Nemo-Instruct-2407MLX
- Apple Silicon
On a Mac, pip install mlx-lm first; this command downloads the MLX weights from Hugging Face and starts an OpenAI-compatible server.
mlx_lm.server --model mlx-community/Mistral-Nemo-Instruct-2407-4bitFile names follow that week's model-library snapshot; the definitions are in the methodology. Methodology
Other models in the Mistral family
| Model | Total params | Active params | 4-bit size |
|---|---|---|---|
| Mistral 7B v0.3Dense | 7.3B | 7.3B | 4.4 GB |
| Ministral 3 8BDense | 8.4B | 8.4B | 5.2 GB |
| Ministral 3 14BDense | 13.5B | 13.5B | 8.2 GB |
| Mistral Small 3.2 24BDense | 24B | 24B | 14.3 GB |