How to Deploy Mistral 7B v0.3 Locally
Mistral 7B v0.3 is a 7B model that Mistral released in May 2024, and this version added function calling. At 4-bit, an 8GB card can open 8K of context, and a 12GB card can run the full 32K. Its problem is age: Ministral 3 8B from the same family needs only a little more VRAM but has 8x the context, and it understands images too.
Mistral 7B v0.3 details
- Min VRAM at 4-bit7 GB · 8K
- Parameters7.3B
- Max context32K
- Architecture
- Dense
- License
- Apache-2.0
- Launch date
- May 22, 2024
- Vendor
- Mistral AI
- Hugging Face repo
- mistralai/Mistral-7B-Instruct-v0.3
- MLX repo (Apple)
- mlx-community/Mistral-7B-Instruct-v0.3-4bit
- Ollama tag
- mistral:7b
- Layers
- 32
- Hidden size
- 4,096
- KV heads
- 8
- Head dim
- 128
How much VRAM does Mistral 7B v0.3 need for local deployment?
| Quantization | WeightsGB | 8K contextGB | 32K context (max)GB |
|---|---|---|---|
| 1-bit | 1.8GGUF | 3.9 | 7.2 |
| 2-bit | 2.5GGUF | 4.6 | 7.9 |
| 3-bit | 3.8GGUF | 5.9 | 9.2 |
| 4-bit | 4.4GGUF | 6.5 | 9.8 |
| 5-bit | 5.1GGUF | 7.2 | 10.5 |
| 8-bit | 7.7GGUF | 9.8 | 13.1 |
| BF16/FP16 | 15.2est. | 17.4 | 20.6 |
Full calculation rules are on the Methodology page.
Can a single GPU run Mistral 7B v0.3?
Easily, and context won't be what holds you back: the model tops out at 32K, and 4-bit at the full 32K stays under 10 GB, so even an 8GB card works with around 8K. That leaves only the quant to choose. With 16GB or more, go straight to 8-bit; the full 32K comes to about 13.1 GB with almost no loss in answer quality. A 12GB card can still run 8-bit if you keep context near 8K. Speed tracks memory bandwidth: 8-bit runs close to 80 tokens/s on an RTX 3090 and about half that on an RTX 5060 Ti 16GB, which is still plenty for chat.
| Device | Price | Speed t/s | Max context |
|---|---|---|---|
 Tesla V100 16GB16 GB | $37024 listings | 172.9 | 32K |
| GeForce RTX 5060 Ti 16GB16 GB | $86958 listings | 98 | 32K |
| Tesla V100 32GB32 GB | $89518 listings | 172.9 | 32K |
| Radeon RX 7900 XTX24 GB | $1,25029 listings | 181.5 | 32K |
| GeForce RTX 309024 GB | $1,799107 listings | 178.1 | 32K |
| GeForce RTX 409024 GB | $3,75868 listings | 188.1 | 32K |
| Ryzen AI Max+ 395 128GBWhole system · 128 GB | $4,46420 listings | 59.5 | 32K |
| RTX A600048 GB | $4,89916 listings | 152.9 | 32K |
| Mac Studio M5 Ultra 96GBWhole system · 96 GB | $7,27412 listings | 216.8 | 32K |
| MacBook Pro M5 Max 128GBWhole system · 128 GB | $7,54920 listings | 127.8 | 32K |
| DGX Spark 128GBWhole system · 128 GB | $8,24022 listings | 63.1 | 32K |
| GeForce RTX 509032 GB | $8,395123 listings | 277.6 | 32K |
| RTX PRO 600096 GB | $22,50011 listings | 277.6 | 32K |
| Device | Price | Speed t/s | Max context |
|---|---|---|---|
| Tesla V100 16GB16 GB | $37024 listings | 117.1 | 32K |
| GeForce RTX 5060 Ti 16GB16 GB | $86958 listings | 63.5 | 32K |
| Tesla V100 32GB32 GB | $89518 listings | 117.1 | 32K |
| Radeon RX 7900 XTX24 GB | $1,25029 listings | 123.6 | 32K |
| GeForce RTX 309024 GB | $1,799107 listings | 121 | 32K |
| GeForce RTX 409024 GB | $3,75868 listings | 128.6 | 32K |
| Ryzen AI Max+ 395 128GBWhole system · 128 GB | $4,46420 listings | 37.7 | 32K |
| RTX A600048 GB | $4,89916 listings | 102.4 | 32K |
| Mac Studio M5 Ultra 96GBWhole system · 96 GB | $7,27412 listings | 150.9 | 32K |
| MacBook Pro M5 Max 128GBWhole system · 128 GB | $7,54920 listings | 84.3 | 32K |
| DGX Spark 128GBWhole system · 128 GB | $8,24022 listings | 40.1 | 32K |
| GeForce RTX 509032 GB | $8,395123 listings | 200.6 | 32K |
| RTX PRO 600096 GB | $22,50011 listings | 200.6 | 32K |
| Device | Price | Speed t/s | Max context |
|---|---|---|---|
| Tesla V100 16GB16 GB | $37024 listings | 75.1 | 32K |
| GeForce RTX 5060 Ti 16GB16 GB | $86958 listings | 39.4 | 32K |
| Tesla V100 32GB32 GB | $89518 listings | 75.1 | 32K |
| Radeon RX 7900 XTX24 GB | $1,25029 listings | 79.5 | 32K |
| GeForce RTX 309024 GB | $1,799107 listings | 77.7 | 32K |
| GeForce RTX 409024 GB | $3,75868 listings | 83 | 32K |
| Ryzen AI Max+ 395 128GBWhole system · 128 GB | $4,46420 listings | 23.1 | 32K |
| RTX A600048 GB | $4,89916 listings | 65.1 | 32K |
| Mac Studio M5 Ultra 96GBWhole system · 96 GB | $7,27412 listings | 98.7 | 32K |
| MacBook Pro M5 Max 128GBWhole system · 128 GB | $7,54920 listings | 53 | 32K |
| DGX Spark 128GBWhole system · 128 GB | $8,24022 listings | 24.6 | 32K |
| GeForce RTX 509032 GB | $8,395123 listings | 135.4 | 32K |
| RTX PRO 600096 GB | $22,50011 listings | 135.4 | 32K |
Only common GPUs and systems are listed, one set for models under 50B and another for larger ones, sorted by price from low to high.
What to know before running Mistral 7B v0.3 locally
What v0.3 changed from v0.2
v0.3 expanded the vocabulary (32768 tokens, with the new v3 tokenizer) and added function calling. 32K context was already there in v0.2; v0.1 had only 8K. For tool calling, note that Mistral requires tool call IDs to be exactly 9 letters or digits, an easy place to slip up when you build the conversation history yourself.
Which framework to use
llama.cpp and vLLM support it directly, and Ollama's mistral:7b is v0.3. Mistral officially recommends mistral-inference; function calling with transformers requires 4.42.0 or later.
Which quantization to pick
4-bit is about 4.4 GB, 5-bit about 5.1 GB and 8-bit about 7.7 GB, only a few GB apart, so use 8-bit if you can. Only on old cards with less than 6GB is it worth considering 3-bit (about 3.8 GB) or lower.
No content moderation
Mistral states that this model has no moderation mechanisms, so any public-facing service needs its own input and output filtering.
Launch commands
Ollama
- Windows
- macOS
- Linux
ollama run mistral:7bllama.cpp
- NVIDIA
- AMD
- Mac
- CPU
llama-server -hf bartowski/Mistral-7B-Instruct-v0.3-GGUF:Q4_K_M -c 8192
vLLM
- Linux
- NVIDIA
- AMD
vllm serve mistralai/Mistral-7B-Instruct-v0.3MLX
- Apple Silicon
On a Mac, pip install mlx-lm first; this command downloads the MLX weights from Hugging Face and starts an OpenAI-compatible server.
mlx_lm.server --model mlx-community/Mistral-7B-Instruct-v0.3-4bitFile names follow that week's model-library snapshot; the definitions are in the methodology. Methodology
Other models in the Mistral family
| Model | Total params | Active params | 4-bit size |
|---|---|---|---|
| Ministral 3 8BDense | 8.4B | 8.4B | 5.2 GB |
| Mistral NemoDense | 12.3B | 12.3B | 7.5 GB |
| Ministral 3 14BDense | 13.5B | 13.5B | 8.2 GB |
| Mistral Small 3.2 24BDense | 24B | 24B | 14.3 GB |