How to Deploy Mistral Small 3.2 24B Locally

• Min VRAM at 4-bit17 GB · 8K
• Parameters24B
• Max context128K

Architecture

Dense

License

Apache-2.0

Launch date

Jun 19, 2025

Vendor

Mistral AI

Hugging Face repo

mistralai/Mistral-Small-3.2-24B-Instruct-2506

GGUF repo

bartowski/mistralai_Mistral-Small-3.2-24B-Instruct-2506-GGUF

NVFP4 repo (Blackwell)

RedHatAI/Mistral-Small-3.2-24B-Instruct-2506-NVFP4

MLX repo (Apple)

mlx-community/Mistral-Small-3.2-24B-Instruct-2506-4bit

Ollama tag

mistral-small3.2:24b

Revision

2506

Layers

40

Hidden size

5,120

KV heads

8

Head dim

128

Full calculation rules are on the Methodology page.

Device	Price	Speed t/s	Max context
Tesla V100 16GB16 GB	$37024 listings	43.3	10K
GeForce RTX 5060 Ti 16GB16 GB	$86958 listings	22.2	10K
Tesla V100 32GB32 GB	$89518 listings	43.3	113K
Radeon RX 7900 XTX24 GB	$1,25029 listings	46	61K
GeForce RTX 309024 GB	$1,799107 listings	45	61K
GeForce RTX 409024 GB	$3,75868 listings	48.2	61K
Ryzen AI Max+ 395 128GBWhole system · 128 GB	$4,46420 listings	12.9	128K
RTX A600048 GB	$4,89916 listings	37.3	128K
Mac Studio M5 Ultra 96GBWhole system · 96 GB	$7,27412 listings	57.9	128K
MacBook Pro M5 Max 128GBWhole system · 128 GB	$7,54920 listings	30.1	128K
DGX Spark 128GBWhole system · 128 GB	$8,24022 listings	13.7	128K
GeForce RTX 509032 GB	$8,395123 listings	81.4	113K
RTX PRO 600096 GB	$22,50011 listings	81.4	128K

Only common GPUs and systems are listed, one set for models under 50B and another for larger ones, sorted by price from low to high.

1. 1

   Don't pull the wrong Ollama tag

   In Ollama, mistral-small:24b is the earlier Small 3, which handles text only with 32K of context. The Small 3.2 tag is mistral-small3.2:24b, which supports images and 128K of context.

2. 2

   Temperature and system prompt

   Mistral recommends temperature 0.15 and says you should always include a system prompt; for a general-purpose assistant, just reuse SYSTEM_PROMPT.txt from the model repository.

3. 3

   Frameworks and images

   Mistral recommends vLLM 0.9.1 or later, but the original BF16 needs about 55 GB on a GPU, so a single card can basically only run a quantized version. For images in llama.cpp, you need to load an mmproj file as well, which bartowski's GGUF repository includes.

4. 4

   3.2 or 3.1

   The two share the same architecture and size, and 3.2 is better across the board: on the instruction-following test WildBench v2 it rose from 55.6% to 65.3%, endless repetitive output in Mistral's internal tests fell from 2.11% to 1.29%, and tool calling is more reliable. Vision scores went up on some tests and down on others, roughly a wash. There's no reason to deploy 3.1 anymore.

5. 5

   The next generation became a big MoE

   Small 4, which Mistral released in March 2026, switched to a 119B-parameter MoE (6B used per token) that doesn't fit on a single 24GB card. If you want to run a Mistral dense model on one 24GB card, Small 3.2 is still the largest option.

ollama run mistral-small3.2:24b

llama-server \
  -hf bartowski/mistralai_Mistral-Small-3.2-24B-Instruct-2506-GGUF:Q4_K_M \
  -c 8192

vllm serve mistralai/Mistral-Small-3.2-24B-Instruct-2506

vllm serve RedHatAI/Mistral-Small-3.2-24B-Instruct-2506-NVFP4

mlx_lm.server \
  --model mlx-community/Mistral-Small-3.2-24B-Instruct-2506-4bit

File names follow that week's model-library snapshot; the definitions are in the methodology. Methodology

Model	Total params	Active params	4-bit size
Mistral 7B v0.3Dense	7.3B	7.3B	4.4 GB
Ministral 3 8BDense	8.4B	8.4B	5.2 GB
Mistral NemoDense	12.3B	12.3B	7.5 GB
Ministral 3 14BDense	13.5B	13.5B	8.2 GB

About Mistral