How to Deploy Llama 3.2 3B Locally
Llama 3.2 3B is a small 3B model that Meta released in September 2024; at 4-bit with 8K it takes about 4 GB, so almost any graphics card can run it. It was trained with Llama 3.1 8B and 70B as teachers (distillation), which suits it to light tasks like query rewriting, summarization and tool calling, or to serving as a draft model for the larger models in its family. For Chinese chat or image understanding, choose one of 2026's new small models.
Llama 3.2 3B details
- Min VRAM at 4-bit5 GB · 8K
- Parameters3.2B
- Max context128K
- Architecture
- Dense
- License
- Llama 3.2 Community License
- Launch date
- Sep 25, 2024
- Vendor
- Meta AI
- Hugging Face repo
- meta-llama/Llama-3.2-3B-Instruct
- GGUF repo
- bartowski/Llama-3.2-3B-Instruct-GGUF
- MLX repo (Apple)
- mlx-community/Llama-3.2-3B-Instruct-4bit
- Ollama tag
- llama3.2:3b
- Layers
- 28
- Hidden size
- 3,072
- KV heads
- 8
- Head dim
- 128
How much VRAM does Llama 3.2 3B need for local deployment?
| Quantization | WeightsGB | 8K contextGB | 32K contextGB | 128K context (max)GB |
|---|---|---|---|---|
| 1-bit | 0.8est. | 2.8 | 5.6 | 16.9 |
| 2-bit | 1.3est. | 3.3 | 6.1 | 17.4 |
| 3-bit | 1.9GGUF | 3.9 | 6.7 | 18 |
| 4-bit | 2GGUF | 4 | 6.8 | 18.1 |
| 5-bit | 2.3GGUF | 4.3 | 7.1 | 18.4 |
| 8-bit | 3.4GGUF | 5.4 | 8.2 | 19.5 |
| BF16/FP16 | 6.4GGUF | 8.4 | 11.2 | 22.5 |
Full calculation rules are on the Methodology page.
Can a single GPU run Llama 3.2 3B?
Any card with 8GB or more runs it comfortably; even 8-bit with 8K of context needs just 5.4 GB. Since memory isn't tight, don't squeeze it down to 4-bit. Small models lose more from quantization, and Meta's own testing showed a clear score drop when the 3B went straight to 4-bit, so 8-bit or the unquantized original is the right choice here. Context is what actually eats memory: the full 128K needs close to 20 GB, several times the model itself. A 16GB card at 8-bit gets a little past 100K, and a 24GB card takes the whole window. Speed is a non-issue, since even the slowest 128GB unified-memory machine does about 50 tokens/s at 8-bit.
| Device | Price | Speed t/s | Max context |
|---|---|---|---|
 Tesla V100 16GB16 GB | $37024 listings | 254.9 | 126K |
| GeForce RTX 5060 Ti 16GB16 GB | $86958 listings | 154.6 | 126K |
| Tesla V100 32GB32 GB | $89518 listings | 254.9 | 128K |
| Radeon RX 7900 XTX24 GB | $1,25029 listings | 265.6 | 128K |
| GeForce RTX 309024 GB | $1,799107 listings | 261.4 | 128K |
| GeForce RTX 409024 GB | $3,75868 listings | 273.8 | 128K |
| Ryzen AI Max+ 395 128GBWhole system · 128 GB | $4,46420 listings | 97.4 | 128K |
| RTX A600048 GB | $4,89916 listings | 229.6 | 128K |
| Mac Studio M5 Ultra 96GBWhole system · 96 GB | $7,27412 listings | 307.9 | 128K |
| MacBook Pro M5 Max 128GBWhole system · 128 GB | $7,54920 listings | 196.2 | 128K |
| DGX Spark 128GBWhole system · 128 GB | $8,24022 listings | 102.9 | 128K |
| GeForce RTX 509032 GB | $8,395123 listings | 375 | 128K |
| RTX PRO 600096 GB | $22,50011 listings | 375 | 128K |
| Device | Price | Speed t/s | Max context |
|---|---|---|---|
| Tesla V100 16GB16 GB | $37024 listings | 201.4 | 120K |
| GeForce RTX 5060 Ti 16GB16 GB | $86958 listings | 116.8 | 120K |
| Tesla V100 32GB32 GB | $89518 listings | 201.4 | 128K |
| Radeon RX 7900 XTX24 GB | $1,25029 listings | 210.9 | 128K |
| GeForce RTX 309024 GB | $1,799107 listings | 207.2 | 128K |
| GeForce RTX 409024 GB | $3,75868 listings | 218.2 | 128K |
| Ryzen AI Max+ 395 128GBWhole system · 128 GB | $4,46420 listings | 71.8 | 128K |
| RTX A600048 GB | $4,89916 listings | 179.3 | 128K |
| Mac Studio M5 Ultra 96GBWhole system · 96 GB | $7,27412 listings | 249.3 | 128K |
| MacBook Pro M5 Max 128GBWhole system · 128 GB | $7,54920 listings | 150.9 | 128K |
| DGX Spark 128GBWhole system · 128 GB | $8,24022 listings | 76 | 128K |
| GeForce RTX 509032 GB | $8,395123 listings | 313.4 | 128K |
| RTX PRO 600096 GB | $22,50011 listings | 313.4 | 128K |
| Device | Price | Speed t/s | Max context |
|---|---|---|---|
| Tesla V100 16GB16 GB | $37024 listings | 143 | 108K |
| GeForce RTX 5060 Ti 16GB16 GB | $86958 listings | 79.1 | 108K |
| Tesla V100 32GB32 GB | $89518 listings | 143 | 128K |
| Radeon RX 7900 XTX24 GB | $1,25029 listings | 150.5 | 128K |
| GeForce RTX 309024 GB | $1,799107 listings | 147.5 | 128K |
| GeForce RTX 409024 GB | $3,75868 listings | 156.4 | 128K |
| Ryzen AI Max+ 395 128GBWhole system · 128 GB | $4,46420 listings | 47.5 | 128K |
| RTX A600048 GB | $4,89916 listings | 125.7 | 128K |
| Mac Studio M5 Ultra 96GBWhole system · 96 GB | $7,27412 listings | 181.9 | 128K |
| MacBook Pro M5 Max 128GBWhole system · 128 GB | $7,54920 listings | 104.2 | 128K |
| DGX Spark 128GBWhole system · 128 GB | $8,24022 listings | 50.4 | 128K |
| GeForce RTX 509032 GB | $8,395123 listings | 237.5 | 128K |
| RTX PRO 600096 GB | $22,50011 listings | 237.5 | 128K |
Only common GPUs and systems are listed, one set for models under 50B and another for larger ones, sorted by price from low to high.
What to know before running Llama 3.2 3B locally
Use 8-bit if you have the VRAM
On a PC, VRAM isn't an issue, so go straight to 8-bit (3.4 GB of weights) or FP16 (6.4 GB). Small models are more sensitive to quantization: Meta found that compressing straight to 4-bit without extra training dropped MMLU from 63.4 to 60.5 (that version was only for comparison and wasn't released). GGUF's Q4_K_M uses a different method, but the same principle applies.
Meta's official phone versions
Meta also released two official 4-bit versions, SpinQuant and QLoRA, built to run on phone Arm CPUs (via ExecuTorch), with context limited to 8K. They score 62 and 62.4 on MMLU respectively, losing less than direct quantization. On a PC, just use GGUF.
As a draft model for bigger models
It shares the same vocabulary as Llama 3.1 8B and Llama 3.3 70B, so you can load it in llama.cpp with
-mdas a speculative-decoding draft model for the larger one: the small model guesses a few tokens ahead and the large model checks them in one pass, which speeds up generation.Tool calling and sampling parameters
llama.cpp natively supports Llama 3.2's tool-calling format; just add
--jinjawhen starting llama-server. In Meta's tests, the 3B's tool-calling score (BFCL V2 67.0) is nearly identical to Llama 3.1 8B's (67.1). Use Meta's sampling parameters:temperature0.6 andtop_p0.9.
Launch commands
Ollama
- Windows
- macOS
- Linux
ollama run llama3.2:3bllama.cpp
- NVIDIA
- AMD
- Mac
- CPU
llama-server -hf bartowski/Llama-3.2-3B-Instruct-GGUF:Q4_K_M -c 8192
vLLM
- Linux
- NVIDIA
- AMD
vllm serve meta-llama/Llama-3.2-3B-InstructMLX
- Apple Silicon
On a Mac, pip install mlx-lm first; this command downloads the MLX weights from Hugging Face and starts an OpenAI-compatible server.
mlx_lm.server --model mlx-community/Llama-3.2-3B-Instruct-4bitFile names follow that week's model-library snapshot; the definitions are in the methodology. Methodology
Other models in the Meta family
| Model | Total params | Active params | 4-bit size |
|---|---|---|---|
| Llama 3.1 8BDense | 8B | 8B | 4.9 GB |
| Llama 3.3 70BDense | 70.6B | 70.6B | 42.5 GB |
| Llama 4 Scout 17B 16EMoE | 108.6B | 17B | 65.4 GB |
| Muse Spark OpenMoE | 816B | 42B | 505.5 GB |