How to Deploy Gemma 4 31B Locally
Gemma 4 31B is the largest and highest-scoring dense model in Gemma 4. At launch, Google said it ranked third among open models on the Arena AI text leaderboard. A 24GB card can run it at 4-bit at about 36 tokens/s, but context tops out at about 66K; the full 256K takes 48GB of VRAM. It understands images but not audio.
Gemma 4 31B details
- Min VRAM at 4-bit21 GB · 8K
- Parameters30.7B
- Max context256K
- Architecture
- Dense
- License
- Apache-2.0
- Launch date
- Apr 2, 2026
- Vendor
- Google DeepMind
- Hugging Face repo
- google/gemma-4-31B-it
- GGUF repo
- unsloth/gemma-4-31B-it-GGUF
- NVFP4 repo (Blackwell)
- nvidia/Gemma-4-31B-IT-NVFP4
- MLX repo (Apple)
- mlx-community/gemma-4-31b-it-4bit
- Ollama tag
- gemma4:31b
- Layers
- 60
- Hidden size
- 5,376
- KV heads
- 16
- Head dim
- 256
How much VRAM does Gemma 4 31B need for local deployment?
| Quantization | WeightsGB | 8K contextGB | 32K contextGB | 128K contextGB | 256K context (max)GB |
|---|---|---|---|---|---|
| 1-bit | 7.7est. | 10.3 | 12.3 | 20.4 | 31.1 |
| 2-bit | 11.8GGUF | 14.4 | 16.4 | 24.5 | 35.2 |
| 3-bit | 15.4GGUF | 18 | 20 | 28.1 | 38.8 |
| 4-bit | 18.3GGUF | 20.9 | 22.9 | 31 | 41.7 |
| 5-bit | 21.7GGUF | 24.3 | 26.3 | 34.4 | 45.1 |
| 8-bit | 32.6GGUF | 35.2 | 37.2 | 45.3 | 56 |
| BF16/FP16 | 61.4GGUF | 64 | 66 | 74.1 | 84.8 |
Full calculation rules are on the Methodology page.
Can a single GPU run Gemma 4 31B?
A single 24GB card runs it at 4-bit, but context is the weak spot. Once the 4-bit weights are loaded on an RTX 3090, 4090 or RX 7900 XTX, little memory is left, so context tops out around 66K at a little under 40 tokens/s, and long documents are exactly where it clearly beats its smaller siblings. For more room, a 32GB RTX 5090 stretches context to about 168K and runs much faster; the full 256K takes a 48GB card or larger. 16GB cards are limited to 2-bit. The 128GB unified-memory machines can hold it, but a dense model reads every weight for each token, so 4-bit crawls along at around 10 tokens/s.
| Device | Price | Speed t/s | Max context |
|---|---|---|---|
 Tesla V100 16GB16 GB | $37024 listings | 51.1 | 41K |
| GeForce RTX 5060 Ti 16GB16 GB | $86958 listings | 26.4 | 41K |
| Tesla V100 32GB32 GB | $89518 listings | 51.1 | 246K |
| Radeon RX 7900 XTX24 GB | $1,25029 listings | 54.2 | 143K |
| GeForce RTX 309024 GB | $1,799107 listings | 53 | 143K |
| GeForce RTX 409024 GB | $3,75868 listings | 56.7 | 143K |
| Ryzen AI Max+ 395 128GBWhole system · 128 GB | $4,46420 listings | 15.3 | 256K |
| RTX A600048 GB | $4,89916 listings | 44.1 | 256K |
| Mac Studio M5 Ultra 96GBWhole system · 96 GB | $7,27412 listings | 68 | 256K |
| MacBook Pro M5 Max 128GBWhole system · 128 GB | $7,54920 listings | 35.7 | 256K |
| DGX Spark 128GBWhole system · 128 GB | $8,24022 listings | 16.3 | 256K |
| GeForce RTX 509032 GB | $8,395123 listings | 95 | 246K |
| RTX PRO 600096 GB | $22,50011 listings | 95 | 256K |
| Device | Price | Speed t/s | Max context |
|---|---|---|---|
| Tesla V100 32GB32 GB | $89518 listings | 34.5 | 168K |
| Radeon RX 7900 XTX24 GB | $1,25029 listings | 36.7 | 66K |
| GeForce RTX 309024 GB | $1,799107 listings | 35.8 | 66K |
| GeForce RTX 409024 GB | $3,75868 listings | 38.4 | 66K |
| Ryzen AI Max+ 395 128GBWhole system · 128 GB | $4,46420 listings | 10.2 | 256K |
| RTX A600048 GB | $4,89916 listings | 29.6 | 256K |
| Mac Studio M5 Ultra 96GBWhole system · 96 GB | $7,27412 listings | 46.3 | 256K |
| MacBook Pro M5 Max 128GBWhole system · 128 GB | $7,54920 listings | 23.9 | 256K |
| DGX Spark 128GBWhole system · 128 GB | $8,24022 listings | 10.8 | 256K |
| GeForce RTX 509032 GB | $8,395123 listings | 65.5 | 168K |
| RTX PRO 600096 GB | $22,50011 listings | 65.5 | 256K |
| Device | Price | Speed t/s | Max context |
|---|---|---|---|
| Ryzen AI Max+ 395 128GBWhole system · 128 GB | $4,46420 listings | 5.8 | 256K |
| RTX A600048 GB | $4,89916 listings | 17.2 | 202K |
| Mac Studio M5 Ultra 96GBWhole system · 96 GB | $7,27412 listings | 27.2 | 256K |
| MacBook Pro M5 Max 128GBWhole system · 128 GB | $7,54920 listings | 13.8 | 256K |
| DGX Spark 128GBWhole system · 128 GB | $8,24022 listings | 6.2 | 256K |
| RTX PRO 600096 GB | $22,50011 listings | 39 | 256K |
Only common GPUs and systems are listed, one set for models under 50B and another for larger ones, sorted by price from low to high.
What to know before running Gemma 4 31B locally
Splitting VRAM on a 24GB card
The 4-bit weights are 18.3 GB, and going from 8K to 128K of context adds about 10.1 GB more, so a 24GB card can't reach 128K. For longer context, drop to 3-bit (15.4 GB of weights) or shrink the context cache in llama.cpp with
-ctk q8_0-ctv q8_0. Don't go up to 5-bit; on a 24GB card it tops out at about 25K.Long documents are its strength
On the 128K multi-needle retrieval test (MRCR v2) in Google's model card, 31B scores 66.4%, while 26B A4B and 12B both land around 44%. For long documents and large codebases, pick 31B.
It's also the pick for agents
On the same model card, 31B scores 76.9% on the Tau2 tool-calling test versus 68.2% for 26B A4B, a wider gap than on most tests. It also natively supports system prompts and function calling.
Frameworks
The Ollama tag is gemma4:31b. Google says vLLM, llama.cpp, MLX, SGLang and LM Studio all supported it on launch day; on a Mac, either MLX or llama.cpp works.
Launch commands
Ollama
- Windows
- macOS
- Linux
ollama run gemma4:31bllama.cpp
- NVIDIA
- AMD
- Mac
- CPU
llama-server -hf unsloth/gemma-4-31B-it-GGUF:Q4_K_M -c 8192
vLLM
- Linux
- NVIDIA
- AMD
vllm serve google/gemma-4-31B-itBlackwell GPUs (RTX 50 series, RTX PRO 6000, B200) run NVFP4 weights natively, at about 30% of the original size.
vllm serve nvidia/Gemma-4-31B-IT-NVFP4MLX
- Apple Silicon
Vision models run on mlx-vlm: on a Mac, pip install mlx-vlm first; this command downloads the MLX weights from Hugging Face and starts a server.
mlx_vlm.server --model mlx-community/gemma-4-31b-it-4bitFile names follow that week's model-library snapshot; the definitions are in the methodology. Methodology
Other models in the Gemma family
| Model | Total params | Active params | 4-bit size |
|---|---|---|---|
| Gemma 4 E4BDense | 8B | 8B | 5 GB |
| Gemma 4 12BDense | 12B | 12B | 7.1 GB |
| Gemma 4 26B A4BMoE | 25.2B | 3.8B | 16.9 GB |