How to Deploy Gemma 4 12B Locally

• Min VRAM at 4-bit9 GB · 8K
• Parameters12B
• Max context256K

Architecture

Dense

License

Apache-2.0

Launch date

Jun 3, 2026

Vendor

Google DeepMind

Hugging Face repo

google/gemma-4-12B-it

GGUF repo

unsloth/gemma-4-12b-it-GGUF

NVFP4 repo (Blackwell)

RedHatAI/gemma-4-12B-it-NVFP4

MLX repo (Apple)

mlx-community/gemma-4-12B-it-4bit

Ollama tag

gemma4:12b

Layers

48

Hidden size

3,840

KV heads

8

Head dim

256

Full calculation rules are on the Methodology page.

Device	Price	Speed t/s	Max context
Tesla V100 16GB16 GB	$37024 listings	83.2	256K
GeForce RTX 5060 Ti 16GB16 GB	$86958 listings	44	256K
Tesla V100 32GB32 GB	$89518 listings	83.2	256K
Radeon RX 7900 XTX24 GB	$1,25029 listings	88	256K
GeForce RTX 309024 GB	$1,799107 listings	86.1	256K
GeForce RTX 409024 GB	$3,75868 listings	91.9	256K
Ryzen AI Max+ 395 128GBWhole system · 128 GB	$4,46420 listings	25.8	256K
RTX A600048 GB	$4,89916 listings	72.2	256K
Mac Studio M5 Ultra 96GBWhole system · 96 GB	$7,27412 listings	108.9	256K
MacBook Pro M5 Max 128GBWhole system · 128 GB	$7,54920 listings	58.9	256K
DGX Spark 128GBWhole system · 128 GB	$8,24022 listings	27.5	256K
GeForce RTX 509032 GB	$8,395123 listings	148.5	256K
RTX PRO 600096 GB	$22,50011 listings	148.5	256K

Only common GPUs and systems are listed, one set for models under 50B and another for larger ones, sorted by price from low to high.

1. 1

   What having no encoder means

   Other multimodal models first use a separate encoder to turn images and sound into vectors, then hand those to the language model. Gemma 4 12B skips that step: images go through one simple projection into the main model, audio is projected directly into the same dimensions as text, and everything runs through the same decoder. Google says this lowers latency and lets the whole model be fine-tuned in one go. Audio is likewise limited to 30 seconds.

2. 2

   Which quantization to pick

   On a 12GB card, use 4-bit (7.1 GB of weights) or 5-bit (8.4 GB); on a 16GB card, use 8-bit (12.7 GB). The original FP16 with 8K already needs 25.3 GB, twice as much as 8-bit, so it can't take much context; you're better off running 8-bit with a long context.

3. 3

   Thinking toggle and multi-turn chat

   Adding <|think|> at the start of the system prompt turns thinking on. With it off, the model still outputs an empty thinking block, which your parser needs to filter out. In multi-turn chats, don't put earlier thinking back into the history; keep only the final answers, except for turns with tool calls. Google uses temperature 1.0, top_p 0.95 and top_k 64 for every use case.

4. 4

   Why long context is so light

   Most of its layers only look at the most recent 1024 tokens, and just a few layers see the full context, so VRAM grows slowly as context gets longer: at 4-bit, going from 8K to 256K adds only about 4.2 GB.

ollama run gemma4:12b

llama-server -hf unsloth/gemma-4-12b-it-GGUF:Q4_K_M -c 8192

vllm serve google/gemma-4-12B-it

vllm serve RedHatAI/gemma-4-12B-it-NVFP4

mlx_vlm.server --model mlx-community/gemma-4-12B-it-4bit

File names follow that week's model-library snapshot; the definitions are in the methodology. Methodology

Model	Total params	Active params	4-bit size
Gemma 4 E4BDense	8B	8B	5 GB
Gemma 4 26B A4BMoE	25.2B	3.8B	16.9 GB
Gemma 4 31BDense	30.7B	30.7B	18.3 GB

About Gemma