How to Deploy Qwen3 8B Locally

• Min VRAM at 4-bit8 GB · 8K
• Parameters8.2B
• Max context40K

Architecture

Dense

License

Apache-2.0

Launch date

Apr 27, 2025

Vendor

Alibaba

Hugging Face repo

Qwen/Qwen3-8B

GGUF repo

Qwen/Qwen3-8B-GGUF

NVFP4 repo (Blackwell)

nvidia/Qwen3-8B-NVFP4

MLX repo (Apple)

Qwen/Qwen3-8B-MLX-4bit

Ollama tag

qwen3:8b

Layers

36

Hidden size

4,096

KV heads

8

Head dim

128

Full calculation rules are on the Methodology page.

Device	Price	Speed t/s	Max context
Tesla V100 16GB16 GB	$37024 listings	105.2	40K
GeForce RTX 5060 Ti 16GB16 GB	$86958 listings	56.6	40K
Tesla V100 32GB32 GB	$89518 listings	105.2	40K
Radeon RX 7900 XTX24 GB	$1,25029 listings	111.1	40K
GeForce RTX 309024 GB	$1,799107 listings	108.8	40K
GeForce RTX 409024 GB	$3,75868 listings	115.8	40K
Ryzen AI Max+ 395 128GBWhole system · 128 GB	$4,46420 listings	33.5	40K
RTX A600048 GB	$4,89916 listings	91.8	40K
Mac Studio M5 Ultra 96GBWhole system · 96 GB	$7,27412 listings	136.3	40K
MacBook Pro M5 Max 128GBWhole system · 128 GB	$7,54920 listings	75.3	40K
DGX Spark 128GBWhole system · 128 GB	$8,24022 listings	35.6	40K
GeForce RTX 509032 GB	$8,395123 listings	182.8	40K
RTX PRO 600096 GB	$22,50011 listings	182.8	40K

Only common GPUs and systems are listed, one set for models under 50B and another for larger ones, sorted by price from low to high.

1. 1

   Thinking switches

   Thinking is on by default, and there are two ways to turn it off. enable_thinking=False is a hard switch that suppresses thinking output entirely; alternatively, add /think or /no_think to your prompt to switch turn by turn. Qwen dropped official support for that second, soft switch starting with Qwen3.5.

2. 2

   Official sampling parameters

   Thinking mode: temperature 0.6, top_p 0.95, top_k 20, min_p 0; Qwen stresses not to use greedy decoding (always picking the most likely token), or the model tends to repeat endlessly. Non-thinking mode: temperature 0.7, top_p 0.8, top_k 20, min_p 0.

3. 3

   Going beyond 32K of context

   Its native context is only 32K, extendable to 131K with YaRN. But Qwen notes that YaRN in open-source frameworks uses a fixed scaling factor, which can hurt results on short text, so only enable it when you actually need to process long documents. In llama-server, just add the rope scaling flags; if you change the config file instead, you need to regenerate the GGUF.

4. 4

   Which quantization to pick

   Qwen's official repository provides GGUFs directly, and the Ollama tag is qwen3:8b. 4-bit is about 5.0 GB and 8-bit about 8.7 GB, only 3.7 GB apart, so use 8-bit if you have the VRAM.

ollama run qwen3:8b

llama-server -hf Qwen/Qwen3-8B-GGUF:Q4_K_M -c 8192

vllm serve Qwen/Qwen3-8B

vllm serve nvidia/Qwen3-8B-NVFP4

mlx_lm.server --model Qwen/Qwen3-8B-MLX-4bit

File names follow that week's model-library snapshot; the definitions are in the methodology. Methodology

Model	Total params	Active params	4-bit size
Qwen3.5 9BDense	9.7B	9.7B	5.7 GB
Qwen3.6 27BDense	27.8B	27.8B	16.8 GB
Qwen3.8 27BDense	27.8B	27.8B	16.5 GB
Qwen3 32BDense	32.8B	32.8B	19.8 GB
Qwen3.6 35B A3BMoE	36B	3B	22.1 GB
Qwen3.8 Flash NextMoE	176B	6B	111.3 GB
Qwen3.8 2.4T A95BMoE	2,446B	95B	1,310.9 GB

About Qwen