How to Deploy Qwen3.5 9B Locally

• Min VRAM at 4-bit8 GB · 8K
• Parameters9.7B
• Max context256K

Architecture

Dense

License

Apache-2.0

Launch date

Mar 2, 2026

Vendor

Alibaba

Hugging Face repo

Qwen/Qwen3.5-9B

GGUF repo

unsloth/Qwen3.5-9B-GGUF

MLX repo (Apple)

mlx-community/Qwen3.5-9B-4bit

Ollama tag

qwen3.5:9b

Layers

32

Hidden size

4,096

KV heads

4

Head dim

256

Full calculation rules are on the Methodology page.

Device	Price	Speed t/s	Max context
Tesla V100 16GB16 GB	$37024 listings	101.5	256K
GeForce RTX 5060 Ti 16GB16 GB	$86958 listings	54.4	256K
Tesla V100 32GB32 GB	$89518 listings	101.5	256K
Radeon RX 7900 XTX24 GB	$1,25029 listings	107.2	256K
GeForce RTX 309024 GB	$1,799107 listings	105	256K
GeForce RTX 409024 GB	$3,75868 listings	111.8	256K
Ryzen AI Max+ 395 128GBWhole system · 128 GB	$4,46420 listings	32.1	256K
RTX A600048 GB	$4,89916 listings	88.5	256K
Mac Studio M5 Ultra 96GBWhole system · 96 GB	$7,27412 listings	131.7	256K
MacBook Pro M5 Max 128GBWhole system · 128 GB	$7,54920 listings	72.5	256K
DGX Spark 128GBWhole system · 128 GB	$8,24022 listings	34.2	256K
GeForce RTX 509032 GB	$8,395123 listings	177.1	256K
RTX PRO 600096 GB	$22,50011 listings	177.1	256K

Only common GPUs and systems are listed, one set for models under 50B and another for larger ones, sorted by price from low to high.

1. 1

   Thinking is on by default

   The official template thinks first by default, and Qwen recommends leaving at least 128K of context to preserve thinking quality; fortunately, 128K at 4-bit is only about 11.1 GB. Qwen3's /think and /no_think switches are no longer supported; to turn thinking off, pass enable_thinking=false in the request. Note that Unsloth's docs say models 9B and smaller don't think by default, which contradicts the official template; with llama.cpp, go by the actual output, and to turn thinking on, add --chat-template-kwargs '{"enable_thinking":true}'.

2. 2

   Official sampling parameters

   Thinking mode, general tasks: temperature 1.0, top_p 0.95, top_k 20, presence_penalty 1.5; for coding, change to temperature 0.6 and presence_penalty 0. Non-thinking mode: temperature 0.7, top_p 0.8, top_k 20, presence_penalty 1.5.

3. 3

   Use the official library in Ollama

   In Ollama, just pull qwen3.5:9b, which supports image input. Unsloth notes that the vision part of third-party GGUFs is a separate mmproj file that won't run when imported into Ollama; use llama.cpp for those files.

4. 4

   Two vLLM switches

   For text-only work, add --language-model-only to skip the vision components and leave the VRAM for context. The model also has a built-in MTP multi-token prediction head, so vLLM and SGLang can enable speculative decoding for more speed; this site's speeds don't include that.

ollama run qwen3.5:9b

llama-server -hf unsloth/Qwen3.5-9B-GGUF:Q4_K_M -c 8192

vllm serve Qwen/Qwen3.5-9B

mlx_vlm.server --model mlx-community/Qwen3.5-9B-4bit

File names follow that week's model-library snapshot; the definitions are in the methodology. Methodology

Model	Total params	Active params	4-bit size
Qwen3 8BDense	8.2B	8.2B	5 GB
Qwen3.6 27BDense	27.8B	27.8B	16.8 GB
Qwen3.8 27BDense	27.8B	27.8B	16.5 GB
Qwen3 32BDense	32.8B	32.8B	19.8 GB
Qwen3.6 35B A3BMoE	36B	3B	22.1 GB
Qwen3.8 Flash NextMoE	176B	6B	111.3 GB
Qwen3.8 2.4T A95BMoE	2,446B	95B	1,310.9 GB

About Qwen