How to Deploy Llama 3.1 8B Locally

• Min VRAM at 4-bit8 GB · 8K
• Parameters8B
• Max context128K

Architecture

Dense

License

Llama 3.1 Community License

Launch date

Jul 23, 2024

Vendor

Meta AI

Hugging Face repo

meta-llama/Llama-3.1-8B-Instruct

GGUF repo

bartowski/Meta-Llama-3.1-8B-Instruct-GGUF

NVFP4 repo (Blackwell)

nvidia/Llama-3.1-8B-Instruct-NVFP4

MLX repo (Apple)

mlx-community/Llama-3.1-8B-Instruct-4bit

Ollama tag

llama3.1:8b

Layers

32

Hidden size

4,096

KV heads

8

Head dim

128

Full calculation rules are on the Methodology page.

Device	Price	Speed t/s	Max context
Tesla V100 16GB16 GB	$37024 listings	107.9	83K
GeForce RTX 5060 Ti 16GB16 GB	$86958 listings	58.1	83K
Tesla V100 32GB32 GB	$89518 listings	107.9	128K
Radeon RX 7900 XTX24 GB	$1,25029 listings	114	128K
GeForce RTX 309024 GB	$1,799107 listings	111.6	128K
GeForce RTX 409024 GB	$3,75868 listings	118.7	128K
Ryzen AI Max+ 395 128GBWhole system · 128 GB	$4,46420 listings	34.4	128K
RTX A600048 GB	$4,89916 listings	94.2	128K
Mac Studio M5 Ultra 96GBWhole system · 96 GB	$7,27412 listings	139.7	128K
MacBook Pro M5 Max 128GBWhole system · 128 GB	$7,54920 listings	77.4	128K
DGX Spark 128GBWhole system · 128 GB	$8,24022 listings	36.6	128K
GeForce RTX 509032 GB	$8,395123 listings	186.9	128K
RTX PRO 600096 GB	$22,50011 listings	186.9	128K

Only common GPUs and systems are listed, one set for models under 50B and another for larger ones, sorted by price from low to high.

1. 1

   Choose quantization by how much context you need

   Between 4-bit (4.9 GB) and 8-bit (8.5 GB), pick the highest level that still fits the context you want. Every extra 1K tokens of context adds about 0.13 GB of cache, so 4-bit at 32K already needs 10.3 GB. Its native context is only 8K; the 128K comes from RoPE scaling.

2. 2

   What the license requires of fine-tunes

   The Llama 3.1 Community License requires that when you distribute it or a model based on it, you prominently display “Built with Llama.” If you release a model trained or fine-tuned using it or its outputs, the name must begin with “Llama,” and companies with more than 700 million monthly active users need a separate license from Meta. Fine-tunes based on it are bound by the same license.

3. 3

   Tool calling

   Llama 3.1 was trained with a tool-calling format and also knows three built-in tool names: brave_search, wolfram_alpha and code_interpreter. llama.cpp's llama-server recognizes them when you add --jinja. For sampling, stick with Meta's defaults: temperature 0.6 and top_p 0.9.

4. 4

   Where it shows its age

   Its knowledge cutoff is December 2023, and it handles text only. It officially supports 8 languages: English, German, French, Italian, Portuguese, Hindi, Spanish and Thai. Chinese isn't among them, and Meta explicitly advises against using it to chat in unsupported languages without fine-tuning. For Chinese, look at Qwen3.5 9B instead.

ollama run llama3.1:8b

llama-server -hf bartowski/Meta-Llama-3.1-8B-Instruct-GGUF:Q4_K_M \
  -c 8192

vllm serve meta-llama/Llama-3.1-8B-Instruct

vllm serve nvidia/Llama-3.1-8B-Instruct-NVFP4

mlx_lm.server --model mlx-community/Llama-3.1-8B-Instruct-4bit

File names follow that week's model-library snapshot; the definitions are in the methodology. Methodology

Model	Total params	Active params	4-bit size
Llama 3.2 3BDense	3.2B	3.2B	2 GB
Llama 3.3 70BDense	70.6B	70.6B	42.5 GB
Llama 4 Scout 17B 16EMoE	108.6B	17B	65.4 GB
Muse Spark OpenMoE	816B	42B	505.5 GB

About Meta