VRAMGlass contains affiliate links. As an eBay Partner, we may be compensated if you make a purchase.As an eBay Partner, we may be compensated.Learn more

GPUs tracked57Price history49 daysPrice points (24h)43,400 pointsBiggest 24h dropGeForce RTX 3060 12GB-11.1%Biggest 24h riseCMP 170HX 40GB+19.8%Updated
$ USD

How to Deploy Llama 3.3 70B Locally

Llama 3.3 70B is a 70B dense model that Meta released at the end of 2024, and Meta says it comes close to Llama 3.1 405B on many benchmarks. At 4-bit with 8K it takes about 46 GB, so you need one card with 48GB or more, or two 24GB cards. A 128GB unified-memory machine can hold it, but a dense model reads all of its weights for every token it generates, so 4-bit only manages about 5 tokens/s.

Llama 3.3 70B details

  • Min VRAM at 4-bit47 GB · 8K
  • Parameters70.6B
  • Max context128K
Architecture
Dense
License
Llama 3.3 Community License
Launch date
Dec 6, 2024
Vendor
Meta AI
Hugging Face repo
meta-llama/Llama-3.3-70B-Instruct
GGUF repo
bartowski/Llama-3.3-70B-Instruct-GGUF
NVFP4 repo (Blackwell)
nvidia/Llama-3.3-70B-Instruct-NVFP4
MLX repo (Apple)
mlx-community/Llama-3.3-70B-Instruct-4bit
Ollama tag
llama3.3:70b
Layers
80
Hidden size
8,192
KV heads
8
Head dim
128

How much VRAM does Llama 3.3 70B need for local deployment?

QuantizationWeightsGB8K contextGB32K contextGB128K context (max)GB
1-bit16.8GGUF20.628.660.8
2-bit27.4GGUF31.239.271.4
3-bit38.1GGUF41.949.982.1
4-bit42.5GGUF46.354.386.5
5-bit49.9GGUF53.761.793.9
8-bit75GGUF78.886.8119
BF16/FP16141.1GGUF144.9152.9185.1

Full calculation rules are on the Methodology page.

Can a single GPU run Llama 3.3 70B?

It depends on the quant. 4-bit with 8K of context needs about 46 GB, so a single card has to have at least 48GB, which rules out every mainstream consumer card. The best single-card home for it is the 96GB RTX PRO 6000, at about 30 tokens/s with the full context window. A 48GB RTX A6000 just fits it, but at less than half the speed and with tight context. An RTX 5090 has to fall back to 2-bit, which runs fine but noticeably hurts answer quality. 128GB unified-memory machines hold even 8-bit, but a dense model reads all of its weights for every token, so 4-bit crawls along at around 5 tokens/s, fine only when you're in no hurry.

eBay
DevicePriceSpeed t/sMax context
Ryzen AI Max+ 395 128GBWhole system · 128 GB$4,46420 listings4.5128K
Instinct MI21064 GB$4,99411 listings27.674K
Mac Studio M5 Ultra 96GBWhole system · 96 GB$7,27412 listings20.9128K
MacBook Pro M5 Max 128GBWhole system · 128 GB$7,54920 listings10.6128K
DGX Spark 128GBWhole system · 128 GB$8,24022 listings4.7128K
RTX PRO 5000 72GB72 GB$10,7223 listings22.8100K
A100 80GB PCIe80 GB$18,99912 listings32.3126K
RTX PRO 600096 GB$22,50011 listings30128K
Mac Studio M3 Ultra 512GBWhole system · 512 GB$22,90013 listings14128K

Only common GPUs and systems are listed, one set for models under 50B and another for larger ones, sorted by price from low to high.

How many GPUs do you need to run Llama 3.3 70B?

Two 32GB RTX 5090 cards are the simplest setup, running 4-bit at about 28 tokens/s. With 24GB cards (RTX 3090, 4090, RX 7900 XTX), vLLM needs four for 4-bit: the model plus 32K of context is about 54 GB, more than a pair's 48GB, and four cards do roughly 15 tokens/s. If you'd rather stick with two 24GB cards, you have two ways out: drop to 2-bit for about 21 tokens/s, or switch to llama.cpp with just 8K of context, where 4-bit squeezes onto the pair with very little headroom.

eBay
DeviceCardsTotal priceTotal VRAMSpeed t/sTotal power
Tesla V100 32GB32 GB per card2 cards$1,79064 GB14.2500 W
Arc Pro B7032 GB per card2 cards$3,70064 GB9.6460 W
Radeon RX 7900 XTX24 GB per card4 cards$5,00096 GB15.11,420 W
GeForce RTX 309024 GB per card4 cards$7,19696 GB14.71,400 W
RTX A600048 GB per card2 cards$9,79896 GB12.1600 W
GeForce RTX 409024 GB per card4 cards$15,03296 GB15.81,800 W
GeForce RTX 509032 GB per card2 cards$16,79064 GB27.71,150 W
GeForce RTX 4090 48GB48 GB per card2 cardsNo price96 GB15.8900 W

Full calculation rules are on the Methodology page.

What to know before running Llama 3.3 70B locally

  1. 4-bit, 5-bit or 8-bit

    The 4-bit weights are 42.5 GB, and with llama.cpp at 8K, that's the highest level two 24GB cards can hold. With a single 96GB card, go to 8-bit (75.0 GB): about 18 tokens/s on the RTX PRO 6000, with about 80K of context. 5-bit (49.9 GB), in between, suits mismatched pairs like a 48GB card plus a 24GB card.

  2. Speed it up with speculative decoding

    A 70B generates slowly, so it's a good candidate for a draft model: the small model guesses a few tokens ahead and the large model checks them in one pass. The vLLM recipe provides an EAGLE-3 draft model trained by Red Hat; in llama.cpp, you can use Llama 3.2 3B, which shares its vocabulary, as the draft with -md.

  3. Frameworks and parameters

    llama.cpp, Ollama (llama3.3:70b) and vLLM all support it directly, and the tool-calling format and sampling parameters (temperature 0.6, top_p 0.9) are the same as Llama 3.1.

  4. Long context takes 40 GB more

    It uses standard attention, so VRAM keeps rising as context grows: at 4-bit, going from 8K to 128K takes VRAM from 46.3 GB to 86.5 GB. Two 24GB cards and a single 48GB A6000 are in about the same position, with context only up to around 20K; for the full 128K, you need a single 96GB card, four 24GB cards or a 128GB machine.

Launch commands

Ollama

  • Windows
  • macOS
  • Linux
Terminal
ollama run llama3.3:70b

llama.cpp

  • NVIDIA
  • AMD
  • Mac
  • CPU
Terminal
llama-server -hf bartowski/Llama-3.3-70B-Instruct-GGUF:Q4_K_M -c 8192

vLLM

  • Linux
  • NVIDIA
  • AMD
Original weights
Terminal
vllm serve meta-llama/Llama-3.3-70B-Instruct
NVFP4 · Blackwell

Blackwell GPUs (RTX 50 series, RTX PRO 6000, B200) run NVFP4 weights natively, at about 30% of the original size. For multiple GPUs, add --tensor-parallel-size with the card count.

Terminal
vllm serve nvidia/Llama-3.3-70B-Instruct-NVFP4

MLX

  • Apple Silicon

On a Mac, pip install mlx-lm first; this command downloads the MLX weights from Hugging Face and starts an OpenAI-compatible server.

Terminal
mlx_lm.server --model mlx-community/Llama-3.3-70B-Instruct-4bit

File names follow that week's model-library snapshot; the definitions are in the methodology. Methodology

Other models in the Meta family

ModelTotal paramsActive params4-bit size
Llama 3.2 3BDense3.2B3.2B2 GB
Llama 3.1 8BDense8B8B4.9 GB
Llama 4 Scout 17B 16EMoE108.6B17B65.4 GB
Muse Spark OpenMoE816B42B505.5 GB

About Meta