Best local LLM for your NVIDIA graphics card.
On a PC the graphics card's VRAM is what counts. We estimate VRAM minus 1 GB is usable. A model that does not fit in VRAM spills into normal RAM and runs much slower.
Published speeds.
Other people's measurements, with sources. Not our tests.
| Machine | Model | Tokens/sec | Engine | Source |
|---|---|---|---|---|
| GeForce RTX 5090 | gpt-oss-20b | 318.4 | llama.cpp · MXFP4 | 2026-10-15 |
| GeForce RTX 4090 | gpt-oss-20b | 242.6 | llama.cpp · Q4_K_M | 2026-10-15 |
| GeForce RTX 3090 (used) | gpt-oss-20b | 147.53 | llama.cpp (Linux, flash attention) · MXFP4 | 2025-10-26 |
Short answers.
- What is the best local LLM for the smallest NVIDIA GPU?
- With 8 GB (GeForce RTX 5060), the best fit is Qwen3.5 4B (needs 5.8 GB). 19 models fit.
- Which NVIDIA GPU runs the smartest local model?
- GeForce RTX 4090 with 24 GB runs Qwen3.8 27B (AA 33.7, about 59% of Claude Opus 5.5).
- What matters more, memory or speed?
- Memory decides which models fit at all. Bandwidth (GB/s) decides how fast they answer. Pick memory first, then the fastest machine you can afford with that memory.
Scores are from Artificial Analysis and LMArena, not our own tests yet. Memory need = 4-bit file × 1.15 + 2 GB; a model fits when that is at most 75% of memory. Models 2026-10-15, machines 2026-10-15.
# Best local LLM for your NVIDIA graphics card (RTX 3060 to 5090) URL: https://insidethemachine.ai/best/local-llm-for-nvidia-gpu Data: machines 2026-10-15, models 2026-10-15. On a PC the graphics card's VRAM is what counts. We estimate VRAM minus 1 GB is usable. A model that does not fit in VRAM spills into normal RAM and runs much slower. | machine | memory GB | usable GB | bandwidth GB/s | models that fit | best fit | needs GB | |---|---|---|---|---|---|---| | GeForce RTX 5090 | 32 | 31 | 1792 | 75 | Qwen3.8 27B | 22.7 | | GeForce RTX 5080 | 16 | 15 | 960 | 44 | K2 Horizon 7B | 8.6 | | GeForce RTX 5070 Ti | 16 | 15 | 896 | 44 | K2 Horizon 7B | 8.6 | | GeForce RTX 5070 | 12 | 11 | 672 | 36 | K2 Horizon 7B | 8.6 | | GeForce RTX 5060 Ti 16GB | 16 | 15 | 448 | 44 | K2 Horizon 7B | 8.6 | | GeForce RTX 5060 | 8 | 7 | 448 | 19 | Qwen3.5 4B | 5.8 | | GeForce RTX 4090 | 24 | 23 | 1008 | 56 | Qwen3.8 27B | 22.7 | | GeForce RTX 4080 Super | 16 | 15 | 736 | 44 | K2 Horizon 7B | 8.6 | | GeForce RTX 4070 Ti Super | 16 | 15 | 672 | 44 | K2 Horizon 7B | 8.6 | | GeForce RTX 4060 Ti 16GB | 16 | 15 | 288 | 44 | K2 Horizon 7B | 8.6 | | GeForce RTX 3090 (used) | 24 | 23 | 936 | 56 | Qwen3.8 27B | 22.7 | | GeForce RTX 3090 Ti | 24 | 23 | 1008 | 56 | Qwen3.8 27B | 22.7 | | GeForce RTX 3060 12GB (used) | 12 | 11 | 360 | 36 | K2 Horizon 7B | 8.6 | | RTX PRO 6000 Blackwell Workstation (96GB) | 96 | 95 | 1792 | 89 | Qwen3.8 27B | 22.7 | | RTX PRO 6000 Blackwell Max-Q (96GB) | 96 | 95 | 1792 | 89 | Qwen3.8 27B | 22.7 | | RTX PRO 5000 Blackwell 72GB | 72 | 71 | 1344 | 83 | Qwen3.8 27B | 22.7 | | RTX PRO 4500 Blackwell 32GB | 32 | 31 | ? | 75 | Qwen3.8 27B | 22.7 | | RTX PRO 4000 Blackwell 24GB | 24 | 23 | 672 | 56 | Qwen3.8 27B | 22.7 | ## Published speeds (decode tokens/sec, not our tests) - GeForce RTX 5090: gpt-oss-20b 318.4 tok/s (llama.cpp, MXFP4, RTX 5090 + Ryzen 7 9850X3D (chat-short workload)) source: https://llm-speed.com/m/gpt-oss-20b (2026-10-15) - GeForce RTX 4090: gpt-oss-20b 242.6 tok/s (llama.cpp, Q4_K_M, RTX 4090 + EPYC 7642 (chat-short)) source: https://llm-speed.com/m/gpt-oss-20b (2026-10-15) - GeForce RTX 3090 (used): gpt-oss-20b 147.53 tok/s (llama.cpp (Linux, flash attention), MXFP4, ) source: https://www.hardware-corner.net/guides/gpt-oss-20b-gpu-benchamrks/ (2025-10-26) ## Questions **What is the best local LLM for the smallest NVIDIA GPU?** With 8 GB (GeForce RTX 5060), the best fit is Qwen3.5 4B (needs 5.8 GB). 19 models fit. **Which NVIDIA GPU runs the smartest local model?** GeForce RTX 4090 with 24 GB runs Qwen3.8 27B (AA 33.7, about 59% of Claude Opus 5.5). **What matters more, memory or speed?** Memory decides which models fit at all. Bandwidth (GB/s) decides how fast they answer. Pick memory first, then the fastest machine you can afford with that memory.