# Best local LLM for your NVIDIA graphics card (RTX 3060 to 5090)

URL: https://insidethemachine.ai/best/local-llm-for-nvidia-gpu
Data: machines 2026-10-15, models 2026-10-15. On a PC the graphics card's VRAM is what counts. We estimate VRAM minus 1 GB is usable. A model that does not fit in VRAM spills into normal RAM and runs much slower.

| machine | memory GB | usable GB | bandwidth GB/s | models that fit | best fit | needs GB |
|---|---|---|---|---|---|---|
| GeForce RTX 5090 | 32 | 31 | 1792 | 75 | Qwen3.8 27B | 22.7 |
| GeForce RTX 5080 | 16 | 15 | 960 | 44 | K2 Horizon 7B | 8.6 |
| GeForce RTX 5070 Ti | 16 | 15 | 896 | 44 | K2 Horizon 7B | 8.6 |
| GeForce RTX 5070 | 12 | 11 | 672 | 36 | K2 Horizon 7B | 8.6 |
| GeForce RTX 5060 Ti 16GB | 16 | 15 | 448 | 44 | K2 Horizon 7B | 8.6 |
| GeForce RTX 5060 | 8 | 7 | 448 | 19 | Qwen3.5 4B | 5.8 |
| GeForce RTX 4090 | 24 | 23 | 1008 | 56 | Qwen3.8 27B | 22.7 |
| GeForce RTX 4080 Super | 16 | 15 | 736 | 44 | K2 Horizon 7B | 8.6 |
| GeForce RTX 4070 Ti Super | 16 | 15 | 672 | 44 | K2 Horizon 7B | 8.6 |
| GeForce RTX 4060 Ti 16GB | 16 | 15 | 288 | 44 | K2 Horizon 7B | 8.6 |
| GeForce RTX 3090 (used) | 24 | 23 | 936 | 56 | Qwen3.8 27B | 22.7 |
| GeForce RTX 3090 Ti | 24 | 23 | 1008 | 56 | Qwen3.8 27B | 22.7 |
| GeForce RTX 3060 12GB (used) | 12 | 11 | 360 | 36 | K2 Horizon 7B | 8.6 |
| RTX PRO 6000 Blackwell Workstation (96GB) | 96 | 95 | 1792 | 89 | Qwen3.8 27B | 22.7 |
| RTX PRO 6000 Blackwell Max-Q (96GB) | 96 | 95 | 1792 | 89 | Qwen3.8 27B | 22.7 |
| RTX PRO 5000 Blackwell 72GB | 72 | 71 | 1344 | 83 | Qwen3.8 27B | 22.7 |
| RTX PRO 4500 Blackwell 32GB | 32 | 31 | ? | 75 | Qwen3.8 27B | 22.7 |
| RTX PRO 4000 Blackwell 24GB | 24 | 23 | 672 | 56 | Qwen3.8 27B | 22.7 |

## Published speeds (decode tokens/sec, not our tests)

- GeForce RTX 5090: gpt-oss-20b 318.4 tok/s (llama.cpp, MXFP4, RTX 5090 + Ryzen 7 9850X3D (chat-short workload)) source: https://llm-speed.com/m/gpt-oss-20b (2026-10-15)
- GeForce RTX 4090: gpt-oss-20b 242.6 tok/s (llama.cpp, Q4_K_M, RTX 4090 + EPYC 7642 (chat-short)) source: https://llm-speed.com/m/gpt-oss-20b (2026-10-15)
- GeForce RTX 3090 (used): gpt-oss-20b 147.53 tok/s (llama.cpp (Linux, flash attention), MXFP4, ) source: https://www.hardware-corner.net/guides/gpt-oss-20b-gpu-benchamrks/ (2025-10-26)

## Questions

**What is the best local LLM for the smallest NVIDIA GPU?**
With 8 GB (GeForce RTX 5060), the best fit is Qwen3.5 4B (needs 5.8 GB). 19 models fit.

**Which NVIDIA GPU runs the smartest local model?**
GeForce RTX 4090 with 24 GB runs Qwen3.8 27B (AA 33.7, about 59% of Claude Opus 5.5).

**What matters more, memory or speed?**
Memory decides which models fit at all. Bandwidth (GB/s) decides how fast they answer. Pick memory first, then the fastest machine you can afford with that memory.