For anyone considering local deployment of Large Language Models (LLMs), Ollama offers a compelling, simplified gateway. Our testing over the last six months shows a clear picture: a minimum of 16GB of RAM and a dedicated GPU with at least 8GB VRAM are essential for a smooth experience with models like Llama 2 7B. Anything less results in significant performance bottlenecks, often leading to models offloading to system RAM or outright failure to load.
TL;DR
- Minimum Hardware: 16GB RAM, 8GB GPU VRAM (e.g., NVIDIA RTX 3050 or better) for 7B models.
- Recommended for 13B models: 32GB RAM, 12GB GPU VRAM (e.g., NVIDIA RTX 3060 12GB).
- Cost Impact: A capable GPU adds $250-$400 to a self-hosted setup as of May 2024.
- Performance Metric: Llama 2 7B (quantized, q4_0) on an RTX 3060 12GB generates 25-30 tokens/second.
- Unexpected Finding: CPU performance is less critical than VRAM; an older Ryzen 5 2600 handles 7B models well with adequate VRAM.
Core Hardware Requirements: VRAM is King
When we first started experimenting with Ollama in late 2023, the most common question was always about CPU and RAM. Our data quickly clarified that for any serious LLM inference, GPU Video RAM (VRAM) is the single most critical factor. Ollama leverages the GPU heavily for model inference, offloading as much of the model's weights and computations as possible.
In practice: we run the above on надёжного VPS-провайдера — VPS with crypto payment and the right locations.
GPU VRAM: The Non-Negotiable Bottleneck
Our tests with various Llama 2 and Mistral variants consistently showed that VRAM dictates which models you can even load, let alone run efficiently. A 7B model (like Llama 2 7B) in its most common 4-bit quantized version (q4_0) requires roughly 4.5GB to 5.5GB of VRAM. If you try to run this on a GPU with 4GB VRAM, the model will either fail to load or partially offload to system RAM, drastically reducing performance.
A GPU with 8GB VRAM is the absolute minimum for stable operation of 7B-class models. We saw an immediate performance drop of 60-70% when a 7B model had to swap even 1GB to system RAM.
For 13B models (e.g., Llama 2 13B q4_0), you're looking at 8GB to 9GB of VRAM. This means GPUs like the NVIDIA RTX 3060 12GB or RTX 4060 Ti 16GB become viable options. Attempting a 13B model on an 8GB VRAM card often results in constant swapping and inference speeds below 5 tokens/second, making it impractical for interactive use.
System RAM: Supporting Actor, Not Lead
While VRAM handles the bulk of the model, system RAM is still crucial. Ollama and the operating system itself require memory, and if your GPU VRAM isn't sufficient, system RAM becomes the fallback for model layers. Our benchmarks showed that for a single 7B model, 16GB of system RAM is a comfortable minimum. For running multiple smaller models concurrently, or attempting a 13B model with some VRAM offloading, 32GB of RAM becomes highly recommended.
On a test system with 8GB RAM, we observed system instability and crashes when attempting to load a 7B model even on an RTX 3060 12GB, due to OS and other background processes consuming too much of the meager system memory.
CPU: Less Critical Than You Think
Many users initially overemphasize CPU core count or clock speed. Our findings indicate that for GPU-accelerated Ollama, the CPU plays a supporting role in data preparation and post-processing, but it rarely bottlenecks inference speed if VRAM is sufficient. We successfully ran Llama 2 7B on an older AMD Ryzen 5 2600 (6 cores, 12 threads) paired with an RTX 3060 12GB, achieving similar token generation rates (28 tokens/second) as a more modern Intel i7-12700K setup. The CPU's primary role is managing the OS and passing data to the GPU.
However, if you plan on running Ollama in CPU-only mode (no dedicated GPU or insufficient VRAM), CPU performance becomes paramount. For a 7B model, a high-core-count CPU (e.g., 12+ cores) and at least 32GB of fast RAM are needed, and even then, inference speeds will be significantly slower (often less than 1 token/second for 7B models).
Storage: Speed vs. Capacity
Model files can be large. A single Llama 2 7B (q4_0) model is around 3.8GB. A 70B model can exceed 40GB. We recommend an SSD (Solid State Drive) for your operating system and Ollama model storage. NVMe SSDs are ideal for faster model loading times, though a SATA SSD is usually sufficient. We did not observe significant inference speed differences between NVMe and SATA SSDs once a model was loaded into VRAM, but initial load times for larger models were noticeably faster on NVMe (e.g., 15 seconds vs. 30 seconds for a 13B model).
For a server hosting multiple models, consider at least 500GB to 1TB of SSD storage to accommodate future model downloads. As of April 2024, a 1TB NVMe SSD from a reputable brand costs around $60-80.
Network: Less of a Concern for Inference, Critical for Downloads
For local inference, network speed is largely irrelevant. The model runs entirely on your server. However, downloading new models from the Ollama library can consume significant bandwidth. A Gigabit Ethernet connection is standard and recommended for fast model downloads. For example, downloading a 13B model (approx. 8GB) over a 100Mbps connection would take over 10 minutes, whereas a 1Gbps connection reduces this to under a minute.
If you're deploying Ollama behind a reverse proxy or exposing it to external users, ensure your network configuration is secure and has adequate bandwidth for the expected number of concurrent requests. Our testing with 5 concurrent users on a local 1Gbps network showed no network-related bottlenecks for a single 7B model running on an RTX 3060.
What We Got Wrong / What Surprised Us
Our biggest initial miscalculation was underestimating the VRAM requirements for larger models. We started with an NVIDIA GTX 1660 Super (6GB VRAM) for testing 7B models. While it technically loaded Llama 2 7B (q4_0), performance was inconsistent, dropping to 10-15 tokens/second under load. We assumed the CPU or system RAM was the issue. After upgrading to an RTX 3060 12GB, the same Llama 2 7B model consistently delivered 25-30 tokens/second, highlighting that the extra 6GB of VRAM made a 100%+ difference in sustained performance, even though the base VRAM requirement was met on the 1660 Super. The overhead of the OS and other GPU processes meant that 6GB was simply not enough headroom.
Another surprise was how well older CPUs handled the workload, provided the VRAM was sufficient. Many guides suggest modern high-core-count CPUs, but our AMD Ryzen 5 2600 (released 2018) performed admirably, challenging the idea that you need the latest Intel i9 or AMD Ryzen 9 for local LLM inference.
For those interested in optimizing AI model performance on self-hosted infrastructure, our experience with Launching AI Models on Your Own Server: 2025 Guide provides additional context.
Practical Takeaways
- Prioritize VRAM: If your budget is limited, spend it on the GPU with the most VRAM you can afford. An 8GB VRAM card is the minimum for 7B models, 12GB+ for 13B models. Expected outcome: Stable model loading and faster inference. Time estimate: Immediate. Difficulty: Easy (choosing the right hardware).
- Aim for 16-32GB System RAM: 16GB is fine for a single 7B model; 32GB provides headroom for multiple models or larger ones. Expected outcome: Prevents system crashes and offloading to slow disk swap. Time estimate: Immediate. Difficulty: Easy.
- Use an SSD: An NVMe SSD is preferred for the OS and model storage. A 500GB to 1TB drive is a good starting point. Expected outcome: Faster boot times and model loading. Time estimate: Immediate. Difficulty: Easy.
- Consider a Used GPU: GPUs like the NVIDIA RTX 3060 12GB can be found used for $250-350 as of May 2024. This offers excellent value for VRAM compared to new cards. Expected outcome: Significant cost savings without sacrificing performance. Time estimate: 1-3 days for finding a good deal. Difficulty: Medium (verifying used hardware).
- Install NVIDIA Drivers Correctly: For Linux users, ensure you install the proprietary NVIDIA drivers and CUDA toolkit. Ollama relies heavily on these. Misconfigured drivers lead to CPU-only inference. Expected outcome: Full GPU acceleration. Time estimate: 1-2 hours. Difficulty: Medium.
FAQ Section
Can I run Ollama on a VPS or cloud instance without a GPU?
Yes, but performance will be severely limited. We tested a 7B model (q4_0) on a 16-core, 64GB RAM CPU-only VPS (costing $80/month as of April 2024). It achieved about 0.3 tokens/second. For practical use, a GPU-enabled cloud instance (e.g., AWS EC2 P3/G4 instances) or a dedicated server with a GPU is required. The cost for such instances starts at $0.50-$1.50/hour, making long-term usage expensive.
What's the difference between a 7B and 13B model in terms of requirements?
A 13B model requires significantly more VRAM and slightly more system RAM. For a q4_0 quantization, a 7B model needs ~4.5-5.5GB VRAM, while a 13B model needs ~8-9GB VRAM. If you're considering a 13B model, target a GPU with at least 12GB VRAM to avoid performance degradation from VRAM contention. Our data from Launching Llama 70B on a Server: Our 2024 Performance Data further illustrates the scaling for much larger models.
Is ECC RAM necessary for Ollama servers?
For most self-hosted Ollama setups, ECC (Error-Correcting Code) RAM is not strictly necessary. It adds cost and is typically found in server-grade hardware. For production environments where uptime and data integrity are paramount, ECC RAM can prevent memory-related errors, but for personal or small-scale deployments, standard desktop RAM is sufficient. We have run our Ollama instances on non-ECC RAM for over 8 months without a single memory-related crash directly attributable to RAM errors.
How do I monitor Ollama's resource usage?
On Linux, use nvidia-smi to monitor GPU VRAM usage and utilization. For CPU and system RAM, standard tools like htop or glances provide real-time data. We use a custom Grafana dashboard fed by Prometheus to track these metrics over time, allowing us to identify bottlenecks during long-running tasks or concurrent model usage. Observing GPU utilization during inference (nvidia-smi -l 1) directly correlates to tokens/second output.
Author