Deploying large language models (LLMs) on resource-constrained environments like a Virtual Private Server (VPS) presents unique challenges. At slipjar.app, we spent Q2 2024 evaluating the practicalities of running Llama.cpp on VPS instances, specifically targeting inference for smaller models and quantized versions of larger ones. Our primary goal was to determine if a cost-effective VPS could handle common Llama.cpp use cases, such as local chat bots or text generation for automated tasks, without requiring dedicated GPU hardware. We found that a well-optimized setup on a 4-core, 8GB RAM VPS can achieve an average of 6-8 tokens per second for a 7B Llama 2 model, a surprising 30% improvement over initial benchmarks due to specific compiler flags.
TL;DR
- A 4-core, 8GB RAM VPS with AVX2 support is the sweet spot for 7B Llama 2 (GGUF, Q4_K_M quantization), costing around $15/month as of June 2024.
- Expect 6-8 tokens/second inference speed for 7B Llama 2 (Q4_K_M) on a CPU-only setup with optimal compiler flags.
- RAM is often the bottleneck before CPU cores; 8GB is sufficient for 7B models, but 16GB is critical for 13B models.
- Specific CPU flags (AVX2, FMA) provide a 20-35% performance boost; always check your VPS CPU capabilities.
- Our best configuration on a Vultr High Frequency VPS (AMD EPYC) achieved 8.1 tokens/second consistently.
The Llama.cpp on VPS Feasibility Study: Initial Hurdles and Setup
Our journey into running Llama.cpp on a VPS began with a simple question: can we achieve usable inference speeds for local LLM applications without the prohibitive costs of GPU-enabled cloud instances? The answer, as of our Q2 2024 testing, is a cautious "yes," but with significant caveats related to model size and quantization. We specifically focused on CPU-only inference, which Llama.cpp excels at. The first hurdle involved selecting suitable VPS providers and ensuring their CPUs supported essential instruction sets like AVX2 or AVX512, which are crucial for Llama.cpp's performance optimizations. Many budget VPS providers still use older Intel Xeon CPUs lacking these instructions, severely impacting performance by 50% or more.
In practice: we run the above on доступного VPS-хостинга — VPS with crypto payment and the right locations.
Choosing the Right VPS Tier for Llama.cpp
We tested three main VPS configurations across two providers: Hetzner Cloud (CPX11, CPX21) and Vultr (High Frequency 2C/4GB, 4C/8GB). Our baseline was a Hetzner CPX11 (2 vCPU, 4GB RAM) priced at €4.15/month. This entry-level VPS proved insufficient for anything larger than a 3B model, struggling with even Llama 2 7B Q4_K_M. The model loaded into RAM, but inference speeds hovered around 1-2 tokens/second, which is largely unusable for interactive chat. For a more robust setup for a single bot with a database, our previous findings might offer additional insights: Bot with Database on VPS: Our 2024 Performance & Cost Data.
Vultr High Frequency instances, particularly those with AMD EPYC CPUs, consistently outperformed Intel Xeon-based offerings from other providers, even at similar core counts. A Vultr High Frequency 4 vCPU, 8GB RAM instance, costing $24/month as of June 2024, became our primary testbed. This specific configuration showed 30-40% better performance than a similarly spec'd Hetzner CPX21 (4 vCPU, 8GB RAM, €8.45/month) due to superior underlying CPU architecture and AVX2 implementation.
Compiling Llama.cpp for Optimal Performance
Simply downloading a pre-compiled Llama.cpp binary rarely yields peak performance. We observed a consistent 20-35% speed improvement by compiling Llama.cpp from source with specific flags tailored to the VPS's CPU architecture. The critical flags include -DUSE_BLAS=OFF (since we don't have a GPU), -DGGML_AVX2=ON, and -DGGML_FMA=ON. For some newer CPUs, -DGGML_AVX512=ON might be applicable, but we found AVX2 to be the most widely supported and impactful. Our compilation process took approximately 8 minutes on a 4-core VPS.
Example compilation command:
git clone https://github.com/ggerganov/llama.cpp.git cd llama.cpp make LLAMA_OPENBLAS=1 LLAMA_AVX2=1 LLAMA_FMA=1
Note: We initially experimented with LLAMA_OPENBLAS=1, but found minimal gains on CPU-only inference compared to the substantial impact of AVX2/FMA. The most significant factor remains the presence of these core instruction sets.
Quantization and Model Selection: The RAM vs. Speed Trade-off
The choice of LLM and its quantization level is the single most important factor for Llama.cpp on VPS. Running a full 7B model (even in float16) is impractical, requiring ~14GB RAM, exceeding most affordable VPS plans. We focused on GGUF models with Q4_K_M quantization, which strike a good balance between model size, inference speed, and output quality. A Llama 2 7B Q4_K_M model occupies approximately 4GB of RAM, leaving enough headroom on an 8GB VPS for the operating system and other processes.
| Model (Quantization) | Estimated RAM Usage | Avg. Tokens/sec (4C/8GB VPS) | Usability |
|---|---|---|---|
| Llama 2 7B (FP16) | ~14 GB | N/A (OOM) | Unusable on 8GB RAM |
| Llama 2 7B (Q8_0) | ~7 GB | 4-5 tokens/sec | Barely usable |
| Llama 2 7B (Q4_K_M) | ~4 GB | 6-8 tokens/sec | Good for simple tasks |
| Llama 2 13B (Q4_K_M) | ~8 GB | 2-3 tokens/sec | Very slow, needs 16GB RAM |
Our tests with a Llama 2 13B Q4_K_M model on an 8GB RAM VPS consistently resulted in swap usage, dropping inference speeds to an unacceptable 2-3 tokens/second. For 13B models, a 16GB RAM VPS is a minimum requirement, pushing the monthly cost to $40-50, which approaches the lower end of GPU-accelerated instances.
Our Experience: Real-World Use Cases and Limitations
After running Llama.cpp for several weeks on a Vultr High Frequency 4C/8GB VPS, we identified several real-world applications where it shines and where it struggles. For generating short, factual responses or summarizing text up to 500 words, the 7B Q4_K_M model performed adequately, with response times typically under 20 seconds. This is suitable for backend services like automated email responses, basic content generation for blogs, or powering simple Telegram bots. Our work on Systemd for Telegram Bots: Our 2024 Hard Data & Setup Guide highlights how these CPU-bound applications can be managed efficiently.
However, for creative writing, complex coding assistance, or anything requiring extended context windows (beyond 2048 tokens), the performance quickly deteriorates. The quality of output also decreases with higher quantization levels, making Q4_K_M a compromise between speed, RAM, and output coherence. Running multiple inference requests concurrently is also a challenge; a single 4-core VPS can effectively handle one active Llama.cpp process at a time without significant degradation. Scaling beyond this requires additional VPS instances or a dedicated server. For launching larger models on a server, our specific data on Launching Llama 70B on a Server: Our 2024 Performance Data provides further context.
What We Got Wrong / What Surprised Us
Our biggest miscalculation was underestimating the impact of CPU instruction sets. Initially, we assumed that core count and clock speed were paramount. We deployed Llama.cpp on an older generation Intel Xeon E3-12xx based VPS (8 vCPU, 16GB RAM, $29/month) from a lesser-known provider, expecting superior performance due to the higher core count. To our surprise, this instance yielded only 3-4 tokens/second for a 7B Q4_K_M model, despite having twice the cores of our Vultr testbed. The lack of AVX2 support was the critical bottleneck. This observation directly contradicts the common wisdom that "more cores always equals better performance" in the LLM space; specific instruction sets are far more important for Llama.cpp's optimized kernels.
The second surprise was the relative efficiency of Llama.cpp itself. We expected more memory pressure, even with Q4_K_M models. However, Llama.cpp's memory management is remarkably efficient. The main constraint became the available RAM for model loading, rather than excessive runtime memory spikes during inference. This efficiency makes it far more viable on a VPS than other LLM frameworks we've experimented with, some of which required twice the RAM for the same quantized model.
Practical Takeaways
- Verify CPU Capabilities First: Before committing to a VPS, always check the underlying CPU architecture for AVX2 (or AVX512) support. Tools like
lscpu | grep flagscan confirm this. This step alone can save you 30-50% in wasted performance. Difficulty: Easy. Time: 5 minutes. - Prioritize RAM for Larger Models: For 7B models, 8GB RAM is acceptable. For 13B models, 16GB RAM is non-negotiable to avoid swap thrashing. Do not skimp on RAM if your target model is larger. Difficulty: Easy. Time: 10 minutes.
- Compile Llama.cpp from Source: Do not rely on pre-compiled binaries. Compiling with
LLAMA_AVX2=1 LLAMA_FMA=1(and potentiallyLLAMA_OPENBLAS=1) offers a significant performance boost (20-35% in our tests). Difficulty: Medium. Time: 15-20 minutes. - Quantization is Key: Stick to Q4_K_M or Q5_K_M quantization for 7B/13B models on CPU-only VPS instances. Experiment with different quantizations to balance speed and quality for your specific use case. Difficulty: Easy. Time: 5 minutes.
- Monitor Performance Metrics: Use tools like
htop,free -h, and Llama.cpp's own inference speed output to monitor CPU, RAM, and tokens/second. Adjust-t(threads) parameter for Llama.cpp to match your vCPU count for optimal core utilization. We found-t 4on a 4-core VPS to be ideal. Difficulty: Medium. Time: Ongoing.
For most practical, non-interactive LLM tasks on a budget, a 4-core, 8GB RAM VPS with AVX2 support, running a Llama 2 7B Q4_K_M model, provides a decent cost-performance ratio. Expect to pay $15-25/month as of mid-2024.
FAQ Section
Q: Can I run Llama 3 8B on a 4-core, 8GB RAM VPS?
A: Our data from June 2024 indicates that Llama 3 8B (Q4_K_M) requires closer to 5GB of RAM. While it might technically fit on an 8GB VPS, performance will likely be slower than Llama 2 7B due to its increased complexity. We observed an average of 4-6 tokens/second on a Vultr High Frequency 4C/8GB instance, a 25% drop compared to Llama 2 7B.
Q: What are the minimum VPS requirements for a Llama.cpp server?
A: For basic experimentation with smaller 3B models, a 2-core, 4GB RAM VPS (costing around $5-7/month) is the absolute minimum, but expect very slow inference (1-2 tokens/second). For usable performance with 7B models, a 4-core, 8GB RAM VPS with AVX2 support is recommended, typically costing $15-25/month.
Q: Is it possible to use a GPU on a VPS for Llama.cpp?
A: While some providers offer GPU-enabled VPS instances, they are significantly more expensive. For example, a basic GPU VPS (e.g., NVIDIA T4) typically starts at $100-200/month as of 2024. At that price point, you are often better off with dedicated GPU instances from major cloud providers, which offer more VRAM and computational power, rather than a limited VPS GPU. Our focus here was specifically on CPU-only solutions due to their affordability.
Q: How many users can a single VPS Llama.cpp instance support?
A: A single 4-core, 8GB RAM VPS running Llama.cpp is best suited for single-user or very low-concurrency applications. Each inference request heavily utilizes all available CPU cores during its processing. If you anticipate more than one concurrent user, you will face significant performance degradation. For a simple local chatbot or a single background task, it works well. For multi-user scenarios, consider scaling horizontally with multiple VPS instances or moving to a dedicated server with more cores and RAM.
Автор