The Physics Don't Lie
Before writing a line of code, you have to be honest about what fits in 4 GB and what doesn't.
A $5/month VPS has 4 GB of RAM. The OS and runtime eat about 400 MB. That leaves roughly 3.6 GB for everything: the model weights, the KV cache, the embedder, the index navigation structures, and working memory. Any project that ignores this budget is not designing for the real constraint. We started here.
What Fits in 3.6 GB
At 4-bit quantization, a 1.7B parameter model takes about 1.1 GB. A 4B model takes about 2.4 GB. A 7B model needs 4 to 4.5 GB, which is already too tight once you add the KV cache and runtime overhead.
So 7B is out on a 4 GB box. The practical ceiling is around 4B, with a default of 1.7B. A monolithic model comparable to GPT-4 would need somewhere around 400 GB at 4-bit quantization. So that’s off the table. Any project that claims otherwise is either using different hardware than they say or running something much weaker than they imply.
The Quantization Floor
The quality cliff in language models sits at around 2.7 bits per weight, not 4 bits. Going from 16-bit to Q4_K_M costs less than 1% on perplexity benchmarks. Going from Q4_K_M to Q3 starts to become noticeable. Going below 3 bits falls off a cliff.
So 4-bit is the sensible floor for quality and the practical ceiling for size given a 4 GB budget. For now, 4-bit Qwen3.
The memory-bandwidth point is less obvious but equally important. On CPU, the bottleneck at inference time is moving bytes from RAM to the processor, not doing arithmetic. This means that halving the model size roughly doubles generation speed. It also means that the design should obsess over bytes-moved-per-token rather than FLOPs. A smaller model with the same quality is a strictly better choice on CPU hardware.
The Moonshot Case: BitNet
There’s a moonshot case at the bottom of this: BitNet b1.58, ternary weights at 1.58 bits, where a 2B model weighs 400 MB instead of 1.1 GB and runs matrix operations as additions and subtractions rather than multiply-accumulate. Microsoft published benchmarks showing it matches similarly-sized float models on most tasks and runs faster on CPU.
The catch is that you have to train from scratch in ternary. There’s no conversion path from an existing model. That’s a much longer project. It’s on the roadmap, not the critical path.
Disk as a First-Class Tier
Disk is the real secret. A cheap VPS comes with 40 to 80 GB of NVMe storage at 1 to 3 GB/s read bandwidth. That’s a first-class storage tier that almost nobody uses in ML system design. Most work assumes either RAM or the network.
The insight here is that retrieval from disk costs about 5ms per query when the index is designed right. DiskANN, a disk-resident approximate nearest-neighbor index from Microsoft Research, searches 1 billion vectors from SSD in under 3ms at 95% recall while keeping only a small navigation graph in RAM.
So the architecture is: small reasoner in RAM, large knowledge base on disk, exact compute in subprocesses, coordination in the Conductor. Each piece fits its tier. Nothing tries to be something it isn’t.
The constraints don’t leave much room for wishful thinking. That’s actually useful. It forces clean architecture.
Key takeaways:
- 4-bit quantization costs under 1% quality while cutting RAM use in half; 7B models don’t fit the budget
- The real inference bottleneck on CPU is memory bandwidth, not arithmetic, so smaller is always faster
- Disk storage at 1-3 GB/s is a first-class tier that makes large knowledge bases practical on cheap hardware
Next: whether retrieval can actually close the capability gap with much larger models.