Download a 70B parameter open model, wait for the file to finish, point it at a graphics card with 8 GB of VRAM, and watch it refuse to load. Not slow, not degraded, just an out-of-memory error before a single token comes out. This happens constantly to people who follow open models closely enough to know the names, Llama, Qwen, Mixtral, DeepSeek, but not closely enough to know that the parameter count printed in a model’s name is a promise about capability, not a promise about whether it fits on the machine in front of them. The quality of these models crossed a threshold of “good enough for most tasks” a couple of years ago. What decides whether any one person can actually use one is memory, full stop.

What actually fits in memory

Once a model is quantized to 4 bits, the tiers are fairly predictable. Around 8 GB of VRAM covers the 7-8B parameter class comfortably, which is where most of the well-known compact models sit. Around 24 GB, the kind of memory a higher-end consumer GPU carries, opens up the 30B class. Getting to 70B-class models takes 40 GB or more, which usually means either a workstation-grade card or two consumer cards pooled together. The very large mixture-of-experts models, the trillion-parameter class covered in the previous piece in this series, need 128 GB or more of memory to hold even in compressed form, though as that piece explained, their actual compute cost per answer is much lower than the parameter count suggests. And at the small end, anything under about 4B parameters runs at a reasonable pace on CPU alone, no graphics card required, which is why so many local assistants and small utilities are quietly built on that class.

What quantization is actually doing

None of those tiers would be reachable without quantization. A model is trained and often distributed with each weight stored at 16 bits of precision, which is far more numerical detail than most of the model’s decisions actually need. Quantization takes those same weights and stores them with fewer bits, commonly 4, and in doing so roughly halves the memory footprint or better, at the cost of a quality drop that is usually small enough to be hard to notice in ordinary use. It is the same 70B model, the same weights in relative terms, just written down with less precision per number. This is the mechanism that turns “needs a server rack” into “needs one GPU,” and it deserves a fuller explanation than fits here, which is why it got its own piece.

The tools people actually reach for

Almost nobody hand-writes the code to load a quantized model and manage its memory. Ollama has become the default for people who want a model running locally within a couple of commands, wrapping the messy parts behind a simple interface. Llama.cpp is the engine underneath a lot of that ecosystem, built specifically to run these models efficiently on ordinary CPUs and consumer GPUs. vLLM targets a different crowd, people serving a model to multiple users at once and needing real throughput, and shows up constantly in self-hosted deployments. LM Studio gives the same capability a graphical interface, aimed at people who want to browse, download, and chat with a model without touching a terminal at all.

The temptation is always to start from “what’s the biggest model I can get my hands on,” the way someone might buy a full-size truck for a life spent entirely on narrow city streets, capable of hauling anything, parkable nowhere, and mostly just idling in traffic it was never suited for. Start from the hardware instead. The best model to run is the largest one your actual GPU or CPU handles without straining, because a model that answers in a second and a half is a tool, and a model that grinds out one token per second because it barely fits in memory is a demonstration. For more on how the compression step itself works, see Quantization and On-Device Models, and for the reason a model with a trillion parameters can still be cheap to run if it’s built as a mixture of experts, see the previous piece in this series on why total parameter count alone doesn’t determine the cost of running a model.