Turn on airplane mode and open the camera. Point it at a menu written in a language you don’t read, and a translated overlay appears right on top of the text, updating as your hand moves. No bars, no wifi, no round trip to a server anywhere. A year or two ago, a feature like that would have needed a live connection to a data center running a model with billions of parameters. Now it runs entirely inside a device that fits in a pocket. The model didn’t get smaller because someone found a clever shortcut in what it knows. It got smaller because of what happened to the numbers inside it.

Fewer decimals, same shape

Every weight inside a trained model is just a number, and numbers can be stored with more or less precision. A model trained and originally stored with 16 bits per weight is carrying a lot of fine-grained detail in each of those values, far more than most of its decisions actually require. Quantization takes those same weights and represents them with fewer bits, often 8, sometimes 4, and stores the model in that lower-precision form. Fewer bits per number means a smaller file, less memory required to hold the model while it runs, and less arithmetic work for the chip doing the computing, which translates directly into faster responses on modest hardware.

The comparison that fits is compressing a high-resolution photo down to a file small enough for a phone screen. The original file might hold detail sharp enough for a gallery print: individual hairs, faint texture in a shadow, gradations no phone screen could display anyway. Compress it, and you lose some of that fine detail, but none of it was visible at that size in the first place. What you gain is a file that loads instantly and that you can store thousands of copies of without filling up storage, while the original still sits somewhere for the print job that actually needs it. Quantization does the same trade for a model: it discards precision that was rarely doing meaningful work, in exchange for a version light enough to live on a phone or a laptop instead of a server rack.

Why this matters more than it looks

It would be easy to file quantization under routine engineering housekeeping, the kind of detail that only shows up in a technical changelog. But where a model runs changes who pays for it and who sees the data flowing through it. A model served from the cloud costs its operator something for every query answered, computing time, electricity, server capacity, and that cost has to be recovered somehow, usually through a subscription or metered API access. A model running on the device costs the operator nothing per query, because the phone’s own chip is doing the work. There is also no server log of what was asked, because the request never left the device to begin with. Quantization is what makes that shift possible for a meaningful class of features. It’s the technical precondition for a company to stop charging per use and stop needing to see the query at all.

A decision wearing an engineering costume

None of this is only about squeezing a model onto smaller hardware. Choosing to run on-device is a decision about what kind of business gets built around the model and what kind of relationship it has with the person using it. On-device means no per-query server bill and no built-in meter for tracking usage the way a hosted API naturally provides one. It also means the user’s data has no reason to leave the device, because there is no server on the other end waiting to receive it. The engineering story, fewer bits, smaller file, faster response, is real and worth understanding on its own terms. But the choice to ship that way in the first place is a business call dressed up as an optimization, made by someone who has already decided how they want to get paid and how much privacy they want to be able to promise.

For a related way of getting a smaller, cheaper model, see Distillation: How a Small Model Learns From a Big One, which trains a compact model on a larger one’s outputs rather than compressing an existing model’s own weights. And for the server-side problem that running on-device sidesteps entirely, see Inference in Production: Serving a Model to Millions.