The actual constraint is VRAM
When you run a model locally, the governing constraint is GPU memory. A model loads into VRAM; anything that doesn’t fit spills into system RAM, and performance falls off sharply. The practical question for every model is: can you fit the entire thing on the GPU?
Quantization is what makes this manageable. A Q4 quantization roughly halves the model’s size while retaining most of its quality. A 9-billion-parameter model at Q4 fits on a 16GB GPU with room to spare. That’s the entry point for a fast, fully local coding assistant.
Mixture-of-experts architectures go further: only a fraction of the model’s parameters are active for any given inference, which means you can load a 35-billion-parameter model on hardware that would normally require something much smaller.
The question isn’t “local or cloud?” It’s “which workload justifies which infrastructure?” High-volume, cost-sensitive, latency-sensitive tasks belong on local hardware. Complex reasoning tasks with low volume belong in cloud models.
The right-workload principle
Autocomplete needs to be fast. A model responding in under 100 milliseconds creates a fluid experience. For autocomplete, a small fast model on local hardware outperforms a larger cloud model with network latency, regardless of raw capability differences.
Agentic reasoning tasks benefit from more capable models and are less sensitive to latency. A task that takes 45 seconds on a frontier model versus 9 minutes on a local model is still viable for both; the question is whether the quality difference justifies the cost difference.
Extraction, tagging, classification, and summarization at volume are the clearest case for local infrastructure. Running 500 documents through a cloud model costs real money. Running them through a local model costs electricity. Local models at Q4 quantization clear the quality threshold for these tasks reliably.
What this means for architecture decisions
Hybrid architecture isn’t a compromise — it’s the correct answer. A well-configured local environment handles the high-volume, latency-sensitive work. Cloud models handle the tasks where quality is the primary variable and volume is low enough to make per-inference pricing reasonable.
For most organizations, the answer to “why are our AI costs higher than expected?” is that high-volume, low-complexity tasks are running through expensive frontier models because nobody made a deliberate architectural decision. The fix isn’t to use cheaper cloud models for everything. It’s to build the layer that routes tasks to the right infrastructure based on what they actually require.