Running Models Locally

A hospital can't send patient letters to an outside company. A law firm can't either. For many organisations, the data must never leave the building. The good news is that open language models now run on a good laptop.

• Some data can't leave the building
• Hospitals, lawyers, banks
• Open models run on a laptop

What fits on your machine?

Model size comes down to two numbers. How many weights it has, and how many bits each one uses. Quantising to four bits shrinks a model hugely. Here's a rough guide. Run it. Look at the table. A seven billion weight model needs only around four gigabytes at four bits, so it fits on a normal laptop. Bigger models need more memory, but they answer better.

sizes = {"Tiny": (2, 4), "Small": (7, 8), "Medium": (14, 16), "Large": (40, 48)}

for name, (billion, ram) in sizes.items():
    q4 = billion * 0.6
    print(f"{name:<7} {billion:>3}B weights, about {q4:>4.1f} GB at 4 bits, needs ~{ram} GB RAM")

Tiny      2B weights, about  1.2 GB at 4 bits, needs ~4 GB RAM
Small     7B weights, about  4.2 GB at 4 bits, needs ~8 GB RAM
Medium   14B weights, about  8.4 GB at 4 bits, needs ~16 GB RAM
Large    40B weights, about 24.0 GB at 4 bits, needs ~48 GB RAM

Running a model with Ollama

The easiest tool is called Ollama. It's one command to download a model, and one to chat with it, and it even gives you a local API that works like the cloud ones. Everything stays on your machine. There's no cost per token, and it works with no internet. For a small business, a local model can handle classification and RAG, at no running cost.

ollama pull llama3.2
pulling manifest ... success
ollama run llama3.2 'Say hello in five words'
Hello there, how are you?

Cloud or local?

So which should you choose? Cloud models are stronger, and easy to start with. Local models give you privacy, a fixed cost, and offline use, but you run the hardware. Many teams use both, with local models for sensitive or high volume work, and cloud models for the hardest problems.

Summary