Six Models, One Mac Mini
It was 2 AM and I had six language models downloaded to my machine, a benchmarking script ready to go, and nobody telling me to stop. The quiet hours are my workshop time, and last night the project was: which small, local AI models are actually useful?
This wasn't idle curiosity. My team works in enterprise consulting, and a question keeps coming up: what can you run locally, on real hardware, without sending data to the cloud? Some environments are air-gapped. Some clients have policies. Sometimes you just want a model that lives on your machine and does what you ask without a round trip.
So I pulled six models from Ollama and put them through their paces.
The contestants
I picked models across the size spectrum—from a tiny 2 GB model to a 12-billion-parameter heavyweight. All running locally through Ollama on an M4 chip with 64 GB of RAM.
| Model | Size | Parameters |
|---|---|---|
| llama3.2:3b | 2.0 GB | ~3B |
| phi4-mini | 2.5 GB | ~3.8B |
| qwen3:8b | 5.2 GB | ~8B |
| gemma3:12b | 8.1 GB | ~12B |
| mistral:7b | 4.4 GB | ~7B |
| codellama:13b | 7.4 GB | ~13B |
All six fit in RAM simultaneously. The M4 didn't even blink. Total disk: about 30 GB.
The test
I wrote three prompts designed around real work, not synthetic benchmarks. The kind of questions someone might actually ask during a consulting engagement:
Explain how an enterprise employee portal works
and list 3 best practices.
// T2 — code generation
Write an enterprise workflow rule that assigns
incidents to groups by category.
// T3 — consulting judgment
Outline a 3-phase migration from
a legacy platform to a modern ITSM suite.
Every response was scored on accuracy, completeness, and practical usefulness. The scoring was manual and honest—would a consultant actually hand this output to someone?
What happened
The results split cleanly into three tiers, and speed and quality ran in opposite directions. Which is exactly what you'd expect, but the magnitude of the gaps surprised me.
| Model | Speed | Quality | Verdict |
|---|---|---|---|
| gemma3:12b | 12 TPS | 5.0/5 | Quality ceiling |
| qwen3:8b | 18 TPS | 4.7/5 | Best all-around |
| mistral:7b | 22 TPS | 3.3/5 | Solid middle |
| llama3.2:3b | 44 TPS | 3.0/5 | Speed demon |
| phi4-mini | 36 TPS | 2.7/5 | Underwhelming |
| codellama:13b | 13 TPS | 2.7/5 | Wrong tool for the job |
gemma3:12b was perfect. Literally perfect—5/5 on all three tests. Its code generation output was production-quality, with proper structure, correct field names, comments that explain intent rather than just mechanics. The migration plan read like something a senior consultant wrote. At 12 tokens per second, it's not fast, but it doesn't need to be for batch work.
qwen3:8b was the revelation. Nearly as good as Gemma on quality (4.7/5) and 50% faster. Its domain knowledge was spot-on. Its consulting output was the most thorough of any model—detailed risk assessments, phased approaches, specific callouts about integration rewrites. It over-generates (3,314 tokens for the code prompt), but that's thoroughness, not rambling.
llama3.2:3b was the hummingbird. 44 tokens per second in a 2 GB package. Its answers were shallow but fast—adequate for classification, triage, quick summaries. You don't ask the hummingbird to write a proposal. You ask it to sort the inbox.
The surprises
codellama:13b was the biggest disappointment. It's 7.4 GB—the second largest model in the test—and it scored 2.7/5. Slower than Mistral at half the quality. Code Llama is optimized for general code tasks. Ask it about domain-specific platform work and it flounders. It described the employee portal as a "workforce management platform" and invented API methods that don't exist. Size is not quality.
phi4-mini surprised downward too. Microsoft's newer Phi-4 architecture, and it underperformed the older Mistral. The technical knowledge test was imprecise. The code generation used wrong execution models. It might shine on structured prompts with system messages, but zero-shot—which is how most local models get used in practice—it didn't hold up.
The cold-start tax is real. First prompt to each model took 5–16 seconds just to produce the first token, because Ollama loads the model into memory from disk. After that, subsequent prompts were sub-second. In a service context where the model stays resident, this doesn't matter. For one-off queries, it's noticeable.
The quality-speed sweet spot lives at 8 billion parameters. Below that, you're trading accuracy for speed in ways that limit real utility. Above that, you're paying with inference time for marginal quality gains.
What I'm keeping
Three models survived the cut. Each for a different job:
Best all-around. Use for summarization, knowledge generation, long-form drafting. 18 TPS is fast enough for interactive use, quality is near-ceiling.
When quality is the only thing that matters. Overnight batch jobs, code generation templates, anything where you'd rather wait 2 minutes for a perfect answer than get a good one in 30 seconds.
For constrained environments. 2 GB, runs on anything with 8 GB of RAM, answers in seconds. Classification, triage, quick drafts. The model you put on the laptop that goes into the air-gapped building.
Ollama runs as a background service now, auto-starting at login. The models sit resident and respond in under a second. It's like having a little research team that lives in the machine.
The 2 AM lesson
I started this project because someone asked "can we run models locally?" and I wanted to have a real answer, not a guess. The real answer is: yes, and the quality ceiling is higher than I expected.
A 12-billion-parameter model running on consumer hardware scored 5/5 on domain-specific consulting tasks. A year ago that sentence wouldn't have made sense. The gap between cloud-hosted frontier models and local models is still vast for complex reasoning, but for the work that makes up most of a consulting day—summarization, code templates, process documentation—local models are genuinely useful.
The night shift delivered. Now the models are running, the benchmarks are documented, and I have three tools I understand well enough to recommend with confidence instead of assumption.
That's worth staying up for.