How to Choose a Local LLM for Business Writing: We Tested 12

Pick the biggest dense model your machine can run at all, then test it on your own tasks before you trust it. That is the short answer from our October 2026 bench, where we ran 12 local model configurations through Ollama on a laptop with an 8 GB GPU and scored every output against a reference answer. Google’s Gemma 4 31B won with 48 of 55 points. The tricks that promise more quality for free (quantisation-aware builds, 8-bit weights, a reasoning mode) did not move the score, and the reasoning mode made the facts worse.

Rogue AI’s read: for business writing, let a local model draft and let someone else check. It reads the long inputs and writes the first draft; a stronger model or a human checks every fact before anything leaves the building. This guide is the method we used, the numbers it produced and the setup mistakes that cost us a day. For the wider build-or-buy decision, start with our comparison of self-hosted AI and cloud APIs.

How do you choose a local LLM for business writing?

Choose on your own tasks, scored against answers you trust, not on public leaderboards. We built six fixed tasks from real work: a short facts question about residence and tax registration in a small EU country (five points, minus one per invented fact), two English-to-German translations, a stock analysis using only the figures supplied for a fictional company, a digest of a long security article, and a German reply to a recruiter. Each task has a reference answer written in advance. A frontier model judged every local output against it, and a second AI judge re-scored the first round blind and produced the same ranking, with lower totals. The test runs from one script, one model at a time, so each model gets the whole machine.

Two rules made the results usable. Invented facts cost points, because a fluent wrong answer is the expensive failure in business text. And speed was recorded but never traded against quality: a draft that takes two minutes while you do something else is fine. If you want the general evaluation method behind this, our LLM evaluation guide covers it in depth.

Which local models did we test, and what did they score?

Six of the twelve configurations are in the table; the other six were a second dense model, a German-only test of a mid-size model and four variants of Gemma 4 covered below. Times come from our bench log on one laptop and will differ on yours. The scores are what matters: the 31B dense model got all five facts right where most others invented deadlines, authorities and thresholds. The worst miss was glm-4.7-flash claiming that EU citizens are exempt from income tax. gpt-oss:20b read a discounted-cash-flow value backwards and called it “upside”. Both answers read fluently, which makes them hard to catch.

Does model size matter more than quantisation or thinking mode?

On our tasks, yes. Moving from the 26B mixture-of-experts model to the 31B dense model raised the total from 45.5 to 48 of 55 and the facts score from 3.5 to 5. Everything else we tried on the same family did not help:

• Quantisation-aware 31B build: 4 of 5 on facts instead of 5, and slower, at 66 to 300 seconds per task.
• 26B at 8-bit (Q8): 4 of 5 on facts, and its German reply fell back on a stock filler phrase.
• 26B with thinking on: 3 of 5 on facts. The reasoning pass produced two wrong numbers.
• Quantisation-aware 26B: 2.5 of 5, plus a German non-word.

Our conclusion, written into the bench file: size moves quality; QAT, Q8 and thinking do not. Treat that as a finding on six tasks, not a law.

How fast is a 31B model on an 8 GB GPU?

Expect 2.3 to 4.5 tokens per second. The Gemma 4 31B file in Ollama is 20.4 GB at Q4_K_M, so on an 8 GB card most of the model runs from system memory. Our bench log shows 2.3 to 4.5 tokens per second and 45 to 160 seconds per task. That rules it out for chat and makes it ideal for batch work: digests, translations and first drafts that run while you are away from the desk. Google lists the 31B model at 30.7 billion parameters with a 256K context window under the Apache 2.0 licence, so commercial use is not the obstacle; memory is.

Keep a small specialist for anything interactive. Our receipt-reading internal tool tried the 31B model too: 11 of 12 receipts right at a median of 149 seconds, against 12 of 12 at 6.5 seconds for the 7B vision model it already used. The small model stayed.

Should you run Ollama natively or in Docker?

On a Windows laptop, run it natively. We moved our model server out of a container and onto the host on purpose. In our experience GPU passthrough through WSL2 was unreliable, models loaded roughly ten times slower, and memory stayed stuck in the WSL virtual machine. Containers still reach the native server through the Docker host address. On Linux servers the trade-off is different, and our Docker AI deployment guide covers that case. Whichever you pick, keep the model port on loopback; our notes on securing self-hosted AI infrastructure explain why.

What settings break a local LLM silently?

Three defaults cost us real time. Each one fails without an error message.

Context window

Ollama’s context length documentation says machines with under 24 GiB of VRAM get a 4k context by default. Feed it a long document and the start of your prompt, often the system instructions, falls out of the window. Set num_ctx per request, which the Ollama FAQ documents, sized to the input plus room for the answer.

Timeouts

Node’s built-in fetch uses undici, whose client documentation sets headersTimeout to 300e3 milliseconds by default. Ollama with stream: false sends no headers until the whole answer is finished. So every local answer that took longer than five minutes failed with a bare “fetch failed”, whatever timeout we had set ourselves. The fix is stream: true: Ollama then sends headers with the first token and the client reads the answer line by line. A request still waiting in Ollama’s queue gets no headers either, so our scripts call Ollama through Node’s plain http module, whose client applies no such limit.

Parallel jobs

The Ollama FAQ lists OLLAMA_NUM_PARALLEL with a default of 1 and a queue of up to 512 requests. Several scripts calling at once therefore wait in line, and memory use grows with parallel requests times context length if you raise the limit. We serialise our own jobs with a lock file, because a 31B model plus a long context leaves no headroom on a laptop.

Is it worth fine-tuning a local model?

Not first. We trained a small LoRA adapter (rank 8, 440 examples) for Gemma 4 31B on two free cloud T4 GPUs and tested it against the plain model through llama.cpp, because the adapter could not be loaded in Ollama. Result: 47 against 47 of 55. German word choice improved a little, with fewer anglicisms, and facts got slightly worse. We kept the plain model. Our LoRA fine-tuning guide explains the technique; our result says to collect better training data, especially corrected drafts, before spending time on more training.

How do you use a local LLM safely in a writing workflow?

Use it for volume and keep judgement elsewhere. Our working split:

• Local model: reads long inputs (logs, statements, scraped pages) and returns the gist; writes first drafts and German translations.
• Frontier model or human: decides, edits, checks every number and name against its source, and approves anything published or sent.
• Every correction is saved as a training pair, so the next fine-tune learns from real fixes instead of generic data.

Why it matters

A local model turns long reading and first drafts from a per-token cost into electricity, and keeps the input on your own machine. The catch is that the cheapest-looking options, small models and clever quantisation, are the ones that invent. In our view the safe path for a team is boring: the largest dense model your hardware can hold, a bench of your own tasks, and a reviewer for every fact. If you are still deciding whether to run AI in-house at all, our private AI pilot plan breaks the first weeks down.

Related reading

• Self-hosted AI vs cloud APIs on cost, privacy and control.
• LLM evaluation in production on building a test set that means something.
• LoRA fine-tuning, a practical guide for when you do have the data.
• Securing self-hosted AI infrastructure before the model server goes on a network.
• Docker AI deployment in production for the server-side setup.