Skip to main content
On-Prem AI · what I run at home

On-Prem AI
that never leaves the building.

An AI assistant over my own documents and systems, running on hardware I own. Nothing leaves the house, every answer cites its source and lands in a log, and the stack is hardened with the discipline I bring from network security.

Isometric cutaway of a glass-walled, self-hosted secure server room with an AI core glowing at its centre, the data sealed inside the building.
Fully local inferenceSource-cited answersImmutable audit logEU / self-hosted
Why this exists

The models are ready. My data stays home.

Bank statements, receipts, health notes, letters, family plans: the most useful things to ask an AI about are exactly the things I would never paste into a cloud chat. So the assistant comes to the data, not the other way round.

The usual answers are a cloud subscription or an expensive appliance. I took a third route: open-weight models and plain software, on a box I already own.

The platform

Four layers. All self-hosted.

Each layer comes from running code extracted from a fleet of 20+ self-hosted AI applications.

Layer 01

Local model serving

Open-weight models served locally through Ollama, routed via a hardened provider gateway with rate-limiting and metrics. Switch models per task. Nothing calls out to a third party.

Layer 02

Zero-egress retrieval

RAG over your own documents with a source citation on every answer, auto-chunking, dedup, and reranking built in. No source, no claim.

Layer 03

Assistant & agents

A chat assistant grounded in your data ships in the first install. Multi-step agents that take real actions across your systems, observable, resumable, idempotent on retry, already run across the fleet and are packaged into the product next.

Layer 04

Governance, audit & access

Every question and answer written to an immutable log. Role-based access, least-privilege database users per workload, secrets in environment only, capabilities dropped at the container.

The difference

Secured by a security engineer, not bolted on later.

The same hardening I apply to networks sits underneath: capability-dropped containers, per-app network isolation, prompt-injection defence and secrets hygiene from day one, not bolted on later.

A design decision

Why software, not an appliance.

I built it as software you run yourself, rather than a hardware box, for a few deliberate reasons.

An appliance would mean
  • A hardware purchase, sized and paid for up front
  • A box to house, power, and maintain
  • Tied to one appliance's hardware and roadmap
  • Capacity fixed at purchase, used or not
Software means
  • Runs on your hardware, your VPC, or a box you already own
  • Installs with one command, docker compose up
  • No appliance, no capex, no vendor lock-in
  • You own and can audit every layer of the stack
Where it is today

Start with one job, done completely.

The first deployment does one thing end-to-end, properly, before anything is added on top.

  1. Point it at your documents
    A folder of my own files: letters, statements, manuals, notes.
  2. Ask in plain language
    It answers only from those documents. No outside knowledge bleeds in.
  3. Every answer cites its sources
    No source, no claim. You can click straight to the passage it used.
  4. Every exchange is logged
    Question, answer, and sources written to an immutable audit trail.
  5. It all runs locally
    docker compose up on your own box. No external API anywhere in the answer path.

Honest status: the core is real and runs every day on my own machine, as the backbone of the apps in the portfolio. Connectors and agents grow as my own needs grow. It is a working lab, not a product launch.

What I use it for

Where it runs every day.

The same stack sits under the household tools in the portfolio: private data that stays at home.

Finances and receiptsCooking and meal plansTraining logsLetters and documentsVoice dictationCoding with an agent

One of the systems I build.

Curious how a piece of it works, or want to build something similar for yourself? Get in touch.

Get in touch
Questions

Questions about running on-prem AI at home

Does any data leave the machine?

No. Models, retrieval and storage all run locally. There is no external API in the answer path unless I explicitly add one for a task.

Do I need a special appliance?

No. It is software. It runs on an ordinary desktop or a single GPU box.

How is this different from a cloud AI assistant?

A cloud assistant sends the data to a third-party provider. This runs entirely at home. The trade-off is that I host it myself, and for private data that is the point.

Is it finished?

The core is real and runs every day. It keeps growing with the apps built on top of it: a working lab, not a product.

What models does it use?

Open-weight models (Llama, Qwen, Gemma and similar) served via Ollama. They can be swapped at any time, never locked to one vendor or model.

Last updated: September 2026