CompliBot

Security questionnaire RAG — Excel in, AI answers out

Built by Rogue AI · Security-questionnaire RAG responder (Excel/CSV in, AI-drafted answers out) · Self-hosted

Built through 2026 as part of the Rogue AI fleet.

CompliBot — Security questionnaire RAG — Excel in, AI answers out

The problem

Vendor security questionnaires — SIG Lite, CAIQ, VSA, NIST CSF and the endless bespoke spreadsheets — are a recurring tax on any company that sells software. The answers barely change between deals, yet someone re-types them by hand into a new Excel file every time, hunting through old questionnaires and policy documents for wording they already wrote months ago. The work is repetitive, error-prone, and pulls security people off real work to copy-paste.

What I built

CompliBot is a Next.js 16 / React 19 app that ingests an Excel or CSV questionnaire, parses out the questions, classifies each by security domain, and drafts an answer using a retrieval-augmented pipeline grounded in your own knowledge base. You build that knowledge base by uploading past questionnaires and policies, or through a guided AI interview that turns tacit answers into structured entries. Each draft comes back with a confidence score and its source passages; reviewers approve, edit or reject side-by-side with keyboard shortcuts, then export the completed sheet back to Excel. Around the core loop sit the things a real product needs: multi-tenant organisations with role-based access, a template library, bulk upload, an immutable audit trail, gap analysis across 19 security domains, and Stripe-backed plan tiers.

Architecture

Hybrid retrieval over pgvector

Knowledge entries are embedded into a 768-dimension vector column in PostgreSQL using pgvector with an HNSW cosine index. SearchService runs semantic vector search alongside keyword matching and fuses the two result sets with reciprocal rank fusion, so exact control IDs and acronyms surface even when the surrounding wording differs.

Self-hosted generation via Ollama

Both embeddings (nomic-embed-text) and answer generation (qwen2.5:7b) run on a shared local Ollama instance, so no questionnaire text or knowledge-base content is sent to a third-party API. An optional provider switch can route generation through a host CLI bridge instead, but the default stays local.

Confidence-gated answer generation

AnswerGenerationService assembles the retrieved passages into a grounded prompt and attaches a confidence score to each draft. A per-organisation threshold decides whether an answer is auto-approved or flagged for human review, keeping a person in the loop on anything the system is unsure about.

Spreadsheet in, spreadsheet out

Parsing uses ExcelJS and Papa Parse to read the messy real-world Excel and CSV files security teams actually receive; export writes the approved answers back to XLSX (or a text/PDF report with org branding and domain coverage), so CompliBot fits the format the rest of the deal already runs on.

Multi-tenant data model with audit trail

A 20-model Prisma schema separates organisations, users, subscriptions and usage, with role-based access between them. An append-only AuditLog records action, entity, user, IP and timestamp for every change — the kind of evidence a compliance reviewer expects to see.

Hardened, isolated Docker stack

App, pgvector-enabled PostgreSQL and Redis each run in their own container on a dedicated /24 network with all ports bound to localhost. Containers drop all Linux capabilities, run read-only with no-new-privileges, use tmpfs for scratch space, and pin every base image by SHA256 digest.

Tech stack

Next.jsPostgreSQLpgvectorOllamaDocker

What broke first

▸
Retrieval quality decides answer quality. Pure vector search misses exact control IDs and acronyms (SOC 2, ISO 27001, CAIQ), so a hybrid of pgvector cosine similarity plus keyword matching, fused with reciprocal rank fusion, beat either approach alone for questionnaire text.
▸
A confidence score is only useful if it gates a human. Surfacing a per-answer confidence and an org-level threshold — auto-approve above, flag for review below — turned the AI into a drafting assistant rather than an unaccountable autoresponder, which is the only posture a security team will accept.
▸
Keeping the model and the embeddings on a self-hosted Ollama instance means the knowledge base — past answers, policies, internal control language — never leaves the box. For a tool whose entire input is sensitive vendor-security material, that data-residency property is the feature, not a footnote.

Outcome

CompliBot works end-to-end as a portfolio demonstration: upload a real questionnaire spreadsheet, watch it parse and classify the questions, get grounded draft answers with sources and a confidence score, review them, and export a finished sheet — all on self-hosted infrastructure where the data never leaves the machine. It is a concrete demonstration of a security-domain RAG pipeline, confidence-gated human review, and a properly hardened multi-tenant Docker deployment, rather than a slideware concept.

Honest limits

Self-hosted and built solo as a portfolio demo. It runs as a local Docker lab — the earlier public-demo VPS was retired — so there is no large production user base behind it, and answer quality depends entirely on the quality and coverage of the knowledge base you load. The default local model (qwen2.5:7b via Ollama) drafts plausible answers but still needs human review before anything ships to a customer; the Stripe billing, plan tiers and multi-tenant scaffolding exist to make it a realistic product, not because real money flows through it.