Methodology
Everything on this site comes from deploying the software and measuring it. This page states exactly how, including the parts that weaken the result.
The short version
One host, one platform at a time, one shared embedding model. Facts are planted in real public documents; a platform scores by whether those facts survive its parser and whether its own retrieval can find them again. No language model judges any result.
Why no model scores anything
A benchmark where a model writes the questions, answers them and marks them is grading its own homework: a confident wrong answer gets full marks. So the answer to every question here is a string lifted verbatim from the source document, chosen by a deterministic rule and verified to occur exactly once in that document.
A model is used in exactly one place — phrasing a natural question around a fact that was already extracted. It never decides what the answer is and never sees a platform's response. A badly phrased question is simply harder for every platform equally; it cannot turn a wrong answer into a right one.
Controls
- One embedding model. Every platform is pointed at the
same locally served
bge-m3. Differences in retrieval therefore reflect chunking and ranking, not embeddings. - One platform at a time. Memory measured while six stacks compete for one machine measures the machine.
- Timing includes the image pull, because that is the wait an operator actually has on a clean host.
- Published images and official compose files wherever upstream provides one that starts. Where a compose file was written for this benchmark instead, the file says why at the top and the reason is repeated on the platform's page.
What each dimension measures
| Dimension | Weight | How it is measured | Model involved |
|---|---|---|---|
| Setup and time to first knowledge base | 20% | Seconds from compose up to a served request, plus the manual edits required before first start | none |
| Parsing fidelity | 20% | Share of planted facts found in the chunks the platform stored | none |
| Retrieval accuracy | 15% | Share of questions whose answer appears in the platform's own top-k context | none |
| Workflow orchestration | 15% | Feature audit against published documentation, stated per platform | none |
| Self-host footprint | 10% | Container count and peak memory sampled every 5s during ingestion | none |
| Ecosystem and integrations | 10% | Count of first-party integrations from public documentation | none |
| Licence and commercial terms | 10% | Reading the licence text | none |
The weighting is a judgement and reasonable people would weight it differently. That is why the raw per-item results are published: anyone who disagrees can recompute the table with their own weights.
The corpus
11 public documents — arXiv papers, SEC filings and US tax forms — each listed with its source URL. Nothing was written for the test: synthetic documents are too uniform to separate one parser from another, and a reader cannot verify them.
Documents above 1300 KB were excluded. Those were large SEC HTML filings whose bulk is markup rather than content; a single 11 MB filing produces more chunks than forty other documents combined, and leaving them in would have made the benchmark a measurement of four outliers. The uncapped manifest is published alongside the capped one.
What this cannot tell you
- Two platforms cannot be measured exactly on parsing. AnythingLLM and Open WebUI expose no API listing a document's stored chunks, so their figure comes from exhaustive retrieval and can only under-report. Those numbers are marked, and a missing fact there means not observed, not dropped.
- Embedding ran on CPU. That affects throughput, not quality, and affects every platform equally — but the latency figures are not what you would see with a GPU.
- One run per platform. Latency percentiles are indicative, not rigorous.
- One hardware configuration. The spec is recorded in the run manifest; results may differ elsewhere.
- Retrieval is measured, not answer quality. Whether a platform then writes a good answer from the retrieved context is a property of the language model, which is held constant here on purpose.
Reproducing it
The harness is open source. On a Linux host with Docker, roughly 32 GB of
RAM and 100 GB of disk, bash scripts/reproduce.sh rebuilds the
corpus from its public sources and re-runs every platform. Expect several
hours; CPU embedding is the bottleneck.