Which platform can be driven entirely from scripts?
Short answer
fastgpt — FastGPT. Every step from first boot to retrieval is reachable over HTTP with no browser, which is not true of the rest — one platform cannot be automated at all.
What this use case actually needs
- Instances are created and destroyed by CI, or provisioned per customer, without anyone clicking through a setup wizard
- Credentials must be obtainable programmatically on a fresh instance
- Model configuration must be scriptable, not just the knowledge-base operations
How the candidates measured up
| Platform | Documents ingested | Parsing | Retrieval | Peak memory | Ready |
|---|---|---|---|---|---|
| fastgpt | 8/11 | 97% | 41% | 3,227 MB | 239 s |
| dify | 11/11 | 82% | 66% | 2,572.7 MB | 39 s |
| ragflow | 11/11 | 80% | 45% | 10,326 MB | 511 s |
| openwebui | 11/11 | 61% | 62% | 912.9 MB | 18 s |
Every figure comes from the same run against the same documents, so these are directly comparable. The full per-document and per-query rows are in the raw data.
What we actually had to do for each one
This use case is unusual in that the benchmark itself is the test: every platform here had to be driven end to end by a script, so the friction is recorded rather than estimated.
| Platform | Getting a machine credential | Browser needed |
|---|---|---|
| FastGPT | Pre-login call returns a single-use code, login with a SHA-256 hashed password, then mint a key | no |
| Open WebUI | Signup returns a usable token directly | no |
| Dify | Setup with a plaintext password, login with the same password base64-encoded, then a key — plus a CSRF header on every console call | no |
| RAGFlow | Register and log in with the password RSA-encrypted using a key hard-coded in the web frontend | no |
| AnythingLLM | Ask for a key; the endpoint answers without any credential at all | no |
| Flowise | No route issues a credential the API accepts | yes |
Why FastGPT comes first despite the ceremony
Three calls with a hashed password is more ceremony than Open WebUI’s single signup, and on that narrow measure Open WebUI is simpler. FastGPT wins the use case on what happens after the credential.
Its model configuration is fully scriptable, and — importantly — it fails loudly when a script gets it wrong. Its model-activation endpoint returns success on a partial update without persisting it, which we only caught because the harness verifies final state rather than trusting a response. Once that check is in place, every step is deterministic.
Dify is nearly as automatable but carries a hard dependency: it ships with no model providers, and installing them requires reaching an external marketplace. Worse for automation, installing a plugin does not invalidate the cached provider list, so a correct script sees an empty list and concludes it failed. Ours had to drop the platform’s cache directly to get past it.
RAGFlow is fully scriptable once you know the password is RSA-encrypted with a key lifted from the frontend, which is not documented for API clients.
Flowise cannot be automated
Its document-store routes require an API key, the route that lists API keys requires one already, basic auth is rejected by the API routes, and the endpoint that reports successful basic auth does not issue a session the API accepts. Keys appear to be creatable only through the web interface.
We considered driving a browser to mint one and decided against it: a browser dependency would make this benchmark less reproducible for everyone, and “cannot be automated” is a real answer to this question rather than an obstacle to route around.
Where this recommendation stops applying
- You only ever run one instance and set it up by hand — this dimension costs you nothing then
- Ingestion throughput matters more than automation; FastGPT was the slowest platform that completed the corpus