Why reproducibility is now mandatory
Three forces converged in 2025-2026 to make reproducibility a hard requirement for any AI-driven research vendor selling to enterprise:
01 — EU AI Act high-risk deadline (August 2, 2026). High-risk AI systems must "technically allow for automatic recording of events over the system's lifetime" and enable human oversight with the ability to understand the system's reasoning. AI research outputs used in regulated decisions fall inside the scope.
02 — Financial regulators auditing AI-derived investment decisions. PE diligence packs, asset manager research, and insurance underwriting all face the same question: "can you replay how this conclusion was reached?" An unreproducible AI brief is now actively risky to act on.
03 — Buyer sophistication. Procurement and risk teams have learned to ask. "Show me a run that was replayed with identical output" used to be a niche engineering question. It's now a standard procurement question.
The category is splitting. AI research vendors that can answer these questions will keep enterprise contracts. The vendors that can't will lose them.
What reproducibility actually requires
Five engineering properties — none optional:
01 — Pinned model versions
Every LLM call records the exact model name and version (claude-sonnet-4.5-20251022, not "claude-sonnet"). When a model is deprecated or retired, the system records the migration and the pre/post comparison.
02 — Snapshotted inputs
The data the LLM read at the time of the run is preserved. Re-running on the same run_id reads the same snapshot — not the live data that has since changed.
03 — Recorded prompts
The actual prompt sent to the LLM, including all interpolated values, is stored against the run_id. Not "the prompt template"; the actual rendered prompt.
04 — Deterministic decoding where it matters
LLM calls that affect downstream structure (extraction, classification, clustering inputs) use temperature 0 with seeded sampling where supported. LLM calls that produce free-form prose (executive summaries) can be higher-temperature, but the prose itself isn't an input to anything structural.
05 — Source-traceable outputs
Every claim in the final deliverable links back to:
- The
snippet_idit was derived from - The
source_urlof the snippet's origin - The aggregation step that combined it with others
Click a claim, see the evidence. No exceptions.
What Theia does (the production reality)
Every Theia pipeline run produces a record with:
| Artefact | Purpose |
|---|---|
run_id | Globally unique identifier (e.g. theia_uk_20260608_143022) |
step_log | Status (pending / running / completed / failed) per pipeline step, with timestamps |
model_versions | Pinned Claude / GPT versions per call |
input_snapshot | Snapshot table reference for upstream data |
prompt_log | Rendered prompts per LLM call, keyed by snippet ID |
cost_log | Per-step token usage and cost, asserted against budget guard |
failure_log | Failed inserts (BigQuery), timed-out tasks (DataForSEO), Reviewer rejections |
source_traceability | Every claim in L1-L4 outputs links to source snippet_id + source_url |
This is what the deterministic pipeline harness maintains in production. It is not a marketing claim — it is the operating reality.
A board asking "how did we conclude X?" gets:
- The exact
run_id - The exact extraction prompt the LLM was given
- The exact source review/transcript/article the claim came from
- The exact aggregation step that incorporated it
If the data is unchanged, re-running the run_id produces an identical brief. If the data has changed, the diff is recorded and explainable.
Where most vendors fail
Three patterns are common across "AI market research" tools we've reviewed:
01 — Live data, no snapshots. The vendor queries the live web each time. Same prompt, different result an hour later. The run is unreplayable.
02 — Unpinned model versions. The vendor calls "gpt-4" or "claude-3". When the underlying model is upgraded, the output silently shifts. The vendor can't tell the user why.
03 — No source traceability. The output is a confident paragraph. Asking "from which source?" gets a vague "from a synthesis of the data" — i.e. no specific source, by design.
These vendors are shipping research that doesn't pass enterprise compliance review in 2026. Many of them don't yet know this is a problem because the regulatory deadline hasn't bitten yet. It will by Q4 2026.
What buyers should ask
If you're commissioning AI research from any vendor:
- "Show me the run_id of the deck you sent me last quarter."
- "Re-run it now. Are the outputs identical?"
- "For this claim on slide 7, show me the source URL it came from."
- "Show me the budget assertion that prevented this run from costing more than expected."
- "What's your EU AI Act readiness for August 2026?"
Vendors that can answer all five are reproducible AI research providers. Vendors that can answer fewer than three are selling research that won't survive its first enterprise audit.
Strategic implication
Reproducibility is the single biggest hidden quality dimension in the AI research market in 2026. Most buyers haven't learned to ask for it yet. The ones that have are quietly consolidating their vendor list around the small number of platforms that can answer.
For Theia, this is structural. The same engineering choices that produce reproducibility — the deterministic harness, Fixed Entity Architecture, the math-for-connections doctrine, the Writer / Reviewer / Senior Analyst architecture — also produce the cost discipline and quality floor that make engagements profitable to deliver and easy to defend.
Reproducibility isn't a cost centre. It's the operating model.