No error. No exception. No alert. Every log green, every stage reporting success — and a question your system used to answer correctly now comes back with nothing.
| before the upgrade | after | |
|---|---|---|
| What notice is required to terminate the agreement early? | ninety days taken from a passage the system retrieved and read |
not stated in retrieved evidence no passage it read mentioned a notice period |
The change between the two runs was routine: an improvement to how documents are cut into passages before they are indexed. One line of configuration. The sort of change that ships on a Tuesday afternoon with a one-line commit message.
The documents did not change. The question did not change. The model did not change. And the second answer is not wrong — the system correctly declined to state something it had no evidence for. That is the part that makes it dangerous: an honest refusal and a silent regression look identical from the outside.
Every stage in that pipeline logged success, because every stage succeeded. Documents were read, converted, cleaned, cut, embedded, indexed, searched, and answered. Nothing in that sequence has anything to say about the fact that one of its steps quietly moved the sentence you needed out of reach.
So the investigation becomes archaeology. Someone re-runs the old configuration, if it still exists. Someone diffs a database. Someone reads documents. In the meantime nobody can tell you whether this affected one question or ten thousand — and that, not the fix, is the expensive part.
A log is testimony: it tells you what a system says it did. A record is evidence: it lets someone else check.
If each stage of the run records what went in and what came out — as a digest, not as a copy — then the two runs can be laid side by side and compared stage by stage. The first stage whose output disagrees is the fault site. Everything above it is identical; everything below it is consequence, not cause.
That comparison does not get harder as your corpus grows, because you are comparing stages, not documents. The number is set by the shape of your pipeline, not the size of your data — which is why the report opposite locates the fault in nine comparisons, and would still be nine at a thousand times the volume.
Two things fall out of the same records without reading a single document: which stage broke it, and how far it spread.
The report shows one document in full under both runs. Before the upgrade it was cut into two passages, and the second one — the one containing "ninety (90) days after notice is given" — was retrieved and read.
After the upgrade the same document is cut into ten passages. That sentence is still there. It is still in the index. It carries the same digest, so it is provably the same text. It sits at position nine of ten — and the search never reached it.
Nothing was lost. Nothing errored. The evidence was in the building the whole time, and the system answered as though it were not.
You can see that on screen: the same digest, present in both runs, marked retrieved in one and not in the other. That is the difference between knowing something went wrong and being able to point at it.
Everything above is a handful of test documents, deliberately small so that anyone can rebuild them. The same method was then run over 47,430 real SEC filings — 19,282,360,450 bytes, with the same one-line change to how documents are cut. This is what it printed, unedited:
stage baseline candidate verdict -------------------------------------------------------------------- document in 0211c293efe0 0211c293efe0 same root converted 0211c293efe0 0211c293efe0 same root cleaned a6ad8533e58a a6ad8533e58a same root split f46e4fd5c847 5973d5fe20b7 FIRST DIFFERENCE embedded 3317f5ba98bd 8879c65c2fe3 downstream indexed 5e023c0999bf 72704d9fe7d5 downstream retrieved cff666115d20 006e8fded426 downstream prompt built 16e73d452a19 64a63d68cc3d downstream answer 9cc0cc2a5861 9cc0cc2a5861 same digest FIRST DIFFERENCE -> 'split' (root/digest comparisons: 9) documents 47430 chunks 3380256 -> 8995469 BLAST RADIUS (chunk-index digests only, no chunk text read) documents whose chunk set moved 47412 of 47430
root/digest comparisons: 9 — the same nine as on thirteen documents, because the
comparison is over stages, not over documents. Nine on thirteen files, nine on forty-seven
thousand, and nine again on a second, larger change to the same corpus.
One configuration line moved the passages of 47,412 of 47,430 documents — 99.96%. Eighteen were provably untouched. That figure comes from comparing recorded fingerprints, as the output says: chunk-index digests only, no chunk text read. On nineteen gigabytes, that is minutes rather than a week, and it is the number the room actually wants: not "something changed" but "this much changed, and these did not."
Both runs ended on the same digest because both gave the same answer: not stated in retrieved evidence. The answer is stable here because it is a refusal — the evidence retrieved was uniformly useless on both sides, and the system honestly declined twice. It is not evidence that the system is robust, and we will not let it be read that way. What it does show is that the three layers move independently: the configuration differed by one line, the artifacts by 2.66×, and the answer not at all.
Three constants instead of one line: chunks went from 3,380,256 to 17,136,566 — 5.07× — and 47,429 of 47,430 documents moved, with exactly one left untouched. First difference still at the same stage, still nine comparisons. The same "minor" change is quiet on one corpus and loud on another, and nothing but the record tells you which one you got.
And the seam in this, before you find it yourself. The vivid failure at the top of this page — ninety days becoming not stated — happened on thirteen documents. On forty-seven thousand, what we demonstrated was reach, not a changed answer: the answer was a refusal before and after. So the small run shows you the failure mode and the large run shows you that locating it and measuring it stays cheap at scale. Neither one shows both. We would rather you read that here than notice it later and wonder what else was arranged.
The design rests on a claim worth testing: that a recorded stage can be rebuilt rather than stored forever. So we ran the same stage on three machines — a Linux container, a Windows laptop, and a rented Linux box with a different minor version of the numerical library — and got byte-identical fingerprints on all three. The rebuild claim holds.
Then the same test on a graphics card, over 300,000 passages, changing nothing but how many items were processed at a time:
| what changed | largest numeric difference | same documents retrieved? | consequence |
|---|---|---|---|
| nothing — run it twice | 0 | yes | reproduced exactly |
| one item at a time | 0.0000000596 | yes | none |
| 256 items at a time | 0.0000000745 | no | different evidence read |
| half precision | 0.000141 | no | different evidence read |
Read the third row again. A difference in the eighth decimal place — far below anything a person would call a change — was enough to reorder which passages came back as most relevant. The system then read different documents. Nothing errored. Every log was green.
Production inference batches dynamically: how many items are processed together depends on how busy the server is. So the same question, asked at two different times of day, can rest on different evidence — and nothing in an ordinary system records that it happened.
Two limits on that, stated here rather than discovered by someone else. The test used a fixed random projection as a stand-in, not a production embedding model, so the mechanism is established and the frequency in a real system is not. And the visible answer did not change in this test either — again because the retrieved evidence was useless and the refusal was honest both times.
Reproducibility is not a property of "the code". It is a property of a specific combination of code, configuration, numeric precision and batching — which is exactly why it has to be recorded per run and measured, rather than assumed.
Partly, and here is the boundary. The ladder on this page is a linear chain — one document in, one answer out, stages in a row. Real pipelines are usually not that. If yours has multi-query fan-out, a reranker, a router, a cache, retries as separate attempts, loops, or tool calls that reach outside the system, then it has a shape, and the comparison above assumes a sequence.
The record format says so in its own specification rather than in a footnote here. Its scope clause covers linear chains; fan-out, fan-in, branches, loops, compensation, exactly-once versus at-least-once delivery, out-of-order arrival and disagreeing clocks are named as a recognised requirement and the largest thing it does not yet do. Two consequences that are in the format today and are worth more than a promise:
So on a pipeline like yours, today, you get a correct answer over the parts that are a chain and an explicit refusal to conclude across the parts that are not. That is less than you want. It is considerably more than a tool that would have given you a confident answer anyway.
One shard of that forty-seven-thousand-document run holds 5,932 receipts over 1,483 documents — four stages each — totalling 8.4 MB. Mean receipt 1,411 bytes, tightly distributed: smallest 1,354, largest 1,476. That is about 5.6 KB of record per document.
Read that as a ratio against your own documents, not ours. The cost is fixed per document per stage, so on SEC filings averaging roughly 400 KB it is about one and a half percent of the bytes it describes — and on two-kilobyte support tickets the same 5.6 KB would be several times the document. Chunk-index entries scale with chunk count rather than bytes, so a chunker that produces more passages costs more record. If your corpus is short documents, measure this before you budget for it.
Time: not isolated as its own number, and we will not invent one. What is measured is the whole ingest with recording switched on — 47,430 documents in 78 minutes 40 seconds, about 100 ms per document, and I/O bound on network storage rather than CPU. Recording was not the bottleneck in that run; that is not the same as saying it is free in yours.
A receipt carries digests, not content. Each stage records what went in and what came out as a hash, along with the stage identity and its configuration digest — not the document, not the passage, not the answer. That is why the comparison can run over nineteen gigabytes without opening anything, and it is the answer to the first question your risk function will ask about residency and retention.
The report is a different object. The page linked above shows passage text, which does not come from the receipts — it comes from the chunk store the run already had. The digests prove that a passage is the same text in both runs; reading the text requires your own store, on your own side of the wall. Keep the receipts and discard the store and you can still locate a first difference and measure its reach; you simply cannot render the passage.
A checker needs nothing from you but the records. A conforming verifier requires no network, no key, no account, no licence and no live third party, and never modifies what it checks. An auditor with the record files and standard tools can reach the same conclusions offline, without credentials to your systems and without asking us anything.
Checkable by you, today, with no trust in us: the record format, the sample records, the reference verifier and ten rejection vectors a conforming verifier must refuse — all published, all runnable with standard tools, on the samples page. The report linked above is assembled from runs of that public fixture set.
Not checkable by you today: the forty-seven-thousand-filing figures on this page. They are measurements we made, read back out of the runs' own records rather than transcribed from a screen, and the records still exist. Rebuilding them yourself means re-fetching nineteen gigabytes of public filings, which is possible and is not a small afternoon. We would rather say that plainly than let the size of a number do work that its checkability has not earned.
What exists today: the record format, published with sample records, a reference verifier and ten rejection vectors anyone can run; and the method — comparing two runs' records to locate a first difference and measure its reach. The report is that method, performed on real runs and assembled into a page by a script.
What does not exist yet: a product you click, and — the question you are actually asking — a library that emits these records from your pipeline for you. The format is specified well enough that you could emit conforming records yourself today, and the verifier that would judge them is published; but the instrumentation that would do it for you is not shipped, and this page is not going to give it a date it has not earned. The three commands that would run the comparison in one step are specified and not shipped either. There is no dashboard, no login, no service. We would rather tell you the method works now than promise you a button.
The exhibit Read the report → — two runs, nine comparisons, the stage they parted at, and the passage that was present and never read. For the engineer The record format → — what each stage writes, how the digests are computed, and how to check any of it with standard tools and no trust in us.This is onetrace, the Prove door of the oneproof suite: forensic provenance for the retrieval pipeline underneath an AI system. It answers, after the fact, what a given answer actually rested on — and when two runs disagree, where they stopped agreeing.
Measured instead of asserted — from the architecture you choose before you build, to the evidence an answer rests on, to the authorization an action required.