Record, check, compare.

1.1 First run

Make a folder with a private Python space in it, and install onetrace with its signing and timestamp support (levels 1.7 and 1.8 need it):

mkdir onetrace-learn
cd onetrace-learn
py -3.12 -m venv .venv
.venv\Scripts\Activate.ps1
pip install "onetrace[crypto]"
onetrace --version

mkdir onetrace-learn
cd onetrace-learn
python3 -m venv .venv
source .venv/bin/activate
pip install 'onetrace[crypto]'
onetrace --version

The last line prints onetrace 0.5.0. The console is not needed for this track. Every later command runs in this folder, with the virtual environment active. If you close the terminal, come back with cd onetrace-learn and the activate line.

Create app.py with this in it: a retrieve stage and an answer stage, with a TOP_K setting recorded by name with ot.constant. It is the Get started example plus that one line.

import sys
import onetrace as ot

DOCUMENTS = [
    "Customers can request a refund within 30 days of purchase.",
    "Shipping is free on orders over 50 euros.",
    "Support is open Monday to Friday, 9:00 to 17:00.",
]

TOP_K = 1  # how many documents to retrieve


@ot.stage("retrieve", instrument=ot.pkg("word-overlap", "onetrace",
                                         kind="retriever", config={"top_k": TOP_K}))
def retrieve(question):
    ot.constant("top_k", TOP_K)
    words = set(question.lower().split())
    ranked = sorted(DOCUMENTS, key=lambda d: -len(words & set(d.lower().split())))
    return ranked[:TOP_K]


@ot.stage("answer", instrument=ot.pkg("first-passage", "onetrace", kind="model-call"))
def answer(question, passages):
    return "Answer: " + passages[0]


@ot.run(stages=["retrieve", "answer"])
def main(question):
    passages = retrieve(question)
    print(answer(question, passages))


if __name__ == "__main__":
    main(" ".join(sys.argv[1:]) or "How long do I have to request a refund?")

Record your first run with a fixed run id, so the level's folder has a known name, and check it:

$env:ONETRACE_RUN_ID = "level-1"
python app.py
onetrace-verify --require-artifacts runs/level-1

ONETRACE_RUN_ID=level-1 python app.py
onetrace-verify --require-artifacts runs/level-1

Pass when: the last line reads 81 pass, 0 fail, 4 not-run -> PASS, and runs/level-1/receipts/ holds three receipts: 01-intake.json (the question going in), 02-retrieve.json and 03-answer.json. The four NOT-RUN lines name checks this record doesn't use (no anchor, no declared plan); they are not failures.

Badge: first verified run. Keep runs/level-1.

1.2 Break it

Copy the level 1 folder, then open the copy's last receipt in a text editor:

Copy-Item -Recurse runs\level-1 runs\level-2
notepad runs\level-2\receipts\03-answer.json

cp -R runs/level-1 runs/level-2
open -e runs/level-2/receipts/03-answer.json   # or any text editor

The receipt is one long line. Find "outputs", and in its digest change the first character after sha256: to another hex digit (f to e, say). Save, then verify:

onetrace-verify --require-artifacts runs/level-2

Pass when: the first line reads record NOT intact: receipts/03-answer.json: digest matches the manifest (3 failing rows) and the summary ends 77 pass, 3 fail, 5 not-run -> FAIL. The three [FAIL] rows are: the receipt's digest no longer matches the manifest; the output file return.json no longer matches the digest the receipt now claims for it, with the expected and found digests printed; and the chain head no longer matches the last receipt. Each comes with a fix line that says how to restore it. You changed one character; the record says which file, which value, and that the chain no longer closes.

Now the other half. Copy the original receipt back and verify again:

Copy-Item runs\level-1\receipts\03-answer.json runs\level-2\receipts\03-answer.json -Force
onetrace-verify --require-artifacts runs/level-2

cp runs/level-1/receipts/03-answer.json runs/level-2/receipts/03-answer.json
onetrace-verify --require-artifacts runs/level-2

It reads -> PASS again. A record is bytes; the check is on the bytes.

Badge: saw the chain hold. Keep the verifier's refusal as text, and runs/level-2 restored.

1.3 Two runs

In app.py, change TOP_K = 1 to TOP_K = 2 and save. Record a second run and compare it with the first:

$env:ONETRACE_RUN_ID = "level-3"
python app.py
onetrace localize runs/level-1 runs/level-3 --out r3
echo $LASTEXITCODE

ONETRACE_RUN_ID=level-3 python app.py
onetrace localize runs/level-1 runs/level-3 --out r3
echo $?

Pass when: the line after the two run names reads localize: diverged; the ladder reads intake same, retrieve FIRST DIFFERENCE and answer reconverged; under retrieve it lists instrument config_digest differs, constant top_k differs and output return.json differs; the CAUSE at node 'retrieve' block names the instrument, its config and constant top_k differs; and the exit code is 1, which is what "the runs differ" looks like to a script. The instrument's name and version are unchanged; its config digest changed too, as top_k is also in its config: the same change, seen as a digest.

Read reconverged carefully: it is the lesson of this level. Retrieve now returns two passages, so its output differs; but the answer stage uses only the first passage, which is the same, so the answer's output is identical. A setting changed, an intermediate result changed, and the final answer did not. The record shows exactly where the difference started and where it stopped. r3/localize.json is the same answer as data.

Badge: found the first difference. Keep runs/level-3 and the comparison report r3/.

1.4 Detective, on the live demo

On the live demo, open pipeline P1 and the run p1-08-candidate-set2. Its page already says where it first differs from p1-02-baseline-set2. The detective work is the next part: open that stage's page in each of the two runs, not the comparison, and find which setting differs there and its two values.

Then open the comparison between the two runs. Its short answer, at the top, is one sentence checked from the records, with no model involved. Below it the demo shows an explanation written by a model and labelled as such; that is not the check.

Pass when: the setting you found and its two values equal the short answer's. If they don't, read the stage pages again: every value on them comes from a receipt.

Badge: read the record. Your answer is the output; the comparison's short answer is the check.

1.5 Noise

Set TOP_K back to 1 in app.py, save, and record the example once more:

$env:ONETRACE_RUN_ID = "level-5"
python app.py

ONETRACE_RUN_ID=level-5 python app.py

Nothing you control changed. Compare it with level 1, and ask onetrace which differences look like release noise (run ids, timestamps, paths, version settings) and could be masked:

onetrace diff runs/level-1 runs/level-5
onetrace mask suggest runs/level-1 runs/level-5

Save the suggestion as a mask file and compare again with it. In Windows PowerShell, > writes UTF-16, which onetrace refuses; Set-Content -Encoding ascii writes the plain bytes it reads.

onetrace mask suggest runs/level-1 runs/level-5 --json | Set-Content -Encoding ascii mask.json
onetrace diff runs/level-1 runs/level-5 --mask mask.json

onetrace mask suggest runs/level-1 runs/level-5 --json > mask.json
onetrace diff runs/level-1 runs/level-5 --mask mask.json

Pass when: the first comparison reads diff: identical with all three stages same; mask suggest says there is nothing to suggest; mask.json holds {"format": "onetrace-mask/0.1"}, a mask with no rules; and the masked comparison reads diff: identical, exit 0. This example records no noise: its outputs hold no run id, time or path. On a pipeline whose outputs do, mask suggest lists each such value with where it was seen, and a comparison equal only once masked reads identical after masking. Read each suggested line before you save it: a version change can be a real difference, not noise.

Badge: read a comparison for noise. Keep runs/level-5 and mask.json.

1.6 Keep an input, then erase it

Inputs are fingerprinted, not stored. Keeping one is a choice you make per input, in the decorator. Change the retrieve stage's decorator in app.py to end with keep_inputs=["question"]:

@ot.stage("retrieve", instrument=ot.pkg("word-overlap", "onetrace",
                                         kind="retriever", config={"top_k": TOP_K}), keep_inputs=["question"])

Record and check the run, then erase the kept text and check again:

$env:ONETRACE_RUN_ID = "level-6"
python app.py
onetrace-verify --require-artifacts runs/level-6
Remove-Item -Recurse runs\level-6\inputs
onetrace-verify --require-artifacts runs/level-6

ONETRACE_RUN_ID=level-6 python app.py
onetrace-verify --require-artifacts runs/level-6
rm -r runs/level-6/inputs
onetrace-verify --require-artifacts runs/level-6

Pass when: before the deletion, the folder holds inputs/02-retrieve/question.json (the question's text) and the check reads -> PASS; after it, the check reads exactly the same, 81 pass, 0 fail, 4 not-run -> PASS. A receipt holds each input's digest, never its text, so the kept file was an extra you could take back without touching the record.

Badge: kept and erased. Keep runs/level-6, with its inputs/ removed.

1.7 Sign

Make a key once, in a folder of its own outside your project, then sign a copy of the level 1 run:

New-Item -ItemType Directory -Force $HOME\onetrace-keys | Out-Null
onetrace keygen --out $HOME\onetrace-keys\recorder.key
Copy-Item -Recurse runs\level-1 runs\level-7
onetrace sign runs/level-7 --key $HOME\onetrace-keys\recorder.key

mkdir -p ~/onetrace-keys
onetrace keygen --out ~/onetrace-keys/recorder.key
cp -R runs/level-1 runs/level-7
onetrace sign runs/level-7 --key ~/onetrace-keys/recorder.key

keygen prints a key_id starting sha256:; it only names your key, so it is safe to share. The key file is the secret: never in a run folder, your project or git. If you already made a key in Get started, use it and skip keygen. Create trust.json in the project folder, with your own key_id pasted in:

{"format": "onetrace-trust/0.1",
 "keys": [{"key_id": "sha256:PASTE-YOUR-KEY-ID-HERE", "label": "my laptop"}]}

onetrace-verify --require-artifacts --trust trust.json runs/level-7

Pass when: the signature line reads [PASS ] … signed by my laptop (sha256:…) and the summary ends -> PASS. Then change one character of the key id in trust.json, save, and verify again: the same signature now reads [SIGNED-UNTRUSTED] … signer not on the reader's trust list, and the run still passes. That is a reading, not a refusal: the signature is valid and shows the record is unchanged since signing, but your list no longer names the key. Put the id back. Your trust list is what says whose key it is.

Badge: signed it. Keep runs/level-7 and trust.json.

1.8 Timestamp (this level goes online)

This is the one level that sends anything: the run's fingerprint, to a public timestamp authority, which signs it with the time. Copy the signed level 7 run, fetch the authority's two certificate files, and anchor the copy:

Copy-Item -Recurse runs\level-7 runs\level-8
Invoke-WebRequest https://freetsa.org/files/tsa.crt -OutFile tsa.crt
Invoke-WebRequest https://freetsa.org/files/cacert.pem -OutFile tsa-root.pem
onetrace anchor runs/level-8 --source rfc3161 `
  --tsa-url https://freetsa.org/tsr `
  --tsa-cert tsa.crt --tsa-root tsa-root.pem

cp -R runs/level-7 runs/level-8
curl -o tsa.crt https://freetsa.org/files/tsa.crt
curl -o tsa-root.pem https://freetsa.org/files/cacert.pem
onetrace anchor runs/level-8 --source rfc3161 \
  --tsa-url https://freetsa.org/tsr \
  --tsa-cert tsa.crt --tsa-root tsa-root.pem

onetrace-verify --require-artifacts --trust trust.json --tsa-cert tsa.crt --tsa-root tsa-root.pem runs/level-8

Pass when: the check gains a line [PASS ] anchors/1.json: originality -- time-stamp token … with the attested time, and ends -> PASS. Verify once more without the two --tsa options: that line reads [NOT-RUN] and names the two options it needs, and the run still passes. The anchor can't be checked without the authority's certificates, and the verifier says so rather than guessing.

Badge: committed to a time. Keep runs/level-8 and the two certificate files.

1.9 Reproduce, and read what it can't check

A comparison says what differs between two runs. Reproduce asks another question: if the recorded code runs again, does it give the recorded outputs? It needs a runner file that says how to run your code. Create runner.json:

{"format": "onetrace-runner/0.1",
 "code": "app.py",
 "argv": ["{python}", "-B", "{code}", "{out}"],
 "cwd": "{code_dir}",
 "timeout_seconds": "600"}

onetrace reproduce runs/level-5 --runner runner.json --out r9

Pass when: every stage reads COULD NOT CHECK with its reason in brackets, and the command exits 4 (echo $LASTEXITCODE, or echo $?). With app.py as level 1.6 left it, the reason is an instrument mismatch: the code's digest is not the one the run recorded, so reproduce refuses to call it the same code. Undo the 1.6 change and run it again in a new terminal, where ONETRACE_RUN_ID is not set, and you get the next reason: this example writes its run under runs/, and takes its arguments as the question, so reproduce finds no run in the folder it handed over.

That is the lesson: COULD NOT CHECK is never "close enough", and it always says what stopped it. Reproducing for real needs code that writes its run to the folder reproduce gives it, which the manual's reproduce section describes.

Badge: read a reproduce report. Keep r9/.