Debugging an agent

One run went wrong. Where do you start?

The nine places a developer looks when an agent does something it shouldn't, what tools give you today, and how onetrace makes each one shorter.

Availableyou can use it today Nextthe next release Laterplanned after that
The incident

An invoice agent paid €4,000 on an invoice for €40.

Yesterday the same pipeline handled the same supplier correctly. Nobody changed the code. The pipeline has five steps, two model calls and a payments tool behind a policy check.

You have logs, a tracing tool, the model provider's console and the payment record. None of them was built to answer the question you have.

An illustrative example, not a real event.

yesterdayintakeextractlookupapprovepay
todayintakeextractlookupapprovepay

The wrong effect shows at the last step. Where it started is the question.

The debug cycle

Most of the hours go into one box: Investigate.

Faced with one bad run, you reach for whatever narrows it down: read the run, find one that worked, look for runs like it, ask what changed, rule out noise, re-run it, check who allowed it, ask an assistant. Each is partly covered by some tool today, and each has a gap that turns minutes into days.

Notice→Investigate→Hypothesis→Test→Fix→Confirm

Nine avenues from one bad run

Avenue 0 · Was it recorded?

"There's nothing useful in the logs for that request."

your app, unchangedonetrace-run python app.pya receipt per stepclose receipt

Today, across tools

  • Logs and traces, if someone added them on this path.
  • A run that crashed looks the same as one that finished early.
  • Anyone with access can edit what was kept.

With onetrace

AvailableReceipts per step

Each step's inputs, outputs, tool and settings, fingerprinted and chained; onetrace-verify checks them with no network calls and no account.

AvailableZero-config recording (onetrace 0.7.0)

onetrace-run python app.py records calls through the OpenAI client, requests and httpx, with no code change.

AvailableThe close receipt (onetrace 0.7.0)

Every run ends with one, so a run that was cut short reads "not closed", never "finished".

AvailableThe console reads onetrace 0.7 runs

From console 0.7.0: zero-config runs and the close receipt open in the console, with whether each run closed and how it ended.

Try it: pip install onetrace-auto in the console's environment (from console 0.7.0 they share onetrace 0.7.1), then onetrace-run python app.py. Zero-config →

Avenue 1 · Read the bad run

"Show me exactly what each step saw and did."

intakeextractlookupapprovepay

inputs and outputs by fingerprint · the model call as recorded · the tool result · the decision · signed and chained

Today, across tools

  • Trace waterfalls: timing, status, attributes.
  • Prompt and response viewers.
  • Which model build answered, with which settings: often missing.

With onetrace

AvailableThe run record

A run's page in the console: every step, its receipt, and the verifier's result.

AvailableThe model-call record

The model's identity, sampler settings and usage, as the call ran, not as it was configured.

AvailableYour app's spans beside each step (console 0.6.0)

Give the console your app's OpenTelemetry export and a step shows its spans: model, tool and HTTP calls, retries, tokens.

AvailableA token as text (console 0.6.0)

On your own console, where the tokenizer files you name hash to the record's.

Try it: onetrace-console runs --open, then open a run. Compare and check →

Avenue 2 · Compare with a good run

"It worked yesterday. Where did today's run part ways?"

goodintakeextractlookupapprovepay
badintakeextractlookupapprovepay

first difference: extract

Today, across tools

  • Side-by-side trace or prompt comparison, in some tools.
  • Diffing exported logs by hand.
  • Nothing says "same inputs, different output" at a step.

With onetrace

AvailableThe first difference

Two runs step by step: the first step whose output differs, the setting that changed there, then the exact text and each JSON field by path.

Available"What changed?"

From a run's page, the console picks an earlier run to compare with, a marked benchmark first.

AvailableConfiguration against the reference (console 0.6.0)

When no twin run exists: this run's settings, code and model step by step against the known-good run's.

AvailableA twin (onetrace 0.7.0)

Re-run these inputs with another run's configuration, from the command line.

LaterInside a nested step

Name the step inside a step where the difference starts.

Try it: onetrace-console runs --compare good bad --open. See one on the live demo →

Avenue 3 · Find runs like it

"Is it this supplier, or everyone?"

today's runsame input · yesterday
today's runnear input · 0.97

exact, by fingerprint · by meaning, with an embedding model you choose (illustrative number)

Today, across tools

  • Filtering traces by tags and metadata.
  • Text search over prompts, in some tools.
  • No notion of "the same configuration" or "the same model files".

With onetrace

AvailableSearch runs by what they recorded

inputs:same-as:, setting:, first_diff:, model: and more: exact matches, nothing guessed.

AvailableA known-good run, marked

Mark a run as a benchmark; comparisons and verdicts read against it.

AvailableRuns with near inputs (console 0.6.0)

Opt-in, on your own console: runs of the same configuration whose kept inputs are near this one's, from a local model or your own endpoint.

AvailableRuns that used the same model files

Listed from each model-call record.

LaterRuns with near outputs

"Which runs answered like this one?"

Try it: Turn it on from your console's settings menu, the gear at the top right. Find similar runs →

Avenue 4 · What changed, and when

"Nobody deployed anything. What moved?"

runs in recorded order · the first run to leave the reference

Today, across tools

  • Deploy logs, git history, feature-flag audits.
  • Provider changelogs, where the provider publishes one.
  • A hosted model can change behind the same name.

With onetrace

AvailableThe kind of difference (console 0.6.0)

One line at the top of a comparison: the step's inputs, a model call on equal inputs, a decision, or an output that wasn't kept.

AvailableThe first run to leave the reference (console 0.6.0)

A pipeline's runs in recorded order, and the first one that departs, by step.

AvailableThe model call, compared on equal inputs

Identity, engine and sampling settings, field by field.

AvailableStep expectations (onetrace 0.7.0)

Declare what a step's output must look like; the verifier checks it after the run.

AvailableExpectations on the run's page

Each one held, broken or not run, in the console (from console 0.7.0).

AvailableExpectations checked during the run (onetrace 0.8.0)

Opt-in: record each result as the step returns, or stop before the next step.

Try it: Open a pipeline in the console and choose First run to leave the reference.

Avenue 5 · Is it just noise?

"If I re-run it, will it even do that again?"

          · 

a known-good run repeated · the bad run, outside the range the step shows on its own

Today, across tools

  • Re-run it a few times and eyeball it.
  • Eval averages, which hide one bad run.
  • Per-token log probabilities from some providers, if requested.

With onetrace

AvailableRepeat sets and the variation profile

Declare what is allowed to vary, then read a run against a benchmark's range.

AvailableThe first token where outputs part (L1)

From the record, the decode step where two outputs first differ.

AvailableL1 to L3 capture for vLLM (onetrace 0.7.0)

Token ids, top candidates and per-step digests for self-hosted models.

NextWas it a near-tie? (L2 in the console)

At the first differing token: was run B's token among run A's top candidates, and at what rank.

Try it: How the record handles repeats →

Avenue 6 · Re-run it safely

"Can I try it again, without paying the invoice twice?"

intakeextractlookupapprovepay
intakegood outputlookupapprovepay

Today, across tools

  • Notebook re-runs with hand-made mocks.
  • Replay tools that stub recorded responses, built per project.
  • Nothing stops a re-run from calling the payment tool.

With onetrace

Availableonetrace reproduce

Re-run from the record and the kept inputs.

AvailableRe-runs stop before a write (onetrace 0.7.0)

Mark the stage effect="write" (onetrace-auto labels the calls it records); a re-run stops before a write, or a step it cannot classify, unless you pass --allow-writes. A step with no label still runs, and is named before the re-run starts.

AvailableA counterfactual (onetrace 0.7.0)

Put the good run's output in at one step and see if the rest recovers.

Availableonetrace check (onetrace 0.7.0)

Check one claim about a run or a pair, recorded as a run of its own.

Next"Prove it" from the comparison

Run the suggested check from the console's comparison page.

Try it: onetrace reproduce runs/<run>, with onetrace 0.7.0 or later for the stop before a write; the install line above brings 0.7.1.

Avenue 7 · Who allowed it

"The agent paid. Which rule let that through?"

agent proposespolicy decidespermit issuedrequest sentrecipient confirms

green: signed by the party that did it

Today, across tools

  • Policy-engine and audit logs.
  • The payment provider's own record.
  • Each held by the party with the most at stake.

With onetrace

AvailableDecisions on gated steps

With onedoor, a gated step cites the decision that allowed it; onetrace-verify fails a step that acted on a deny.

AvailableThe recipient's signed confirmation (onedoor 0.10)

The receiving system signs which permit and request it acted on, and the outcome.

AvailableThe confirmation in the run's record (onetrace 0.8.0)

A onetrace step records the permit it presented and the recipient's confirmation, and onetrace-verify --recipients reads five confirmation rows. A confirmation is the recipient's word, signed, for an effect or a refusal.

LaterThe authorization page

Proposal, decision, permit, request and confirmation on one page.

Try it: onedoor, on its own or with onetrace →

Avenue 8 · Ask for an explanation

"Just tell me what probably happened."

checked factsread-only probesAI reading, citedchecks to run

Today, across tools

  • Pasting logs into a chat assistant.
  • AI summaries inside observability tools.
  • Plausible answers with nothing to check them against.

With onetrace

Availableonetrace-explain 0.3.1

What changed and the evidence first, with no model and no network calls. Then, if you ask, a model's reading, labelled AI-generated: a claim that cites nothing is dropped.

AvailableRead-only probes

Variation, masking ids and times, and the model-call record, before the model writes anything. Nothing is re-run.

AvailableThe Explain button

On a comparison page in the console, with your own model and key.

NextMore evidence in the console

The first run to leave the reference, configuration against the reference, the kind of difference, a near-tie, and your app's spans, as probes.

AvailableSet up the model in the console

The model's settings in a form, the key in your system's keychain (from console 0.7.0, with onetrace-explain 0.3).

Try it: onetrace-explain runs/good runs/bad. Explain a difference →

Put together

One command reads the record for you.

onetrace-explain takes the good run and the bad one. It prints what changed and the evidence from read-only probes, with no model and no network calls. If you configure a model, it adds a reading labelled as AI-generated, where every claim cites the record or is dropped. Checks that would run something are listed for you, never run.

Illustrative output for the invoice example. The four parts and their headings are the ones onetrace-explain 0.3.1 prints.

Terminal
onetrace-explain runs/yesterday runs/today

What changed (checked from the records)
  First difference at extract; 3 changed after it, 0 the same again
Evidence (read-only, no re-runs)
  P1 variation   outside the benchmark's range (12 runs)
  P3 model_call  model build differs on equal inputs
What may have happened (AI-generated, may be wrong)
  1. the provider's model build changed     cites P3
Checks a human could run (not run)
  onetrace reproduce runs/today extract --runner <your runner>
At a glance

The avenues, and where each one stands.

AvenueAvailableNextLater
0 Recorded?receipts, verify, zero-config, close receipt, the console reads 0.7 runs——
1 Read the runrun record, model-call record, app spans, token as text——
2 Comparefirst difference, text and JSON diff, "What changed?", configuration vs reference, twin—nested steps
3 Runs like itsearch, benchmarks, near inputs, same model files—near outputs
4 What changedkind of difference, first run to leave the reference, expectations, expectations in the console, checked during the run——
5 Noise?repeat sets, variation, L1, vLLM capturenear-tie (L2) in the console—
6 Re-runreproduce, stop before a write, counterfactual, check"Prove it"—
7 Who alloweddecisions on gated steps, the recipient's confirmation (onedoor), confirmation in the record—authorization page
8 Explainonetrace-explain 0.3, probes, the Explain button, setup in the consolemore evidence—

No dates: each release ships when its checks pass, and planned items can change. From console 0.7.0, the install line above gives onetrace 0.7.1, and the console reads runs recorded with onetrace 0.7 and earlier, zero-config runs included: pip install onetrace-auto goes in the same environment. The items marked onetrace 0.8.0 need onetrace 0.8.0: the console pins onetrace 0.7.1, so give the console an environment of its own beside it until a console release supports it.

Try it

Nine avenues. One record they all read from.

Recorded once and checkable by anyone, so a bad run becomes a short list of evidence, not a week of guessing.

Install
pip install onetrace "onetrace-console[neighbours]" onetrace-explain