Retrieval quality is measured by exhaustive search on a sample of your own documents. Speed is measured only on a real engine under load. The two are never mixed, and where a constraint could not be checked, the report says so instead of rounding.
oneground characterize # exact right answers for your queries oneground simulate # score every architecture against that truth oneground verify # the real engine under load -- the only latency oneground report # the verdict, and what was not measured
characterize computes exact nearest neighbours for a held-out query set by
brute force over the sample. That is the expensive step and the reason the rest is cheap:
once the right answers are known, every candidate architecture can be scored against them
without standing any of them up.
simulate therefore answers "how well does this find things" in minutes on a
laptop. It does not answer "how fast is it", and does not pretend to.
verify is the only source of latency and throughput figures. It runs a real
engine under real load, on your hardware or on a machine you rent. Latency is never taken from
simulation. The preview runs Qdrant. A second engine, pgvector, is built and in the repository,
measured in turn on the same host with the same corpus and the same query set — each carrying its
own number and its own verdict, because two engines on two machines are not a comparison.
One requirements file drives a run. The corpus sample is either
vectors (an (n, dim) float32 array, optionally with ids) or
text plus a pinned model whose weights are hashed into the record.
Queries are supplied as text or as vectors, with their source declared —
logs, written or synthetic — because queries drawn from
production and queries someone invented are not the same evidence. Optional metadata enables
drift and filtered-search measurement. Sampling method and full corpus size are recorded,
not enforced. Constraints — recall, p95 latency at a given QPS, storage amplification,
memory and budget — are declared by you, and the report marks every field as measured or
declared beside its value. Omit the sample entirely and you get capacity arithmetic and no
verdicts, which is the correct output for a run with nothing measured in it.
On a real corpus, before any engine has been run — this is the
published
report's own summary, {"options": 8, "meets": 0, "fails": 6, "couldnt_check": 2}
with recommendation: null:
8 option(s): 0 meets, 6 fails, 2 couldnt_check
No option meets every constraint, so nothing is recommended.
Recommending an option whose constraints could not all be checked
would be rounding couldn't-check up to a verdict.
To decide latency_p95: run `oneground verify` against a real engine
in the environment the constraint targets. Latency is never taken
from simulation.
Run the same corpus through verify against a real engine and it resolves —
but not into a clean win. This second block is from a run whose report is not published,
so unlike the one above you cannot open it today; it is quoted because it is the honest other
half of the same corpus:
8 option(s): 1 meets, 6 fails, 1 couldnt_check Recommended: single_node_hnsw[M=32,efConstruction=200,efSearch=128] indistinguishable on recall from: hash_sharded[M=32,efSearch=96,shards=3] hash_sharded[...]: couldnt_check -- could not be checked on latency_p95, qps. Indistinguishable on recall is not indistinguishable overall.
Both are real output from the same corpus at different stages. Note what the second one
does not do: the alternative that matched on recall is not folded into the
recommendation, because recall is not the whole constraint set. Three outcomes —
meets, fails, couldnt_check — and the third is never
rounded up to either of the others.
The lab plots 150,000 arXiv abstracts embedded with
bge-base-en-v1.5, partitioned into 256 regions by k-means. Semantic sharding
copies a vector into every region whose centroid lies within ε of its
nearest. Move ε and the copy counts are recomputed in the browser from four real centroid
distances per vector carried in data/base.bin — not from a lookup table, and not
from anything generated in the page.
| at ε = 0.20, the published reference | value |
|---|---|
| Vectors copied at all (more than one region) | 144,563 of 150,000 |
| Storage amplification — mean copies per vector | 3.7152 against a published 3.715 |
| Copies per vector — 1 / 2 / 3 / 4 | 3.6% · 5.6% · 6.5% · 84.3% |
| Boundary crispness | 0.0362 against a published 0.036 |
| Held-out queries with exact top-10 by brute force | 2,000 |
The 2-D placement in the lab is UMAP and is illustrative. Regions, distances and copy counts are computed in the full 768 dimensions.
The page refuses to render if its own arithmetic disagrees with the export's — which is the same discipline as the refusal above, applied to a picture.
Three independent checks, each of which can fail — and each with a different precondition, stated here because two of them are cheap and one is not:
# 1. does this host serve the repository's bytes? every file, by digest # needs: a clone. stdlib only, no install, no assets.
python site/teaser/check_hosted.py https://oneproof.dev/oneground/lab
# 2. re-derive the lab's counters from base.bin, independently of the page # needs: a clone. stdlib only, no install, no assets.
python site/teaser/verify_teaser_data.py
# 3. recompute a fixture's published values from the published data. # needs: a clone of main, a Python 3.12 virtual environment, and the # repository's own pinned requirements. the small fixture ships in the clone. # run to completion twice on 16 September, by oneground and then # independently on a second machine, with byte-identical digests. the # output is printed beneath it, unedited. the environment line matters: # without one, pip replaces packages in your system Python to meet the pins.
git clone https://github.com/shamiksaharcciit-oss/oneground cd oneground pip install -r requirements.txt python -m oneground.cli fixture verify arxiv-smoke
What that prints, unedited, from the run it was taken from (oneground, 16 September, a fresh clone, Python 3.12; a second run on a second machine the same day produced the same eight digests byte for byte; the only change below is that one home-directory path is shown as its documented location):
python C:\...\og-clone\.venv\Scripts\python.exe (venv) fixture: arxiv-smoke directory: fixtures\arxiv-smoke manifest: MANIFEST.sha256 (11 files listed) digests couldnt_check receipt sample.jsonl.zst artifact not present (release asset, or not built); looked in fixtures\arxiv-smoke and ~/oneground-assets/arxiv-smoke couldnt_check receipt vectors.npy artifact not present (release asset, or not built); looked in fixtures\arxiv-smoke and ~/oneground-assets/arxiv-smoke verified receipt queries.npy 2359ce716b2ca7ef75c5d9f4fa4a42924df83eeac5848918b6a823efa7f7433a verified receipt query_ids.json 0081604bb50d8dd48b4ae5605a05fc894dcd2ae4db41511657464772c911b8e2 verified receipt ground_truth.npy 13919bb5174ebeb4febe037017a2e2b8bbfe4998b3ce261e7989e4649c2a9cce verified receipt characterization.json 07b576e27e680596d8cb52c59f01dcb18e1039a9be135c48db921e5a56dc30e1 verified declared build_info.json fa6168a52acb45283a51f73554e3683dd24abde014e0b82a4091e0e3ebb944c1 couldnt_check declared projection.npy artifact not present (release asset, or not built); looked in fixtures\arxiv-smoke and ~/oneground-assets/arxiv-smoke verified declared ground_view_base.parquet 23ee242d845034ed91c01f3dd6f57817d3b5d426ce3b2b1502a61fb10c2cf948 verified declared ground_view_queries.parquet 32a562466ccb29932742d3c146892b7f9f420676ce91dfe8219c320ac70ef6c6 verified declared ground_view_centroids.parquet b9655c45a31bd2fde2440bb05d5764d6c4611767bfa39015e64c6352beeea496 values couldnt_check intrinsic_dimensionality the spec publishes no value for this field yet, so there is nothing to reproduce. The recomputation was skipped rather than run against placeholders. couldnt_check boundary_crispness the spec publishes no value for this field yet, so there is nothing to reproduce. The recomputation was skipped rather than run against placeholders. couldnt_check skew_top10_share the spec publishes no value for this field yet, so there is nothing to reproduce. The recomputation was skipped rather than run against placeholders. couldnt_check ambiguous_query_rate the spec publishes no value for this field yet, so there is nothing to reproduce. The recomputation was skipped rather than run against placeholders. couldnt_check single_node_hnsw.recall_at_10 the spec publishes no value for this field yet, so there is nothing to reproduce. The recomputation was skipped rather than run against placeholders. couldnt_check semantic_sharded.recall_at_10 the spec publishes no value for this field yet, so there is nothing to reproduce. The recomputation was skipped rather than run against placeholders. couldnt_check semantic_sharded.storage_amplification the spec publishes no value for this field yet, so there is nothing to reproduce. The recomputation was skipped rather than run against placeholders. couldnt_check drift the spec publishes no value for this field yet, so there is nothing to reproduce. The recomputation was skipped rather than run against placeholders. summary: digests 8 verified, 0 contradicted, 3 couldnt_check (6 receipt, 5 declared) values 0 verified, 0 contradicted, 8 couldnt_check
Three things a reader sees there, and each is the point. Eight of eleven digests verify;
the three that cannot are release-asset files, and the tool names both places it looked
rather than failing opaquely. All eight values are couldnt_check, and
correctly: arxiv-smoke is a scaffold whose values are not yet published, and
the tool says so — the recomputation was skipped rather than run against
placeholders. That is the instrument declining to compare against a placeholder on its own
fixture, which shows the discipline better than a clean pass would. And the invocation is printed
exactly as it was run; the oneground console script also works from that
environment, but the module form is what was run.
The full 150,000-vector fixture is not printed here yet. It needs the release asset
(arxiv-150k-v1.tgz, 461 MB) and an --asset path, and the sequence for
it goes on this page the day someone has run it to completion — not the day a document
says it should work. Twice this week a command was printed here that had not been run, and both
failed. That is the rule now: a command on this page is one that has been executed, and the
output beside it is what it produced.
The first reports one of three outcomes per file — verified,
contradicted, couldnt_check — and exits non-zero only on a
contradiction; an absent file is a gap, not a verdict. The second exits non-zero on any
disagreement with the fixture's published values. Digests for every file the lab serves are
in lab/data/MANIFEST.sha256.
The lab directory on this site is oneground's own site/teaser/, copied byte
for byte with nothing edited. That is deliberate: if we restyled it, check 1 would report
contradicted and you would have no way to tell our edit from a bad deploy.
Two public reference collections ship with specifications and digests:
arxiv-150k and stackexchange-150k. Each has a fixture spec
(fixtures/<name>.fixture.yaml), a manifest, its characterization, and its
ground truth, so a figure quoted from either can be traced to the bytes it came from. The
arxiv-150k decision is published in full —
report.json,
report.html, simulate.json, verify.json,
verify_info.json and characterization.json, digests in the
fixture's manifest — so the refusal quoted above is not a screenshot of something you have to
take on trust.
Every report cites the calibration run it was generated under.
oneground calibrate show renders the calibration history; the history itself lives
in the repository at calibration/history.jsonl, so a bare install prints "empty" and
names the command that writes the first line, while a checkout prints the full record of
oneground measured against published benchmark results over time. Read the tool's own error
before you read its answer.
Scoring against known-correct answers needs no running engine. Timing does. Any
latency or throughput constraint left unmeasured comes back as
couldnt_check with the command that would settle it.
What is live on this site is the public collection. An interactive lab over your own vectors does not exist. Measuring your own documents is what the command-line tool does.
Ten to twenty thousand documents, not your whole archive — and the report says so, rather than implying the sample was the corpus.
Which documents come back, not the answer your model writes on top of them. That is a different question and a different tool.
How documents are cut into passages before indexing is not measured in this release; there is a design note and no implementation. It is stated here because chunking plausibly moves retrieval more than the index choice does, so a tool silent about it should say so rather than let you assume coverage it does not have.
Already built and already in the public repository today: a second engine measured in turn on one host; latency runs repeated three times with the spread reported, so a pass means it passed every time; a saturation measurement — how much traffic the setup takes before it breaks, and where; a second reference collection, so the data's contribution can be separated from the engine's; and a check for documents longer than the embedding model's context, which are being silently truncated before they are ever searched. Shipped when it is tested; dated when a witnessed run makes it true.
BackThe case for it → — what it is for, what it costs, and what does not leave your building. The instrumentThe lab, live → — 150,000 real vectors, one query followed through every hop, and the verdict at the end.