Proof, not benchmark theater.

The one benchmark claim we publish

Retrieval determinism: whether the same question, asked again against the same pinned build, comes back as the same answer. It is the property we can state exactly, so it is the only benchmark result on this page. Everything else we measured is described below without its values.

What the benchmark measured

Four axes, scored identically on both arms. We name every axis, including the ones whose values we are withholding, so you can see the shape of the evidence and the size of what is still unpublished.

The withheld values are not missing. They are measured, recorded in the internal proof package, and held back until someone outside Firelock can run the harness and check them.

How the run was set up

A benchmark is only as trustworthy as the protocol behind it, so the protocol is the product. Here is exactly what was pinned, what was symmetric between the two arms, and what would have voided the run.

The suite

A frozen suite of 26 Multi-SWE-Bench Go tasks over a single repository (the gh CLI), pinned as a hashed task-id list so the task set cannot drift between runs or between arms.

The two arms

Same agent, same model, temperature 0, seed 0, same context budget, same tool-call cap, no arm-specific tuning. The only thing that changes is the retrieval tool: Kin’s graph-backed locate and context, versus a grep-driven agent baseline that works the repository with grep and find.

The passes

Three passes per arm. The Kin arm must reproduce byte-identically across all three or the run is void and nothing from it is citable. The baseline’s cross-pass variance is recorded and scored rather than assumed away.

The scoring

One official evaluation module with a gold denominator, applied identically to both arms, scoring localization at file, symbol, span, and line granularity plus token and tool-call cost. No vendored, arm-asymmetric scorer.

The registration

Suite, model config, symmetric knobs, metrics, scorer version, and the decision rule are written down and committed before the run that they govern, so the analysis cannot be chosen after seeing the outcome.

How we hold ourselves to it

Four rules that decide whether a result is allowed to leave the lab.

What we do not claim yet

This list is the point, not a footnote. Our competitors publish numbers they can’t reproduce. We’d rather under-claim than overclaim. So here is everything we are not asserting.

• ×No published accuracy figures. Precision, recall, F1, and per-granularity localization values stay unpublished until an independent party can verify them.
• ×No token-reduction percentage. We make no “X% fewer tokens” claim.
• ×No broad “beats text search” or “beats version control” headline. On file finding, Kin is a statistical tie with the grep-driven agent baseline today, and we say so.
• ×No merge-time percentage. No “N% less time to merge” or “N× faster review.”
• ×No bug-catch rate. No “catches N× more bugs” or defect-detection percentage.
• ×No merge-gate accuracy, precision, recall, or false-block rate. Those measurements exist internally and are not public claims.
• ×No adoption or customer numbers beyond what’s publicly verifiable on GitHub.
• ×No automatic merge-blocking claim. Review is report-only by default today; enforcement is not the default product behavior.
• ×No investor-citable V8 merge-trust metric. The retained V8 decisions reconcile, but the run lacks the required raw per-arm v2 seals and exact segment ledger. A new sealed run is required.

When one of these clears independent verification under the same proof-gate, it graduates to the page above with its value attached. Until then, it stays here.

Pillar maturity

Three pillars, three honest maturities. We label each one so you always know whether you’re looking at a measured result, live code, or the roadmap.