What the reviewer actually filed

How this is scored

After a review lands, a human reads it against the code and writes a verdict for every finding: confirmed (a real problem), false positive (it is not a problem, or the diagnosis is wrong), out of scope (real, but not something this PR should be asked about), or missed (a problem someone found later that the reviewer said nothing about). Precision is confirmed / (confirmed + false positives). Nothing is scored by a model, and nothing is scored by the reviewer itself.

The rounds

One row per review. r1, r2… are re-reviews of the same pull request after the author pushed fixes. Most of these repositories are private, so they appear by letter — the counts are what matters, and inventing public rows to look better would defeat the purpose of the page.

Where the false positives landed

Not all wrong findings cost the same. A wrong LOW is noise you scroll past. A wrong BLOCKING tells someone to stop shipping over a bug that is not there, and it is the one that makes people uninstall a reviewer. Four of ours were BLOCKING, and they are the reason severity calibration is a standing piece of work rather than a solved problem.

The pattern in the table above is worth noticing too: on the pull requests that got a second round (r1 → r2), precision usually goes up. The first pass on an unfamiliar codebase is the noisy one.

Against another reviewer

We ran Kaniscope over 30 public pull requests that CodeRabbit had already reviewed, and lined the findings up by file and line (within ±8 lines counts as the same place). 263 findings across both.

One signal in it does not depend on ground truth, though, and it does not flatter us: the shape of what each reviewer raised alone. By their own labels their 124 unique findings were 68 Major and 11 Critical. By ours, our 111 were 61 LOW and 7 HIGH. Whatever the truth of the individual findings, we are producing more low-severity material on the side where we are alone, and their unique findings cluster in functional correctness, data integrity and stability. That is the gap we are working on, and it is why this page exists in the state it is in rather than a tidier one.

Reproduce it

examples/bench.rs in the engine scores the reviewer against a corpus of pull requests with known issues and reports precision, recall, F1 and token cost, so a feature can be A/B’d by re-running with the flag flipped. It is a dry run — it posts nothing — and it needs your own model key. Blast radius, complexity metrics and the choice of backend were all measured this way. Blast radius, for the record, showed no recall improvement on well-named repositories; it stays on because it may still help on large monorepos, not because we measured it helping.

How this compares to a per-seat reviewer →