March 10, 2026
How we stop our AI reviewer from making things up
AI code review has a trust problem: plausible-sounding claims nobody can verify. Here's Striff's answer, a facts-first, neurosymbolic pipeline where the graph is measured before the AI writes a word, plus the findings we deliberately refuse to post.
The most common complaint about AI code review isn’t that it misses things. It’s the opposite: it says too much, too confidently, about too little. Plausible-sounding comments that don’t survive a second look. Speculation dressed as analysis. Enough noise that developers do the rational thing: they stop reading the bot.
This is a real adoption killer, and it can’t be fixed with a better prompt. So when we built Striff, we made a structural decision: the AI is not allowed to be the source of any claim. Here’s what that means concretely, including the findings we deliberately don’t post, and a bug that proved the design right.
Prose-first vs. facts-first
The standard way to build an AI reviewer is prose-first: hand the diff (plus retrieved context) to a language model and let it write what it notices. The output quality is the model’s judgment; there’s nothing underneath to check against. When it’s right, it’s useful. When it’s wrong, it’s wrong in fluent, confident English, and the reviewer has to redo the analysis just to find out which one they got.
Striff inverts the order. The measurement happens before any AI is involved, and the AI’s job is reduced to narrating results it cannot alter:
Two pipelines, opposite trust models
Diff + context
The PR text and whatever retrieval surfaces.
LLM reads & judges
One step is both the analysis and the source of truth. Nothing exists to verify it against.
Fluent prose
Right or wrong, it reads the same. The reviewer inherits the verification work.
Parse
Both sides of the PR are parsed into a component graph. Deterministic, no AI.
Measure
Coupling, new and deleted edges, layer depths, cycle paths: computed, not generated.
Detect
Structural rules and anomaly scoring run over the measured graph and produce the findings.
Narrate
Only now does an LLM write, constrained to the extracted facts. It phrases; it doesn't discover.
The neurosymbolic split: symbolic layers (1–3) own truth, the neural layer (4) owns readability. A hallucination in step 4 would have to contradict its own input, and gets caught against the facts, not the reviewer's patience.
What “grounded” looks like on a real finding
Abstract claims about grounding are cheap, so here’s the actual anatomy of one finding from apache/pinot #19073. Every load-bearing phrase in the note maps to a measured fact:
Every claim traces to a fact
SegmentProcessorAvroUtils in core.util now calls directly into plugin.inputformat.avro①, the first edge ever in this direction②, skipping a layer on the way③. It also modifies a contract that at least 24 components depend on④.If a fact isn't in the extracted set, the sentence can't say it. The model can't claim "this creates a cycle" unless the cycle detector found one, and can't cite "24 dependents" unless the parser counted 24. Phrasing is neural; every number and every edge is symbolic.
This design has a second benefit that pure-LLM systems can’t offer: when we’re wrong, we’re debuggably wrong. Recently our C# parser had a bug where generic types were registered under their parameterized names and dropped out of the graph, which meant some findings were built on missing edges. Because the fact layer is deterministic, this was an ordinary software bug: reproducible, testable, fixed at the parser level, and verified by re-scanning. You cannot do that with a hallucination. There’s no failing test for “the model felt confident.”
The findings we refuse to post
Grounding kills fabricated claims, but there’s a second species of noise: findings that are structurally true and practically useless. A fact-based system can generate these all day, so the quality bar has to be about decision relevance, not just truth. Some patterns we’ve deliberately suppressed or downranked:
True, but not worth your attention
| Pattern | Structurally true? | Posted? |
|---|---|---|
| "Production code depends on test modules" when the edge is an artifact of how test helpers are laid out | ✓ Yes | ✗ No. We hit exactly this scanning FastAPI, and it read as a false positive. It was suppressed, not defended. |
| Small coupling deltas on leaf components nothing depends on | ✓ Yes | ✗ No. A +3 on a quiet leaf is churn, not risk. It feeds the trend data instead. |
| High absolute coupling that's been stable for years | ✓ Yes | ✗ No. Deltas on high-dependency nodes matter; old news doesn't. |
| A first-ever boundary crossing, a new cycle, a contract change with dozens of dependents | ✓ Yes | ✓ Yes. Severity-ranked, few in number. |
On real public PRs, Striff posts 4 to 8 findings per pull request, not forty. A finding that makes a reviewer shrug costs more trust than a finding we skip.
Why this matters more right now
AI assistants are pushing PR volume up across the industry, and automated review comments are multiplying in the same feeds. The scarce resource is no longer analysis. It’s developer attention and trust. A reviewer tool only works if engineers still read it in month six.
That’s the bet behind the facts-first design: fewer findings, every one traceable to a measured property of your dependency graph, in the PR where the risk appears. When we analyzed merged refactor PRs from popular repos, the structural risks Striff surfaced appeared in nobody else’s review, human or bot, because nobody else was measuring the graph.
See what the pipeline extracts from your own code: install the browser extension and open any pull request. Every claim it makes, you can check.