A code review agent that knows production
Most AI code reviewers only read the diff. They catch syntax and logic bugs, and miss everything that depends on how the code behaves once it ships. So we built Sherlock, a review agent connected to the same places our engineers look when something breaks. This is what we give it, and why more context made it a much better reviewer.
Agents write most of the code my team ships, so review became our queue. The obvious move was an AI reviewer. The problem is that a reviewer who only sees the diff reviews code the way a new hire would: carefully, and without knowing which endpoint is already slow, which file caused last month’s incident, or what the team has agreed never to do again.
Our senior engineers know those things because they have looked at the dashboards and been on the incident calls. So we gave the reviewer the same access.
What the reviewer reads before it judges a line
- CODEThe diff, and the whole files around itEvery changed file in full, not just the hunk. Most AI reviewers stop here.
- ERRORSSentry: open issues on the files you touchedEach one comes with the root-cause analysis from when it was investigated.
- PERFDatadog: endpoints, monitors, slow queriesRequest rate and latency for the endpoints in the diff, monitors that are firing, and slow queries on the tables it touches.
- HISTORYPast investigations and past reviewsA search over earlier incident investigations and reviews on the same repo. If this code broke before, the reviewer knows how.
- TEAMBest practices and learned rulesThe coding standards we keep adding to, plus rules learned from our feedback and approved by a person.
The first layer is what every AI reviewer has. The other four are what make Sherlock useful. They are collected once, before the review starts, and handed to the agent as a single snapshot. Then it gets tools to dig further: read any file at the PR’s commit, search the codebase for other callers, and search past investigations.
Performance data turns opinions into findings
For each PR, Sherlock looks up the Datadog services the repo runs and pulls recent stats for the endpoints the diff touches, any monitors that are alerting, and slow queries on the affected tables. It goes into the prompt as plain text the model can reason over:
## APM endpoint stats (last 24h)
GET /api/reports/{id} 1,940 req/h p95 1.8s errors 0.4%
POST /api/reports/export 120 req/h p95 6.2s errors 2.1%
## Monitor alerts
[warn] reports-export latency above 5s for 3 of the last 24h
## Open Sentry issues on touched files
reports/export.py TimeoutError in build_pdf() (37 events, 7 days)
Past root cause: one query per row while rendering large reports.
Now a finding can carry evidence. Instead of “this loop might be slow”, the reviewer can say the new loop adds a query per row to an endpoint that is already at 6 seconds, already has a latency alert, and already timed out for the same reason last week. That gets fixed. A vague worry gets ignored.
The prompt is explicit about it: if a touched endpoint or query has error or slow-query history, the bar for the change goes up, and the finding has to cite the data rather than just assert a risk.
Past incidents are the best review checklist you have
Sherlock started as an incident tool. Engineers ask it things like “why was the system slow at 6:20 pm?”, and it investigates across logs, traces and code. Every one of those investigations is stored and searchable.
The reviewer uses that history in two ways. Open Sentry issues on the touched files arrive with their root-cause analysis already attached. And when a diff touches code with a history of incidents, or looks like something reviewed before, the agent searches past investigations and past reviews on that repo and cites what it finds instead of working it out from scratch.
diff touches reports/export.py
-> search_memory("report export timeout, per-row queries")
<- investigation from August: export timed out at month end,
root cause was one query per row in build_pdf()
-> finding: the new row renderer reintroduces the same pattern
A generic reviewer can’t know your incidents. That is exactly the knowledge that stops the same outage happening twice.
It learns from the people it reviews for
The reviewer gets better every week because every review produces feedback:
- Direct feedback. Engineers rate findings, tag them and leave a note. They can also reply to Sherlock in the PR thread and get an answer.
- Outcomes. When the PR closes, each finding is marked as applied, resolved or dismissed, based on what actually happened to that line.
- Human reviewers. Comments people left on lines Sherlock never flagged are recorded as misses. Replies pushing back on a finding are recorded as pushback.
Once a week those signals are clustered per repo. Where a pattern repeats, a model proposes a rule. A person reads it and promotes it or throws it away. Active rules are added to the reviewer’s prompt on every run.
[review · active]
When a change adds a new API endpoint, check that a permission test
exists for it. Source: 4 human review comments the agent missed.
Next to the learned rules sit the rules we write ourselves. Our best practices live in the repo, and we keep adding to them whenever a review comment comes up more than once. The agent reads both.
More context needs more honesty
Context has a trap. For months, the slow-query section of every snapshot was empty, not because queries were fast but because that Datadog integration had never been switched on. An empty list from a healthy system and one from a missing integration look identical, and the model read the silence as good news.
Now the snapshot starts with a coverage note for any signal the repo doesn’t have, placed above the empty section so the caveat is read first:
## Signal coverage (read before trusting an empty section)
- Slow-query monitoring is not configured for this repo. An empty
section below means UNKNOWN, not healthy. Say you could not assess
database impact. Do not imply you checked.
The same applies to effort. A reviewer with lots of context can still skim. We give it a floor (open every changed file before submitting) and a tool budget that grows with the size of the PR, so a 60-file change gets read as carefully as a 3-file one.
Generic reviewer, or one that knows your product
A generic reviewer knows code. Sherlock knows which endpoints are slow, which files caused incidents, what our reviewers keep asking for, and which of its own findings people ignored. The model is the same one anyone can use. The context is what we built.
If you are setting up an AI reviewer, my advice is to start with the context you already have. Connect your error tracker and your monitoring, store your incident write-ups somewhere searchable, write down the review comments you keep repeating, and give people a way to tell the agent when it is wrong.