Asking an agent why production was slow at 6:20 pm
My team runs an agent in production that answers questions like “why was the system slow at 6:20 last night?” It reads the same dashboards, logs and code an on-call engineer would, and comes back with a cause and a confidence score. These are the things that made its answers worth trusting.
When something breaks at night, the first hour usually goes to the same routine. Open the dashboards, find the time window, check the errors, look at the database, check what was deployed, read some code. It is careful work, and most of it is the same every time.
So we built an agent that does that routine. You type a question in plain English and it streams back an investigation: what it looked at, what it found, the chain from symptom to cause, and how confident it is. It has been running in production for a while now, and the quality of its answers came less from the model than from the steps around it.
What happens when you ask
- PARSETurn the question into a windowA small model call extracts the time range, the services and the symptoms. If there is nothing to anchor on, it asks one question first.
- SNAPSHOTFetch the system state for that windowHost and database metrics, worker health, 5xx errors, slow queries, and traffic on the busiest endpoints compared with the same window a day earlier.
- CHECKFlag anomalies in plain codeAny endpoint running at twice its normal traffic gets a line at the top, before the model sees anything.
- INVESTIGATEDig with tools until it can answer “why now?”Errors, logs, traces, metrics, recent deploys and the code itself, reasoned through as a chain from symptom to root cause.
- SCORESay how sure it isA separate small call scores confidence, with a hard cap when the evidence was incomplete.
Two of those five steps involve no model at all. That is deliberate. The parts that can be done with plain code are done with plain code, so the model spends its attention on the part that needs judgment.
Turn the question into a window first
“Why was it slow at 6:20 last night?” is a fine question for a person and a vague one for a system. Before anything else, one short model call turns it into something the rest of the pipeline can use:
{
"from": "2026-09-14T18:05:00+05:30",
"to": "2026-09-14T18:35:00+05:30",
"services": ["api"],
"symptoms": ["slow responses"]
}
Relative times are read in the team’s time zone, and the window gets a margin on both sides, because the cause usually starts a few minutes before anyone notices.
If the question names no time, no service and no symptom, the agent doesn’t guess. It asks one short multiple-choice question and starts once it has an anchor. Guessing the wrong window wastes a whole investigation.
Fetch the snapshot before the agent thinks
An agent starting from nothing spends its first ten tool calls finding out what any engineer would check first. So before the investigation starts, we collect a snapshot of the window and give it to the agent as its starting point:
## Host and database
api host CPU 91% memory 78% IO wait 4%
database CPU 64% connections 182 / 200
## Errors
nginx 5xx 312 in window (usual: ~20)
## Top endpoints: request rate vs same window yesterday
GET /api/assets/search 2,410/min baseline 590/min x4.1
POST /api/inspections 180/min baseline 170/min x1.1
Every source is fetched separately, so a monitoring outage leaves one section empty instead of killing the investigation. A follow-up question about the same window reuses the snapshot instead of fetching it again.
Let plain code flag the obvious
The snapshot also gets a short summary of what looks off. When any endpoint runs at two or more times its baseline, plain code adds a line about it at the top of that summary, before the model writes anything.
def traffic_anomalies(endpoints, threshold=2.0):
"""Deterministic, so the model can't reason its way past it."""
return [
f"{e.name} is at {e.ratio:.1f}x its usual request rate"
for e in endpoints
if e.baseline and e.ratio >= threshold
]
A model reading a page of metrics can miss the one line that matters, or explain it away. A rule can’t. Use code for anything you can define precisely and save the model for the things you can’t.
Don’t stop at the first broken thing
The first thing the tools show you is rarely the cause. It is the most visible symptom. The early version of the agent would find a slow query or a worker restart and call it the root cause. That was the most common way it was wrong.
So the prompt makes it reason as a chain and name each link:
Symptom API responses slow from 18:12 to 18:31
Proximate database CPU at 95%, one search query dominating
Trigger search traffic 4x normal from 18:08
Root a frontend release at 18:04 fires a search on every keystroke
The rule that made the biggest difference: you are not done when you can name a broken component. You are done when you can answer why it broke now, and not yesterday. A slow query that has been slow for months did not cause tonight’s incident. Something changed.
There is also a self-check. Before blaming infrastructure (a crash, memory, a dropped connection), the agent has to confirm it actually looked at endpoint traffic and host metrics for that service. If it skipped either, it is not finished.
A confident wrong answer is worse than “not sure”
Every investigation ends with a confidence score from a separate, tiny model call that only reads the final narrative. The scale is simple: a clear root cause and a clear trigger score high, a named symptom with no cause scores low.
And there is a cap. If the narrative mentions truncated results, partial data or an unknown trigger, the score can’t go above 0.5, however convincing the story sounds.
With all of this in place, the agent does the first hour of the routine in a few minutes. The engineer still decides what to do about it. But they start from a written chain of evidence instead of an empty dashboard.