How sudoX works
Watch, notice, investigate, plan, ask, act, prove, remember.
Eight steps, one agent each, in the order they happen — a shift that never ends. Three of them are things you can operate on this page: the baseline a metric learns, the catalog a log stream becomes, and the rules that decide where a page lands. Nothing here is a video.
The whole thing, in one picture
Reactive
Something broke
Alerts arrive by webhook from anything that can POST JSON, and the agent reads your logs and metrics directly — learning what normal looks like and raising something only when it stops. A second agent then investigates with read-only tools and writes up what it found and how.
Proactive
Something is wrong and nobody noticed
On a schedule, sudoX inspects the estate scope by scope — an account, a region, an environment — and records what it finds as a tracked finding: the observation, the evidence, the recommended fix, and how to prove that fix worked. Re-running a sweep updates what exists instead of duplicating it.
1 · watch — metrics
It learns what normal looks like.
Every metric signal the platform watches — a PromQL query against Prometheus or Cloud Monitoring, a Metrics Insights query against CloudWatch — is read on a tick and folded into a baseline: a running mean and spread per signal. Nothing is judged until the baseline is mature. After that, a sample outside k standard deviations is an event, and what happens to the event depends on the mode.
in the console Watching · Services · config/metric_signals.yaml · derived signals from the estate
What to notice. Switch to training and inject a spike: the spike is absorbed, and the band widens to include it — training's job is to learn, and it learns whatever it sees. Switch to shadow and inject the same spike: the sample is judged first and kept out of the baseline, so the band stays honest and the event is recorded as "would have raised". detect is the same judgement with a consequence: an incident, and on-call.
Deploy moves the level and leaves it there. In shadow or detect the first samples after it are events — that is correct, something did change — and after thirty in-band samples the baseline re-learns and the new level is normal. A learner that never re-learned would page forever about a deploy that worked; one that re-learned instantly would never notice anything.
The sensitivity is per signal in the product. Three sigma is the default because it is the point where a Gaussian signal produces about one false event in three hundred samples — at a thirty-second tick, one a few hours — and shadow mode exists so you can see that rate for your own signals before a single person is paged.
1 · watch — logs
Logs become patterns, and a pattern never seen is the signal.
A log stream is too fast to read and too varied to alert on by string. So each line is reduced to its template — numbers, ids and paths replaced by <*> — and the templates are the catalog: a few hundred patterns, each with a count and a rate per service. A line that matches a known pattern is a count. A line that matches nothing is the thing worth looking at.
the catalog
the stream
in the console Logs (the catalog and the detect log) · Services · Alert fatigue (learn exclusions)
What to notice. A normal line is a count going up. A rare line is still a count — rarity is not novelty, and the retry pattern with forty sightings is known even if it is unwelcome. The never-seen line goes to the top of the catalog marked new, and the mode decides whether that is a lesson (training), a recorded judgement (shadow) or an incident (detect). The severity comes from the words in the line — FATAL, refused, timeout — not from a model guessing.
An operator can hold a service or a pattern out of the catalog. That exclusion is deployment-wide, because there is one catalog: the learner reads the sources named in the deployment's own configuration, and a per-tenant exclusion would be a control the console could express and the learner could never apply.
2 · notice, 3 · investigate
Three agents, one question each.
detect
Is this worth a person?
Runs on every tick, over every event the baseline and the catalog produced. Judged on cost, because it runs constantly: a small model, rate-limited, that answers "raise / hold / learn" and records why. In shadow its answers are graded against what a person would have done.
analyze
What is actually wrong?
Runs once per incident with read-only tools — describe a service, query a metric, read a pattern's history, read a runbook — and writes a hypothesis with the calls it made attached. Read-only is enforced by the tool surface, not by the prompt.
act
What would fix it, exactly?
Drafts the plan: ordered steps, each with a risk class, a target, a command and a rollback. Judged on being right, so it gets the larger model. It never executes anything; it produces an artifact for the gate.
A fourth surface, Ask, lets a person put the same read-only tools to a question of their own — "what changed in payments since Tuesday" — and get an answer with its evidence. Every agent's model is chosen per task and per organisation, and changed without a restart. One model for all four is wrong for at least half the work.
4 · plan, 5 · ask
Nothing changes without a person
Reading is autonomous. Everything that writes stops at a gate.
An AI never touches infrastructure directly. It produces a plan — an inspectable artifact of ordered steps, each carrying a risk class, a target, and how to undo it. What the planner wrote is exactly what a named operator approves, and exactly what runs. There is no gap between the three for something to slip through.
| A step carries | Meaning |
|---|---|
| risk | read, reversible_write, or destructive. The policy matches on it; the two floors are keyed on it. |
| target | The connection and resource it acts on — which also names the environment the step is in. |
| command | The tool call, verbatim. What you read is what runs. |
| rollback | The tool call that undoes it. A write without one always needs a human. |
| dry run | Runs the read steps and reports what the writes would do. Never counts as an execution. |
Who must say yes is the approval policy's decision — nobody, one named person, or two different people — and there are two floors it cannot lower. Build a plan and watch it decide. Where an environment is armed, plans the policy cleared run unattended, three a pass, oldest first; the policy is re-evaluated at the moment of execution, and a plan whose answer changed goes back to pending. Run a pass.
6 · act, 7 · prove
A claim is not a result
This is the distinction the product is built around, and the reason its numbers mean anything. Walk a finding through it.
in the console Findings · Impact · Audit
| Stage | What it means |
|---|---|
| open | Found, with evidence recorded. Nothing done yet. |
| planned | A remediation is drafted. It still needs approval to run. |
| resolved | The plan completed. A claim — it says work happened, not that it worked. |
| verified | The estate was read again and compared against what was recorded when the problem was found. A result. |
Only verified outcomes count as impact. A dry run never counts as an execution and a rollback never counts as a success. If a problem comes back it is recorded as a regression rather than quietly reopened — and only a verification, which does the before-and-after comparison, is allowed to say so. A sweep that sees the same title again may only raise the question.
5 · ask — who hears about it
Where a page lands, and what is held.
Two policies sit between an alert and a person. Alert routing says where it goes — Slack, email, Teams, PagerDuty, on-call — matchable on service, severity and environment. The emit policy says what is held back: duplicates inside a window, a class of alert an operator has decided is noise. Held alerts are recorded, never dropped, so "why didn't I hear about this" has an answer. Switch the rules.
in the console Alert policy · Alert fatigue · Environments (the environment travels from the source onto the alert)
What to notice. The last alert has no environment, and a rule that names one does not match it — an unassigned connection is not secretly production, and it is not secretly staging either. It lands in the default channel, which is honest, and the Environments page counts it in front of you so that it stops being unassigned. The environment on the other five did not come from a naming convention: it travelled from the signal source, which belongs to a connection, which belongs to an environment.
Rules are read in order and the first match wins, which is why "critical in prod → on-call" sits above "any alert in prod → #ops" — turn the first off and the pool exhaustion goes to a channel nobody is watching at three in the morning. The console renders each rule as the sentence you see here, so a rule that reads wrongly looks wrong before it is saved.
The agent earns its way up
Three modes, in order. Each has to be worth leaving before the next.
| Mode | Behaviour | Who hears about it | Unattended execution |
|---|---|---|---|
| training | Watches and learns what normal looks like. Absorbs everything, judges nothing. | nobody | off |
| shadow | Judges, and records what it would have alerted on — so you can grade it against what people actually did. | nobody | off |
| detect | Opens real incidents and triggers on-call. The only mode in which cleared plans run without a person present. | on-call | where the policy clears it |
The mode is set per deployment and can be narrowed per environment — production can watch in shadow for its first weeks while staging runs in detect. It can never be widened per environment, so the org-wide switch stays a brake. Try both dials.
8 · remember
What happens after the fix.
Verification, on a schedule
A runner takes resolved findings — and verified ones a later sweep has questioned — five a pass, oldest first, and reads their scope again. It needs fresh evidence to compare against; without it, it says so and leaves the finding as it was rather than guessing. Every verification is audited with the environment named.
Impact, counted
Only measured before-and-after pairs. The Impact page shows how many findings were verified, how many regressed, and what each verified fix was worth in the unit the evidence was recorded in — seq scans per second, refused connections, dollars a month. Never an estimate.
Postmortems that lead with where
A postmortem is drafted from the incident, the analysis, the plan and the verification, and it leads with the environment — "prod (aws/…)" — because it is read by people who were not on the call, and four characters beat an account id they have to recognise.
Memory that has to be accepted
An episode is recorded automatically. A runbook is drafted from the result and offered. Knowledge the agent extracts — "payments depends on the ledger pool" — is a proposal in the graph until a person accepts it. What the agent reads back next time is what people agreed was true.
What it plugs into
Mostly through MCP, so a new system is a config entry rather than a code change. Read-only servers join investigations; write-capable ones are reachable only through an approved plan, and the passthrough's verb allowlist is compiled in.
Models are equally swappable — OpenAI, Anthropic, Gemini, DeepSeek, Qwen, or something self-hosted through Ollama — and a different model per task. Detect runs on every tick and is judged on cost; act is judged on being right. One model for both is wrong for half the work.
Why any of it can be believed
Every strong claim in the product is backed by something that would have caught it being wrong.
Discovery runs a fixed list of read-only commands compiled into the binary, never a command a model chose. A probe that fails is shown as a gap rather than hidden, because an unreadable subsystem must not look like an absent one. Impact counts only measured before-and-after pairs, never an estimate. Tenancy answers not found rather than forbidden, so one tenant cannot probe another's existence. And every simulation on this site runs the same function the product does — the dial you turned is the rule, not a picture of it.