sudoX · reference
Inside sudoX
A level below the overview: what each part actually does, and the judgement calls that decide how it behaves. For someone who will operate or extend it.
The system
One Go binary. Two ingestion paths converge on one emission pipeline; every alert is persisted before anybody is notified; the estate is reached only through tool providers, never directly.
Signals in
Three ways something reaches sudoX. Two of them are learned; the third is pushed at it.
Logs
Seven backends — Elasticsearch, Loki, CloudWatch Logs, Google Cloud Logging, Graylog, Splunk and plain files. Lines are clustered into templates by Drain-style mining, and each template gets an EWMA baseline of its arrival rate. Something is raised only for a pattern that is genuinely new, or one arriving far faster than usual.
Cloud Logging is shaped like Elasticsearch rather than CloudWatch, and the reason is in what each API orders by. CloudWatch returns events in ingestion order, so a strict "newer than the cursor" never skips anything. Cloud Logging orders by the entry's own timestamp — the producer's clock — while visibility depends on when Google indexed it. Each tick therefore re-scans an overlapping window and deduplicates on insertId, because losing log lines silently is the worst failure an incident platform can have: nothing in the output distinguishes "nothing happened" from "we did not look".
Metrics
Golden signals — traffic, errors, latency, saturation — learned per service from PromQL, CloudWatch Metrics Insights, or Google Cloud Monitoring, all behind one learner with one set of statistics. Cloud Monitoring needed no client of its own: Google serves a Prometheus-compatible query API, so a GCP project is a Prometheus endpoint with a different address and credential. Two gates decide whether a sample is worth raising, and both must hold.
Statistical: at least N standard deviations from the learned mean. Without it, ordinary movement on a noisy metric alerts.
Practical: the sample also moved by a minimum fraction of normal. Without it, a very stable metric alerts on trivia — 100 ± 1 reading 103 is three sigma out and three percent different.
Direction is judged by kind: errors, latency and saturation only matter when they rise, because a service getting faster is not an incident. Traffic is the exception — demand collapsing usually means something upstream broke. Samples far enough out are held out of the mean, so a sustained incident cannot become the new normal and silence the detector while it is still happening.
Webhooks
Anything that can POST JSON. These arrive already classified, so they skip learning and enter the incident lifecycle directly.
Try it
The reorder window, entry by entry.
Six entries arrive per burst. One is out of order, one is a duplicate insertId, one is very late. The window holds what arrived until it is old enough to be placed in order; the cursor moves only past what was emitted. Change the window and send the next burst.
arrived this tick
held in the window
emitted, in order
in the console Watching (each source's cursor and last error) · pkg/signalsources/gcplogging.go
What to notice. With a 0s window the out-of-order entry arrives after the cursor has passed it and is counted as too late — never emitted, and never silently lost either: the count is on the source's row. With 5s it is caught and emitted in its proper place. The very late entry, twenty seconds old, is caught only by the 30s window, which is the trade the operator makes: a wider window is more complete and further behind. The duplicate is dropped under every setting, because the same insertId is the same entry however many times Google shows it.
From alert to record
An alert is persisted before notification fans out. An operator opens an investigation, and a read-only ReAct loop writes back an analysis record with its tool-call trail — every call it made, in order, so the conclusion can be checked rather than trusted.
The agents
Eight single-purpose model calls. None of them touches infrastructure — every one produces a record a human reads.
| Agent | Runs when | Produces |
|---|---|---|
| detect | a pattern is new or spiking | an incident classification |
| analyze | an operator asks about an incident | root-cause hypotheses with evidence and a tool trace |
| ask | an operator asks the estate a question | an answer whose every citation was checked against what the tools returned |
| act | an operator asks for a fix | an ActionPlan awaiting approval |
| inspect | an estate assessment, or a verification | findings, and measured before/after outcomes |
| extract | an operator enriches the graph | entity and relationship proposals |
| draft | a plan has executed | a runbook draft |
| postmortem | an incident closes | a postmortem from the records the platform already holds |
Only analyze and ask call tools, and only read-only ones — enforced twice, by a compile-time import guard for native tools and a runtime mode gate for MCP ones. Each task can run on its own model, because detect runs on every tick and is judged on cost while act is judged on being right.
Investigation and the knowledge graph
Seven deterministic record sources — services, incidents, episodes, patterns, runbooks, hosts, providers, and now environments — build a typed knowledge graph read by the analyze agent, the planner and the console. Operations questions are relational: "what does this depend on", "what fixed this before". The graph makes those hops first-class, so one call returns an evidence chain instead of loosely related snippets.
Try it
What depends on this?
A small estate as the graph holds it: an environment, two providers, the services they host, the databases the services read, and a finding about one of them. Hover or click a node. The answer to "what depends on it" is every node with a path into it — and the environment at the top says which estate the answer is about.
in the console Knowledge graph · Topology · Analyses (the evidence chain an analysis reads) · pkg/knowledge
What to notice. Click the RDS database: the two services that read it, the load balancer that fronts one of them, and the finding about it all light up, and the answer names prod. That is the hop an analysis makes in one call when an incident mentions the database — and the reason the environment is a node rather than a label: a label is a string on a record, a node is something a path can go through.
Enrichment: proposals, not writes
The extract agent turns completed analyses into pending proposals. An operator either accepts one into the graph, flagged as proposed so its provenance stays visible, or rejects it into a history that is never retrieved. Nothing a model invented enters retrieval on its own.
remediates joins an episode to an incident,
observed_in joins a pattern to a service and nothing
else. This is the Action Engine's approval gate, one layer down at
the data.
The Action Engine
The one path to changing anything. A plan is an artifact, not a conversation: what the planner wrote is what an operator approves and what the executor runs, with no gap between them.
{{step.field}}
output references, persists every transition, and on failure runs
the rollback chain in reverse order.
| Risk class | Meaning | Gate |
|---|---|---|
| read | observes, changes nothing | runs autonomously |
| reversible_write | changes something with a defined undo | needs approval |
| destructive | cannot be undone | needs two different people |
- Risk floors are clamped at parse time. A tool that can write cannot be smuggled into a plan labelled as a read.
- Dry run is real. Read steps execute for actual; write steps resolve their arguments and report what they would have invoked. Nothing is persisted, and failures still surface.
- Rollback runs on failure, each step's own undo, in reverse.
- Two-person control counts credentials, not names. One person holding two tokens is one approver. It used to count display names, and one human with
ayse-cliandayse-webcould pass the gate alone. - Where an environment is armed, cleared plans run unattended — three a pass, oldest first, with the policy re-evaluated at execution rather than trusted from creation.
Where tools come from
External MCP servers reach the platform through a client hub whose mode gate decides which catalog a tool may enter: a read-only server's tools join investigations, a write-capable server's tools are reachable only through an approved plan. The native cloud passthroughs (run_aws, run_gcloud, run_az) and the on-prem toolset feed both catalogs directly. A verb allowlist compiled into the binary decides what a passthrough may run — never the model.
Claim and result
A finding is a problem the platform found in the estate, tracked from discovery to a measured outcome. The lifecycle exists to keep one distinction honest — the gap between "a plan completed" and "the problem is gone".
needs_reverify — a question, not a
verdict. A broad sweep does not compare against the evidence
recorded when the finding was created, so it is not entitled to
overturn a verified result; the proven outcome stands until a
verification answers the question, and only a verification may
declare a regression.
Re-running an assessment updates the finding it already has, matched on scope and title, rather than creating a second one. That property is what makes the sweep safe to put on a timer — and verification runs on a schedule too, so "resolved" becomes "verified" without anybody remembering to check.
The loop closes
Executed plans become episodes that feed recall back into analysis and planning, and can be drafted into runbooks that join the corpus the analyze agent searches. Successes teach what to repeat; rollbacks teach what to avoid.
The estate
A cloud account's problems are relational, and relational problems are invisible in a per-resource listing. sudoX draws the estate as a typed graph, keeps every snapshot, and compares consecutive ones to answer what changed.
- The probe list is compiled in. A fixed set of describe/list calls per provider, never a command a model chose — otherwise "read-only" is a hope about the model rather than a property of the code.
- A failed probe is shown, not hidden. An unreadable subsystem must not render identically to an absent one.
- Drift is deterministic. If it says a security group opened at 14:20, that is a fact rather than an inference — and exposure changes are ordered above every other kind of change.
- The first look runs once per scope. Connecting a cloud draws it, derives what to watch, assesses it and reads its bill — and "once" means "there is no snapshot for it yet", so a reconcile that fires on every edit cannot double-spend.
Knowledge & memory
| Store | Holds | Read by |
|---|---|---|
| episodes | every executed plan and how it ended | the planner, so it repeats what worked |
| runbooks | written procedures, searchable by meaning | investigations, grounded in your own docs |
| graph | services, incidents, hosts, environments and how they relate | both, for connected evidence chains |
The graph is built deterministically from records the platform already trusts. An extraction agent can propose additions, but those are proposals until someone accepts them.
Configuration
Five files, each owning one concern. Everything that is a decision people revisit — the mode, the sweep schedule, what is watched, which environments exist — lives in the console instead, so changing it does not need a deploy.
| File | Declares |
|---|---|
| config.yaml | the agent, storage, notification channels, on-call, a model per task |
| agent_sources.yaml | which log backends to read, and which environment each watches |
| metric_signals.yaml | which PromQL to learn baselines for, and the CloudWatch / Cloud Monitoring backends beside it |
| mcp_servers.yaml | which providers and databases to connect, and their trust mode |
| onprem.yaml | the SSH inventory and per-host allowlists |