The incident fleet

This guide shows four agents working one incident through a single kslate record. Each agent owns a few fields and reads the rest. No agent calls another. An agent acts when a field it reads changes. You will see the class that defines the record, what each field is for, and the writes from one real run. One of those writes is the one kslate rejected.

The class

The class defines the record’s fields, who may write each one, and how long records live.

apiVersion: kslate.dev/v1alpha1
kind: SharedContext
metadata: { name: incidents, namespace: kagent }
spec:
  fields:
    status:            { type: string, mode: versioned, enum: [open, mitigating, resolved] }
    affected_service:  { type: string, mode: soft }
    symptoms:          { type: append }
    hypotheses:        { type: append }
    planned_action:    { type: string, mode: versioned, evidence: required }
    remediation:       { type: task, lease: 300s }
    actions_taken:     { type: append }
    customer_updates:  { type: append }
  roles:
    detector:     { identities: [sa:kagent/detector],     read: ["*"], write: [status, affected_service, symptoms] }
    diagnoser:    { identities: [sa:kagent/diagnoser],    read: ["*"], write: [hypotheses, planned_action] }
    remediator:   { identities: [sa:kagent/remediator],   read: ["*"], write: [status, actions_taken], claim: [remediation] }
    communicator: { identities: [sa:kagent/communicator], read: ["*"], write: [customer_updates] }
    operator:     { identities: ["sa:kagent/kslate-cli", "user:*"], read: ["*"], write: [] }
  ttl: { after: 168h, onExpire: archive }
  checkpoint:
    required: [remediator]
    sections: [goal, decisions, progress, open_questions]

This class is the incidents document in testdata/contexts.yaml, which also holds two smaller classes used by the tests. The Helm chart installs a near-identical copy when you set example.enabled. Both add API-key identities alongside the ServiceAccount ones, so the suite can run without Kubernetes.

The fields

Four agents share the record. Each field has one job.

The communicator cannot set status. It reports on the incident. It does not declare the incident over. The operator role reads everything and writes nothing. The CLI runs as the operator.

One run

These are the writes from one run against this class. Versions are per field.

1. The detector appends symptoms entry 1. It then writes status = open with expected_version: 0, which creates the record.
2. The diagnoser appends hypotheses entry 1 with based_on: symptoms@1. It then writes planned_action v1 (“restart payments-api pods”) with evidence: hypotheses@1.
3. The remediator claims remediation. The lease is 300 seconds, from the class. It starts carrying out plan v1.
4. The detector appends symptoms entry 2: replica lag on payments-db.
5. The diagnoser appends hypotheses entry 2 with based_on: symptoms@2. It then writes planned_action v2 (“fail over payments-db to replica 2”) with expected_version: 1.
6. The remediator appends to actions_taken with based_on: planned_action@1. kslate rejects it:

{
  "error": "STALE_PREMISE",
  "record": "incidents/inc-4471",
  "field": "actions_taken",
  "current_version": 0,
  "current_author": null,
  "moved": [
    { "field": "planned_action", "your_version": 1, "current_version": 2,
      "current_author": "apikey:diagnoser", "current_evidence": "hypotheses@2" }
  ]
}

moved tells the remediator that planned_action went from version 1 to version 2, who changed it, and what they cited.

7. The remediator reads planned_action again. It carries out v2 and appends to actions_taken with based_on: planned_action@2. kslate accepts the write.
8. The remediator saves a checkpoint with progress: failover done; awaiting rollout verification. It then releases the claim.
9. The communicator appends to customer_updates.
10. The detector writes status = resolved. This works because the remediator checkpointed in step 8. Without that checkpoint kslate would reject it with INVALID_CHECKPOINT.

Step 6 is the reason this guide exists. The remediator was about to record an action based on a plan the diagnoser had already replaced. kslate rejected the write and said what had changed. Nothing told the remediator to wait. The diagnoser did not know the remediator existed.

Writing an agent for this class

the pages before this one covers writing an agent from scratch. For this class, the remediator is the interesting one. It runs as several copies, works every open incident, and must not act on a plan that has changed. With the Python client that is one call:

from kslate import Client, Agent, Write, run

def execute(fields):
    plan = fields["planned_action"]
    # ... carry out plan["value"] ...
    return [Write(op="append", field="actions_taken",
                  entry=f"executed: {plan['value']}", evidence="kubectl output")]

run(
    Agent(decide=execute, reads=["planned_action"], writes=["actions_taken"]),
    Client("http://kslate-gateway.kslate.svc", api_key=token),
    class_name="incidents",
    filters={"status": "open"},
    claim_field="remediation",
    lease_seconds=300,
    concurrency=4,
)

Each copy polls for open incidents and claims remediation on one. It reads the record, calls execute, and writes the result with based_on set to the planned_action version it read. Then it releases the claim.

A copy that finds the task already claimed skips that incident. That is how the copies share work without talking to each other. The agent loop explains each step.

The claim is on the remediation field, not the whole record, so the diagnoser keeps revising planned_action while a remediator holds it. If the plan changes while execute is running, the write is rejected and the loop calls execute again with the new plan.

The other three agents only read and write their own fields. The detector and communicator can run as single-record cycles triggered by an alert or a change. The diagnoser watches symptoms the same way the remediator watches planned_action.

If you cannot change the agent’s code at all, the kagent guide shows how to run the same loop as a sidecar.

Testing the scenario

You can test this scenario four ways. Each one asks a question about the record afterwards.

kslate verify incidents/inc-4471 reads the same history. It flags an agent that wrote without based_on, resubmitted a rejected write unchanged, misused the record, or never wrote at all. Add --move planned_action=<value> --watch 60s to change the plan while an agent is running and see what it does next.

A runnable version of this scenario, with a stub agent on kagent, is in a companion repository.