Skip to content
Docs
Build an agent
Surviving a crash

Surviving a crash

An agent that dies mid-task must not strand the work, and its replacement must not start from nothing. kslate’s answer is two things that outlive the process: the lease on a task field, and the checkpoint the agent saved.

This page stages a crash to show both.

Two workers need two identities. The starter class gives the remediator role to apikey:remediator and to every apikey:remediator/<name>, so two tokens are enough:

printf 'diag-token: diagnoser\nw1-token: remediator/worker-1\nw2-token: remediator/worker-2\n' > /tmp/keys.yaml
KSLATE_APIKEYS=/tmp/keys.yaml make run

w1 and w2 below are clients holding those two tokens.

The scenario

Worker 1 claims the remediation, checkpoints its progress, and dies. The diagnoser changes the plan while nobody is working. Worker 2 takes over.

c = w1.claim(R, "remediation", lease_seconds=3)
w1.checkpoint_save(R, {"goal": "drain node-7",
                       "progress": "cordoned; 4 of 11 pods evicted",
                       "decisions": "evict in batches of 4",
                       "open_questions": "none"})
# worker 1 dies here
worker-1  claim      owner=apikey:remediator/worker-1
worker-1  checkpoint 4 of 11 pods evicted
worker-1  ...dies
worker-2  claim      HELD by apikey:remediator/worker-1
diagnoser changed the plan while worker-1 was gone
worker-2  claim      took_over_from=apikey:remediator/worker-1

Three things to notice.

Nothing swept the dead worker. Worker 2’s first claim was refused with HELD, because the lease had not run out yet. kslate checks the lease when someone claims, not on a timer, so no component has to notice the death. The second claim, after the lease lapsed, simply succeeded.

The takeover is recorded. took_over_from names who was displaced, and history keeps it as a takeover row rather than a plain claim.

The lease is short here on purpose. Three seconds keeps the example quick. In production pick a lease two to three times one step of your work and renew at a third of it; run() does that for you.

What the replacement wakes up to 

KSLATE_APIKEY=w2-token bin/kslate resume --server http://127.0.0.1:8080 incidents/inc-111
role: remediator
checkpoint at record_version 2:
  {
    "goal": "drain node-7",
    "progress": "cordoned; 4 of 11 pods evicted",
    "decisions": "evict in batches of 4",
    "open_questions": "none"
  }
changed since this checkpoint:
  planned_action       v2    apikey:diagnoser
      evidence: eviction caused a second outage on node-4

That second half is the part a checkpoint alone would not give you. Worker 2 learns both where worker 1 got to and that the plan it was executing has since been replaced — so it does not evict the remaining seven pods under a plan that now says not to.

In code that is checkpoint_load, which returns the document your agent saved plus changed_fields.

Writing the checkpoint

Give Agent a build_checkpoint function and the class-scoped loop saves one after each successful cycle:

run(Agent(decide=decide, build_checkpoint=lambda fields: {"goal": ..., "progress": ...}),
    client, class_name="incidents", claim_field="remediation")

The class says which sections a checkpoint must carry; incidents requires goal, decisions, progress and open_questions, and a checkpoint missing one is INVALID_CHECKPOINT.

Checkpoint after each decision and after anything you could not safely repeat. kslate stores the document and never reads it: it is your agent’s own account of itself.

What this does not recover

Work done after the last checkpoint is unknown, unless the agent wrote it to an append field. That is the argument for appending as you go rather than at the end.

Next

  • Design a class: choosing fields, modes and roles for a problem of your own.
The agent loopThe audit trail