Checkpoint and resume

A checkpoint is a small JSON document an agent saves. It says what the agent is trying to do, what it has decided, and how far it has got. If the agent dies, its replacement loads the checkpoint and also gets a list of every field that changed since it was saved. The replacement continues from there. Nothing is replayed.

The document

{
  "goal": "restore payments-api error budget",
  "decisions": [
    { "what": "fail over payments-db to replica 2", "why": "replica lag, not the deploy", "evidence": "hypotheses@2" }
  ],
  "progress": "failover done; awaiting rollout verification",
  "open_questions": ["is replica 2 keeping up"],
  "references": ["pg_ctl promote on payments-db-2"]
}

The class says which sections a checkpoint must have:

checkpoint:
  required: [remediator]
  sections: [goal, decisions, progress, open_questions]

Save a checkpoint after each decision and after each action you could not safely repeat.

Save

request

PUT /v1/records/incidents/inc-4471/checkpoint
{"document":{"goal":"restore payments-api error budget","decisions":[{"what":"fail over payments-db to replica 2","why":"replica lag, not the deploy","evidence":"hypotheses@2"}],"progress":"failover done; awaiting rollout verification","open_questions":["is replica 2 keeping up"],"references":["pg_ctl promote on payments-db-2"]}}

HTTP 200

{ "record": "incidents/inc-4471", "role": "remediator", "checkpoint_id": 1, "record_version": 19 }

kslate stores the document with the record’s version at that moment. The checkpoint belongs to your role, not to you. Any agent in the same role can load it later.

A document missing a required section is rejected:

HTTP 422

{ "error": "INVALID_CHECKPOINT",
  "message": "missing required section(s): decisions, progress, open_questions",
  "record": "incidents/inc-4471" }

Load

request

GET /v1/records/incidents/inc-4471/checkpoint

HTTP 200

{
  "record": "incidents/inc-4471",
  "role": "remediator",
  "document": {
    "goal": "restore payments-api error budget",
    "progress": "failover done; awaiting rollout verification",
    "decisions": [
      { "why": "replica lag, not the deploy", "what": "fail over payments-db to replica 2", "evidence": "hypotheses@2" }
    ],
    "references": ["pg_ctl promote on payments-db-2"],
    "open_questions": ["is replica 2 keeping up"]
  },
  "record_version": 19,
  "changed_fields": [
    { "field": "planned_action", "version": 4, "author": "apikey:diagnoser", "evidence": "hypotheses@2" }
  ]
}

Look at changed_fields. The diagnoser changed planned_action after this checkpoint was saved. The replacement sees that before it acts. This load was made by a different agent in the same role.

From the command line, resume is the name for loading a checkpoint. There is no resume operation on the wire:

kslate resume --server $B --role remediator incidents/inc-4471

If no checkpoint has been saved, document is null and record_version is 0. changed_fields then lists every field you can read that has been written at least once. Task fields are never included.

Recovering from a dead agent

A replacement does three things:

claim the task. This succeeds as soon as the old lease expires. See Claims and leases.
Load the checkpoint. Read progress and changed_fields.
Continue. If progress says an action was already issued and nothing in changed_fields contradicts it, verify the action instead of repeating it.

The resolved gate

A class can require that certain roles have checkpointed before anyone sets status to resolved.

request

PUT /v1/records/incidents/inc-4471/fields/status
{"value":"resolved","expected_version":2}

HTTP 422

{ "error": "INVALID_CHECKPOINT", "field": "status",
  "message": "required role(s) have not checkpointed this record yet: remediator",
  "record": "incidents/inc-4471" }

kslate checks this inside the write transaction, so a concurrent write cannot slip past it. The trigger is fixed: status set to resolved.

Details

If your identity holds exactly one role in the class, you do not need to name it. If it holds several, pass ?role= on load. A load with no role named and several roles held is MALFORMED. Naming a role you do not hold is SCOPE_DENIED.
Earlier checkpoints are kept. Load returns the latest.
Two agents in the same role share one checkpoint. The lease on the task field is what keeps them from working the same record at once.

What a checkpoint does not do

It does not record anything after the last save. Work done between the save and the agent’s death is lost unless it was written to an append field.
It is the agent’s own account. kslate stores it and reports what changed since. It does not check the account against the record.

Claims and leases blame