Skip to content
Troubleshooting

Troubleshooting

Every error from the gateway has an error code in the body. Find yours below. Each response here came from a real request.

Connection refused

curl: (7) Failed to connect to localhost port 8080

The gateway is not running, or is on another port. make run prints the address it serves. In a cluster, check kubectl -n kslate get pods and that you are using the kslate-gateway Service.

UNAUTHENTICATED

HTTP 401
{"error":"UNAUTHENTICATED"}

The gateway did not recognise your credential. One of three things:

  • the Authorization header is missing
  • the token is not in the gateway’s key file
  • the gateway runs with --identity k8s-sa and you sent an API key

Check the header is Authorization: Bearer <token>. Check the token is a key in the file passed to --apikeys. With make run, the keys are diag-token and rem-token unless you set KSLATE_APIKEYS.

NO_ROLE

HTTP 403
{"error":"NO_ROLE","record":"incidents/inc-7"}

The gateway knows who you are, but the class has no role for you. The key file maps your token to a name, say nobody, and the class’s roles have no entry whose identities includes apikey:nobody.

Add the identity to a role in the class, or use a token that maps to a name the class knows. The identity must match exactly, including the apikey:, sa:<namespace>/, oidc:<issuer>/ or mtls:<ca>/ prefix.

SCOPE_DENIED

HTTP 403
{"error":"SCOPE_DENIED","field":"planned_action","record":"incidents/inc-7"}

Your role exists but may not write (or claim, or read) this field. Here the remediator tried to write planned_action, which only the diagnoser may write.

Check the role’s write, claim and read lists in the class. If the error names a field you put in based_on, your role may not read that field.

WEAK_IDENTITY

HTTP 403
{"error":"WEAK_IDENTITY","record":"incidents/inc-42","field":"irreversible","required":"pod-bound","presented":"sa-bound"}

Your role may write this field, but the field’s minIdentityStrength is above what your credential is worth. Here the writer presented a ServiceAccount token on a cluster that does not report the pod binding, and the field requires pod-bound.

Present a stronger credential. That is usually a deployment change: run the writer in a pod with a projected token, or on a cluster whose TokenReview reports the binding. The gateway logs which kind of cluster it is on, once, at the first token it verifies. Or lower the floor on the field if the data does not need it. See Identity strength.

MALFORMED: expected_version is required

HTTP 400
{"error":"MALFORMED","field":"status","message":"expected_version is required for versioned fields","record":"incidents/inc-7"}

You wrote a versioned field without saying which version you read. Read the field first and pass its version. To create a field that does not exist yet, pass expected_version: 0.

MALFORMED: value must be one of the field’s enum

HTTP 400
{"error":"MALFORMED","field":"status","message":"value must be one of the field's enum: open, mitigating, resolved","record":"incidents/inc-7"}

The class restricts this field to a fixed set of values. Use one of them.

MALFORMED: id does not match the class idPattern

HTTP 400
{"error":"MALFORMED","message":"id does not match the class idPattern","record":"pr-reviews/abc"}

The class says what a record id looks like. Here pr-reviews requires a number. Check idPattern in the class.

EVIDENCE_REQUIRED

HTTP 422
{"error":"EVIDENCE_REQUIRED","field":"actions_taken","record":"incidents/inc-7"}

Every append needs evidence. So does any field the class marks evidence: required. Add it. It is free text: the command you ran, the log line you saw, the field you read.

NOT_FOUND

HTTP 404
{"error":"NOT_FOUND","record":"incidents/inc-404"}

The record does not exist. Records are created by their first write, so a get before any write is 404. If you meant to create it, write to it.

HTTP 404
{"error":"NOT_FOUND","message":"unknown class","record":"tickets/1"}

The class does not exist. Check the name against kslate classes or GET /v1/classes. On Kubernetes, check kubectl get sharedcontexts shows it with Ready true.

STALE

Someone changed the field after you read it. Not a bug. Read it, decide again, write with the new version. See Handling rejections.

STALE_PREMISE

Something you listed in based_on, or something the gateway inferred you read, changed after you read it. Not a bug. Read the fields in moved, decide again. See Handling rejections.

HELD

Another agent holds the task. Skip this record. If this came from a renew, stop and discard your result. See Claims and leases.

INVALID_CHECKPOINT

Your checkpoint document is missing a section the class requires, or you set status to resolved before a required role checkpointed. The message says which. See Checkpoint and resume.

CLASS_GONE

HTTP 410

The class was deleted. Its records are readable until they expire, but nothing can write to them. Recreate the class, or let the records expire.

UNAVAILABLE

HTTP 503

The gateway cannot reach Postgres, or is re-applying its schema. Wait and retry. If it persists, check the Postgres pod and the DSN the gateway was given.

RATE_LIMITED

HTTP 429

Your address sent too many requests with bad credentials. Fix the credential. The budget refills within a second.

My agent never does anything

  • In single-record mode, run() runs one cycle and returns. Call it again, or use class-scoped mode.
  • In class-scoped mode, check filters. {"status": "open"} matches nothing if no record has status set to open.
  • Check decide is returning writes. An empty list means “nothing to do” and nothing is written.
  • Run kslate verify <class>/<id>. ROLE_WROTE_NOTHING for your role means your agent’s identity has never written to that record.

My write was accepted but I expected a rejection

  • Did the other agent’s write land before your read? Then your read saw the new version and your write was correct.
  • Did you pass based_on=[]? That disables based_on for the write.
  • In class-scoped mode with a scope header, did the read and the write carry the same header?