Durable Execution & Workflow Checkpointing
The pattern of checkpointing a program's progress to a durable external system (a dedicated orchestrator or a relational database) so execution can resume after a crash from the last completed step — the concurrency, idempotency, and fault-tolerance primitives (queue claiming, unique-constraint checkpoints, leases) that implement it, and the build-vs-buy question of whether a dedicated engine or existing infrastructure should host it.
Also known as: durable execution, durable workflows, checkpointed execution, workflow orchestrator, external orchestrator, Temporal, AWS Step Functions, lease-and-sweeper, idempotent checkpoint, SKIP LOCKED, crash recovery (workflow)
How authors use it
Oldest first (extraction date when no publication date is known).
| Author | Their term | Their meaning |
|---|---|---|
| Article: Implementing Durable Workflows on Postgres Without an External Orchestrator Raman Varma · 2026-09-14 | durable execution | A program that checkpoints its progress to an external durable system as it runs, so that if the process dies, another process can reload the last checkpoint and continue from there — "autosave from a video game, but for backend code." |
| Article: Implementing Durable Workflows on Postgres Without an External Orchestrator Raman Varma · 2026-09-14 | SKIP LOCKED | The `SELECT ... FOR UPDATE SKIP LOCKED` clause used as the sole concurrency primitive: it lets one worker claim a row while every other concurrent poller skips past already-locked rows instead of blocking, giving one-worker-per-row claim semantics with no broker, leader election, or external lock service. |
| Article: Implementing Durable Workflows on Postgres Without an External Orchestrator Raman Varma · 2026-09-14 | idempotent checkpoint | A step's output written exactly once via a primary-key/unique constraint on `(execution_id, step_id)`; a rerun after a crash is detected by a failed insert (or a check-before-execute read) and returns the prior stored result instead of repeating the step's side effect. |
Documents (4)
- Google Open-Sources AX a Kubernetes Style Orchestrator for Autonomous AI Agents Olimpiu Pop · 2026-09-22 source ↗ Reader 🖥️
Google's AX is a Kubernetes-style, open-source control plane that treats AI agents as stateful, checkpointable actors on a purpose-built runtime (Agent Substrate) rather than as microservices or batch jobs, solving the idle-compute and cold-start economics of long-running agentic workloads at the price of inheriting full Kubernetes operational complexity — a trade-off the technical community is already visibly split on rather than one the article itself adjudicates. - Presentation: The Agent Harness: Control Planes, Invariants, and Approval Boundaries for Production AI Agents Vinoth Govindarajan · 2026-09-21 source ↗ Reader 🖥️
The reliability of a production AI agent depends not on the model but on its harness — whether every fact an agent depends on has a single owner and replay path, concurrent state mutations are serialized through one ordered commit path, every unit of work has a bounded lifecycle, every approval is a scoped and expiring authority object rather than a remembered click, and every action is proven with a receipt that crosses the user-visible edge — because a model can only propose an action, and it is the harness, not the model, that commits it and proves it happened. - Article: Implementing Durable Workflows on Postgres Without an External Orchestrator Raman Varma · 2026-09-14 Reader 🖥️
Durable workflow execution is fundamentally a database-checkpointing problem, so a team that already operates a relational database like Postgres at scale can implement production-grade durable execution — concurrent dispatch, idempotency, crash recovery, durable waits, and observability — as a handful of well-understood SQL primitives, without deploying a separate stateful orchestrator such as Temporal or AWS Step Functions. - Technology Radar, Volume 34 Thoughtworks (Technology Advisory Board) · 2026-04-14 source ↗ Reader 🖥️
AI-assisted, agentic software development is moving faster than the industry's ability to evaluate it, which means teams must simultaneously retain long-held craftsmanship principles (testability, deliberate design, feedback loops) and rethink team/agent topologies, trust boundaries and metrics that were built for a pre-agent world.
Propositions touching this concept (10)
By publication month, oldest first (extraction month when no publication date is known).
2026-05
- design claimestablishedA mature background-job library (JobRunr) can serve as the durable execution substrate for an LLM agent.
2026-08
- explicitdevelopedMapping each CI pipeline stage to a Cloudflare Workflow step gives the pipeline checkpointed execution: a failed step retries with its state preserved, and a run can resume from that step instead of repeating the whole pipeline.
2026-09
- explicitDurable execution does not require a dedicated external orchestrator like Temporal or AWS Step Functions; a relational database you already operate can play that role and remove a stateful, external system from your critical path.
- explicitcorroboratedSELECT ... FOR UPDATE SKIP LOCKED turns an ordinary Postgres table into a concurrent work queue where each row is claimed by exactly one worker, with no message broker, leader election, or external lock service.
- explicitdevelopedCrash recovery can be implemented with a lease-and-sweeper pattern: workers heartbeat the rows they own, and a periodic query re-enqueues any execution whose lease expired.
- explicitAn external orchestrator becomes a stateful single point of failure sitting in the critical path of every workflow, and requires its own deployment, security, monitoring, and upgrade lifecycle plus its own access-control and audit handling.
- explicitDurable execution is fundamentally about checkpointing state to a durable external system, comparable to autosave in a video game applied to backend code.
- explicitSKIP LOCKED does not preserve FIFO ordering under contention, so this pattern is unsuitable for workloads that require strict ordering.
- explicitIf a system needs fan-out across thousands of workers with sub-millisecond dispatch latency, an external durable-execution engine like Temporal may be the better architectural choice.
- implicitThe article implicitly treats "one fewer external stateful system to operate" as an unqualified architectural good, without weighing that binding workflow durability to the same database that is also the system of record raises the blast radius of a Postgres outage — a coupling cost the author names once at the very end but does not weigh against the isolation an external orchestrator would have provided.
Open gaps (11)
- incompleteNo independent benchmarks or production case studies for AX's headline performance claims (sub-second resume, zero cold-start, 'dozens of tasks' multiplexing density) — unlike even the self-reported figures other infra pieces in this corpus supply (the 1M-suspended-sandboxes talk, GKE Agent Sandbox's 300 sandboxes/sec, 70% TTFT reduction).Google Open-Sources AX a Kubernetes Style Orchestrator for Autonomous AI Agents — read next: Watch for a follow-up InfoQ/HN post-adoption piece with real production numbers, or check the google/ax GitHub repo's own benchmarks/issues.
- uninformedThe piece never explains what happens to in-flight, non-idempotent side effects (e.g. a half-completed call to a third-party API) when an actor is checkpointed mid-execution — a known hard problem in durable-execution/workflow-checkpointing systems that AX's 'sub-second suspend with zero cold start' claim glosses over entirely.Google Open-Sources AX a Kubernetes Style Orchestrator for Autonomous AI Agents — read next: Check AX's GitHub docs/issues for how it handles non-idempotent external calls across a suspend/resume boundary; compare against extract-varma-durable-workflows-postgres's treatment of the same problem in a different system.
- illogicalThe article implicitly treats 'the technical community is divided' as itself validating AX's significance, when a distinct and more useful move would be to disentangle two separate questions it conflates by reporting them together: whether the underlying actor-substrate genuinely solves a real cost problem (an architecture question), versus whether Google's 'ergonomic workflows' marketing claim is accurate (a rhetoric question). A platform can be architecturally sound and still be overmarketed as easy to adopt — the piece never separates these.
- incompleteThe 'harness owns the order' prescription (serialize same-process writes with a queue, cross-process writes with a lock) assumes a single reachable, reliable lock/queue authority. It does not address genuinely distributed harness deployments — multi-region or multi-process across data centers — where the lock/queue mechanism itself must be distributed and has its own failure modes (partition, lease expiry, split-brain).Presentation: The Agent Harness: Control Planes, Invariants, and Approval Boundaries for Production AI Agents — read next: Read alongside extract-varma-durable-workflows-postgres, which operationalizes exactly this at the database layer (SELECT ... FOR UPDATE SKIP LOCKED, lease-and-sweeper crash recovery) and whose own gaps (g1, g5) already flag where 'exactly-once claim' guarantees and cross-engine locking semantics break down under contention and heterogeneity.
- incompleteThe claim that OpenClaw's contributor 'Peter' submitted roughly 600 pull requests in a day because of the proof-required review culture is offered with no independent verification, no baseline for comparison, and no accounting for PR size or quality — the same self-reported-evidence problem the corpus has flagged in other vendor/practitioner production-metrics talks.Presentation: The Agent Harness: Control Planes, Invariants, and Approval Boundaries for Production AI Agents — read next: Read alongside extract-anthropic-details-how-it-contains-claude#g4, which raises an identical evidentiary-independence gap about self-reported production security metrics, to see whether unverified headline numbers are a systemic weakness across 'here is our production agent's numbers' talks.
- uninformedThe document is an industry-wide, cross-client synthesis produced by a 22-person advisory board; it does not — and structurally cannot — speak to how any of these recommendations play out inside a specific team's existing culture, legacy codebase, or organizational politics. Every blip is stated as a general recommendation, not a contingent one.Technology Radar, Volume 34 — read next: Pair with team-level case studies or first-person practitioner accounts (e.g. this corpus's own harness-engineering and agent-config-decay extractions) to ground the Radar's generalizations in a specific team's experience.
- illogicalThe article states that SKIP LOCKED gives "exactly-once processing guarantees," but the mechanism it actually describes only guarantees exactly-once *claim* — a row is assigned to exactly one worker at a time. Exactly-once *effect* (the property practitioners usually mean by that phrase) depends on the separate idempotent-checkpoint mechanism introduced two sections later. Stating the stronger claim in the SKIP LOCKED section conflates dispatch-level and effect-level guarantees.Article: Implementing Durable Workflows on Postgres Without an External Orchestrator — read next: Kleppmann's 'Designing Data-Intensive Applications' on effectively-once/exactly-once semantics, or Temporal's own documentation explaining why it separates at-least-once activity delivery from exactly-once effects via idempotency keys — the same distinction this article's own idempotency section implicitly relies on but doesn't name when making the SKIP LOCKED claim.
- incompleteThe idempotency mechanism (unique constraint on operation_outputs, checked before executing a step) only guards against a duplicate *checkpoint write* — it does not, by itself, prevent a duplicate *external side effect* when a worker performs the side effect (e.g., an API call, a Slack message, a GitOps PR) and then crashes before the checkpoint insert commits. The article presents 'the database enforces idempotency' without discussing patterns other durable-execution systems use to close exactly this window (transactional outbox, idempotency keys passed to the external call itself, or a two-phase intent-then-commit record).Article: Implementing Durable Workflows on Postgres Without an External Orchestrator — read next: Temporal's or AWS Step Functions' documented patterns for making side-effecting activities idempotent under at-least-once retry semantics — the same reading direction already flagged by extract-cloudflare-ci-pipelines-typescript-workflows#g3, which this article partially but not fully answers.
- incompleteNo versioning strategy is described for evolving workflow or step logic while executions are in flight for long periods (e.g., a two-week human-approval wait). If the code for a step, or the shape of the JSONB input, changes between when an execution is enqueued and when it resumes, there is no discussed mechanism for detecting or handling that drift.Article: Implementing Durable Workflows on Postgres Without an External Orchestrator — read next: Temporal's workflow-versioning / 'patched' API documentation, which exists specifically to solve this problem for long-running, resumable workflows.
- incompleteThe two central scaling claims — 'tens of thousands of workflows per second on a single instance' and the 'sub-millisecond dispatch latency' threshold for preferring Temporal — are stated as bare figures with no benchmark methodology, hardware specification, step complexity, or measurement conditions, making the quantitative core of the scaling argument unverifiable by a reader trying to decide whether Kestrel's experience transfers to their own workload.Article: Implementing Durable Workflows on Postgres Without an External Orchestrator — read next: A follow-up benchmark piece from the author or Kestrel's own engineering blog, if published, describing load-test methodology and hardware.
- uninformedThe article credits MySQL 8.0+, MariaDB 10.6+, Oracle, Db2, and SQL Server with SKIP LOCKED-equivalent mechanisms in a single aside, implying the pattern ports cleanly across databases, but does not note that these engines differ meaningfully in isolation-level defaults and MVCC/locking implementation (e.g., MySQL's gap-locking behavior under REPEATABLE READ) — real portability risk for a reader who lifts the exact SQL onto a different database.Article: Implementing Durable Workflows on Postgres Without an External Orchestrator — read next: Each database's own locking-read documentation, cross-compared for isolation-level defaults, before assuming this pattern is a drop-in replacement across engines.