Article: Implementing Durable Workflows on Postgres Without an External Orchestrator
Raman Varma · rss · 2026-09-14 Reader 🖥️
Cascading Failure & System Resilience PatternsDurable Execution & Workflow CheckpointingLong-Running & Managed AgentsSoftware Design FundamentalsSRE & Incident Response
Durable workflow execution is fundamentally a database-checkpointing problem, so a team that already operates a relational database like Postgres at scale can implement production-grade durable execution — concurrent dispatch, idempotency, crash recovery, durable waits, and observability — as a handful of well-understood SQL primitives, without deploying a separate stateful orchestrator such as Temporal or AWS Step Functions.
practical · mixed — Practical: the piece argues for a specific engineering decision (build durable workflow execution on Postgres instead of adopting an external orchestrator like Temporal). Form is mixed — expository technical walkthrough (schemas, SQL, code) carrying an argumentative case, opened and closed by narrative framing from the author's own system (Kestrel Workflows).
Structure
| Part | Role |
|---|---|
| Opening: the Kestrel Workflows origin story and the autosave analogy | context |
| "The Default: Using an External Orchestrator" — describes the standard orchestrator model and enumerates its costs (stateful single point of failure, own access-control surface, KV data model resists relational analysis) | counter-position, set up to be rejected |
| "Postgres as the Orchestrator" — the workflow_executions table and the core reframing (durable execution is a checkpointing problem, and we already have a checkpoint store) | thesis statement / pivot |
| "One Clause That Makes Everything Safe" (SKIP LOCKED) — concurrent dispatch without a broker or leader election | evidence (technique 1: concurrency) |
| "Let the Database Enforce Idempotency" — unique-constraint checkpoint table | evidence (technique 2: idempotency) |
| "Crash Recovery: Leases and a Sweeper" | evidence (technique 3: fault tolerance) |
| "Durable Sleeps and Human Approvals" — waits as rows with wake_at | evidence (technique 4: long/arbitrary waits at constant cost) |
| "Why This Scales" — throughput/availability claims, and an explicit scope limit (favor Temporal for sub-millisecond, thousands-of-workers fan-out) | rebuttal to an anticipated objection, with a conceded boundary condition |
| "Operational Considerations" — connection pressure/PgBouncer, dead-tuple bloat/autovacuum/partitioning | honest caveats / operational cost |
| "Observability for Free" and "Reliability and Security Collapse to One Dependency" | conclusion, restating the unity in benefit terms |
Key terms
- durable execution Durable Execution & Workflow Checkpointing
A program that checkpoints its progress to an external durable system as it runs, so that if the process dies, another process can reload the last checkpoint and continue from there — "autosave from a video game, but for backend code." - SKIP LOCKED Durable Execution & Workflow Checkpointing
The `SELECT ... FOR UPDATE SKIP LOCKED` clause used as the sole concurrency primitive: it lets one worker claim a row while every other concurrent poller skips past already-locked rows instead of blocking, giving one-worker-per-row claim semantics with no broker, leader election, or external lock service. - idempotent checkpoint Durable Execution & Workflow Checkpointing
A step's output written exactly once via a primary-key/unique constraint on `(execution_id, step_id)`; a rerun after a crash is detected by a failed insert (or a check-before-execute read) and returns the prior stored result instead of repeating the step's side effect. - lease
A time-bound ownership claim on a workflow_execution row, recorded as `owner_id` + `lease_expires` and renewed by a periodic worker heartbeat; when a lease lapses without renewal, a sweeper reclaims the row. - sweeper
A periodic query (not a separate service) that resets `running` executions whose lease has expired back to `enqueued`, and separately re-enqueues `waiting` executions whose `wake_at` has passed. - exactly-once processing guarantee
Used to describe the property that SKIP LOCKED gives: no two workers ever claim the same row.
Propositions
- explicitDurable execution does not require a dedicated external orchestrator like Temporal or AWS Step Functions; a relational database you already operate can play that role and remove a stateful, external system from your critical path.conf: highsupportsJobRunr Introduces ClawRunr, an Open-Source Java AI Agent · p2 — Same underlying claim — reuse an already-operated piece of infrastructure as the durable-execution substrate instead of adopting a dedicated engine — applied to Postgres directly instead of a Java background-job library.
- explicitcorroboratedSELECT ... FOR UPDATE SKIP LOCKED turns an ordinary Postgres table into a concurrent work queue where each row is claimed by exactly one worker, with no message broker, leader election, or external lock service.conf: highextendsCloudflare Turns CI Pipelines into TypeScript Workflows · p1 — Both describe checkpointed, resumable step execution as the core benefit of a workflow substrate; this article extends the pattern to a self-hosted, database-only substrate instead of a managed Workflow platform.supports ←Presentation: The Agent Harness: Control Planes, Invariants, and Approval Boundaries for Production AI Agents · p6 — Both converge on 'exactly one worker/writer claims a mutable unit of work at a time' as the core concurrency invariant — this talk's harness-level 'lane' and Varma's database-level SKIP LOCKED queue are the same pattern at different layers.
- explicitdevelopedMaking step checkpoints a primary-key/unique constraint lets the database enforce idempotency, so a step that reruns after a crash reads its prior result instead of repeating a side effect.conf: highextendsCloudflare Turns CI Pipelines into TypeScript Workflows · p3 — Cloudflare's piece states the idempotency requirement for retryable steps as a bare constraint on the developer; this article supplies a concrete database-enforced mechanism (unique-constraint checkpoint table, check-before-execute) for satisfying it.extends ←Presentation: The Agent Harness: Control Planes, Invariants, and Approval Boundaries for Production AI Agents · p4 — Applies the same 'one owner, one replay path' invariant this talk states in general harness terms to durable-workflow engineering's concrete mechanism (a unique-constraint checkpoint), grounding the same design principle in a specific, testable implementation.
- explicitdevelopedCrash recovery can be implemented with a lease-and-sweeper pattern: workers heartbeat the rows they own, and a periodic query re-enqueues any execution whose lease expired.conf: highextendsThe Production Gap: 5 Patterns for Building Long-Running AI Agents · p3 — Turing Post names checkpoint progress, partial-failure handling, and idempotency as the three required properties of any long-running process; this article gives a concrete Postgres implementation of exactly those three properties for workflows.extends ←Presentation: The Agent Harness: Control Planes, Invariants, and Approval Boundaries for Production AI Agents · p9 — Generalizes the lease-and-sweeper pattern from workflow-step crash recovery to agent tool-call liveness and timeout handling — 'a lease gives runs and tools a deadline' is the same mechanism applied one layer up.
- explicitStoring workflow state in the primary database makes observability a plain SQL query and collapses reliability and security to one dependency.conf: high
- explicitAn external orchestrator becomes a stateful single point of failure sitting in the critical path of every workflow, and requires its own deployment, security, monitoring, and upgrade lifecycle plus its own access-control and audit handling.conf: high
- explicitExternal orchestrators typically use a key-value data model optimized for workflow state, which makes ad-hoc analysis harder than querying Postgres directly.conf: high
- explicitDurable execution is fundamentally about checkpointing state to a durable external system, comparable to autosave in a video game applied to backend code.conf: high
- explicitThe claim transaction that dispatches work must be kept tiny — lock the row, flip its status, commit — because holding the FOR UPDATE transaction open while the step runs pins a row lock for the duration of potentially long-running external work.conf: high
- explicitSKIP LOCKED does not preserve FIFO ordering under contention, so this pattern is unsuitable for workloads that require strict ordering.conf: high
- explicitLease duration is a tradeoff: too short lets a slow-but-alive worker's work be stolen (wasteful but safe, given idempotent checkpoints); too long delays recovery after a genuine crash.conf: high
- explicitIdempotent checkpoints are a precondition for using aggressive, short lease durations safely, because without them a false-positive sweep would cause double-execution with real side effects rather than merely duplicated (wasted) work.conf: high
- explicitDurable sleeps and human-approval waits can be represented as a single row with a wake_at timestamp rather than an in-memory thread or goroutine, so an arbitrarily long wait — two hours or two weeks — costs the same: one row.conf: high
- explicitIn the author's experience, a single Postgres instance backing this design can handle tens of thousands of workflows per second without read replicas or sharding tools like Citus.conf: medium
- explicitAvailability under this design inherits directly from the team's existing Postgres high-availability setup (streaming replication, managed multi-AZ failover), because workers are fungible and can recover each other's state.conf: high
- explicitHigh-concurrency polling can exhaust Postgres' connection limit; this is mitigated with PgBouncer transaction pooling and back-off/jitter on idle workers' poll intervals.conf: high
- explicitA busy queue table under high churn accumulates dead-tuple bloat from UPDATE/DELETE traffic, addressed by tuning autovacuum, keeping the hot row set small with partial indexes, and partitioning completed executions away from the live queue.conf: high
- explicitIf a system needs fan-out across thousands of workers with sub-millisecond dispatch latency, an external durable-execution engine like Temporal may be the better architectural choice.conf: high
- explicitBecause no workflow data ever leaves the database, this approach avoids adding a new attack surface and a new system to audit.conf: high
- implicitThe argument implicitly treats the team's existing Postgres operability investment as fully transferable to workflow-shaped load, even though the article's own "Operational Considerations" section (connection pressure, dead-tuple bloat, autovacuum tuning, partitioning) describes tuning burdens specific to queue-shaped access patterns that a system-of-record workload alone would not surface.conf: medium
- implicitThe article implicitly treats "one fewer external stateful system to operate" as an unqualified architectural good, without weighing that binding workflow durability to the same database that is also the system of record raises the blast radius of a Postgres outage — a coupling cost the author names once at the very end but does not weigh against the isolation an external orchestrator would have provided.conf: medium
Arguments
A relational database a team already operates (Postgres) is a viable, and for a defined class of workloads a superior, alternative to a dedicated external orchestrator for durable workflow execution. strength: high
Explicit premises:
- Durable execution is fundamentally a checkpointing problem, and checkpointing requires writing to a durable external system.
- The team already operates Postgres as its system of record and has invested in making it perform at scale.
- Postgres primitives (SKIP LOCKED, unique constraints, timestamp columns) can implement concurrent dispatch, idempotency, and crash recovery.
- An external orchestrator adds a new stateful system with its own deployment/security/monitoring lifecycle, sits as a single point of failure in the critical path, needs its own access-control/audit surface, and uses a data model that resists ad-hoc relational analysis.
Implicit premises:
- Removing a system from the architecture is a net win whenever an existing system can absorb its function — the cost of one system doing two jobs is treated as categorically lower than the cost of two systems each doing one job.
- The workloads in scope are I/O-bound automation pipelines with modest concurrency, running seconds to hours — the argument does not implicitly generalize to all workflow use cases, though the article's opening bullets and headline read as a general recommendation before this scope limit appears near the end.
Inferences:
- Binding workflow durability to the same database as the system of record raises the blast radius of a single Postgres outage or major-version upgrade — the article names this coupling but does not weigh it as a cost against the isolation a separate orchestrator would provide.
- The 'reuse infrastructure you already operate well' argument generalizes beyond workflow orchestration to other typically-external stateful services (job scheduling, feature flags, rate limiting) that a team with a mature Postgres operation could evaluate the same way — the author does not extend the argument this far.
Aggressive, short lease durations are safe to use because checkpoints are idempotent. strength: medium
Explicit premises:
- If a lease is too short, a slow-but-alive worker can have its execution reclaimed and rerun by another worker.
- Idempotent checkpoints, enforced via a unique constraint, guarantee that a rerun step returns the prior stored result instead of repeating a side effect.
Implicit premises:
- The step's side effect and its checkpoint write are treated as if they were effectively one atomic unit — but the article's own mechanism (worker performs the side effect, then inserts into operation_outputs) leaves a window where a crash between the side effect and the checkpoint commit still produces a duplicate side effect on retry, which the unique constraint cannot prevent because nothing was written yet to conflict against.
Inferences:
- The claim 'the database, rather than the application code, enforces idempotency' is only fully true for the checkpoint record itself; for steps whose side effect is not atomic with the checkpoint write, correctness still depends on application-level design (e.g., making the external call itself idempotent via a passed idempotency key), which the article does not flag as a residual responsibility.
Narratives
- "It is like autosave from a video game, but for backend code" — the analogy used to introduce durable execution.
→ Makes the abstract checkpoint-to-external-system definition of durable execution (p8) intuitive without formal argument, priming the reader to accept the author's narrower redefinition of the term before the orchestrator-vs-Postgres comparison begins. - The motivating Kestrel Workflows scenario: a workflow triggers on a failing Kubernetes workload, runs an AI root-cause analysis, generates a fix, waits for human Slack approval, then opens a GitOps pull request — and any of these systems being rescheduled mid-workflow (likely, in Kubernetes) must not lose the execution.
→ Grounds the abstract need for durable execution in a concrete, high-stakes failure scenario (losing a human-approved incident fix) before any technical solution is presented, making the cost of not having durability vivid rather than asserted. - The paired Figure 1 (external orchestration diagram) and Figure 2 (Postgres-backed durable execution diagram).
→ A visual argument that shows architectural simplification (fewer distinct systems/boxes, one dependency) rather than stating it in prose — carries part of the unity's persuasive weight through diagram contrast alone.
Questions addressed
- ✅ How do you make workflow execution survive process death (crashes, node drains, pod evictions, reschedules) without losing progress?
Checkpoint state to Postgres after each step, plus a lease-and-sweeper pattern that detects and re-enqueues executions whose owning worker went silent. - ✅ How do you get exactly-once dispatch of queued work across multiple concurrent workers without a message broker or leader election?
SELECT ... FOR UPDATE SKIP LOCKED inside a tiny claim transaction. - ✅ How do you make a workflow step idempotent so retries don't repeat its side effect?
A unique constraint on (execution_id, step_id) in a checkpoint table, checked before a step executes — though see gap g2: this does not fully close the case where the side effect itself is not atomic with the checkpoint write. - ✅ At what scale, latency, or fan-out does the Postgres-backed approach stop being the right choice?
When you need fan-out across thousands of workers with sub-millisecond dispatch latency, an external engine like Temporal is named as the better choice — but this is asserted briefly, near the very end, without quantifying the actual threshold. - ✅ How do you keep the queue table itself from degrading under sustained high-churn UPDATE/DELETE load?
Autovacuum tuning, partial indexes on the hot row set, and partitioning completed executions away from the live queue. - ⭕ What happens to in-flight leases, waits, and transaction visibility during planned Postgres maintenance (major-version upgrades, planned failovers), as distinct from unplanned crash recovery?
Gaps
- illogicalThe article states that SKIP LOCKED gives "exactly-once processing guarantees," but the mechanism it actually describes only guarantees exactly-once *claim* — a row is assigned to exactly one worker at a time. Exactly-once *effect* (the property practitioners usually mean by that phrase) depends on the separate idempotent-checkpoint mechanism introduced two sections later. Stating the stronger claim in the SKIP LOCKED section conflates dispatch-level and effect-level guarantees.
Read next: Kleppmann's 'Designing Data-Intensive Applications' on effectively-once/exactly-once semantics, or Temporal's own documentation explaining why it separates at-least-once activity delivery from exactly-once effects via idempotency keys — the same distinction this article's own idempotency section implicitly relies on but doesn't name when making the SKIP LOCKED claim. - incompleteThe idempotency mechanism (unique constraint on operation_outputs, checked before executing a step) only guards against a duplicate *checkpoint write* — it does not, by itself, prevent a duplicate *external side effect* when a worker performs the side effect (e.g., an API call, a Slack message, a GitOps PR) and then crashes before the checkpoint insert commits. The article presents 'the database enforces idempotency' without discussing patterns other durable-execution systems use to close exactly this window (transactional outbox, idempotency keys passed to the external call itself, or a two-phase intent-then-commit record).
Read next: Temporal's or AWS Step Functions' documented patterns for making side-effecting activities idempotent under at-least-once retry semantics — the same reading direction already flagged by extract-cloudflare-ci-pipelines-typescript-workflows#g3, which this article partially but not fully answers. - incompleteNo versioning strategy is described for evolving workflow or step logic while executions are in flight for long periods (e.g., a two-week human-approval wait). If the code for a step, or the shape of the JSONB input, changes between when an execution is enqueued and when it resumes, there is no discussed mechanism for detecting or handling that drift.
Read next: Temporal's workflow-versioning / 'patched' API documentation, which exists specifically to solve this problem for long-running, resumable workflows. - incompleteThe two central scaling claims — 'tens of thousands of workflows per second on a single instance' and the 'sub-millisecond dispatch latency' threshold for preferring Temporal — are stated as bare figures with no benchmark methodology, hardware specification, step complexity, or measurement conditions, making the quantitative core of the scaling argument unverifiable by a reader trying to decide whether Kestrel's experience transfers to their own workload.
Read next: A follow-up benchmark piece from the author or Kestrel's own engineering blog, if published, describing load-test methodology and hardware. - uninformedThe article credits MySQL 8.0+, MariaDB 10.6+, Oracle, Db2, and SQL Server with SKIP LOCKED-equivalent mechanisms in a single aside, implying the pattern ports cleanly across databases, but does not note that these engines differ meaningfully in isolation-level defaults and MVCC/locking implementation (e.g., MySQL's gap-locking behavior under REPEATABLE READ) — real portability risk for a reader who lifts the exact SQL onto a different database.
Read next: Each database's own locking-read documentation, cross-compared for isolation-level defaults, before assuming this pattern is a drop-in replacement across engines.
Closes gaps elsewhere
- Cloudflare Turns CI Pipelines into TypeScript Workflows · g3 — Supplies the concrete idempotency-retrofit pattern that gap called for — check-before-execute against a checkpoint table guarded by a unique constraint on (execution_id, step_id) — though this article's own gap g2 shows even this pattern doesn't fully close the dual-write window for side effects that aren't atomic with the checkpoint write.
Related documents
- The Production Gap: 5 Patterns for Building Long-Running AI Agents
- JobRunr Introduces ClawRunr, an Open-Source Java AI Agent
- Cloudflare Turns CI Pipelines into TypeScript Workflows
- 5 Things World-Class Engineers Do That You Don't
- AI Is Changing Software Engineering. The Fundamentals Still Matter.
- Article: Implementing Chaos Engineering in Financial Payment Systems: Lessons from Enterprise ECS Deployments
- DoorDash Uses Envoy and Valkey for a 1.5M RPS Proxy Cache with 99.99999% Availability
- Google Open-Sources AX a Kubernetes Style Orchestrator for Autonomous AI Agents
- Presentation: How To Run on Three Clouds at Once, and When Not To
- What is “loop engineering?”