Choose a deterministic workflow harness for LLMs by writing one workflow's input, output and allowed-action contract, then locating the code and records that enforce it. Compare candidates with the same restart, retry and review probes. The useful evidence is a confirmed transition or receiver outcome that an operator can inspect. This guide provides the checklist, a documented three-pattern comparison and 17 measured checks from an original local fixture.
Start with one workflow and its action contract
Use one task throughout the comparison: review a pinned pull-request diff, produce a structured summary, and post one comment after human approval. The publishing step is an illustrative extension to the workshop example below. A model's review verdict supplies input to this gate; it does not authorize posting or merging.
Write the contract before choosing the runtime:
| Contract field | This example |
|---|---|
| Input | Repository, PR number and exact diff revision |
| Proposed output | Review summary and proposed comment |
| Allowed action | One comment to the approved destination |
| Identity | Stable run ID and logical operation ID |
| Review | Named reviewer approves the exact action |
| Bounds | Explicit attempt limit and deadline |
| Outcome | Completed, rejected, failed or awaiting reconciliation |
This makes the selection question concrete: which layer enforces each row, and what record survives if that layer stops? A database of review summaries helps with inspection, but recovery also needs code that reads the saved state and decides what can run next.
What deterministic workflow control does—and does not—mean
Anthropic's architectural distinction is useful here: workflows use predefined code paths, while agents let the model direct their process and tool use. You can place a flexible model step inside a workflow whose surrounding transitions remain explicit.
Keep four properties separate:
| Property | Question to verify |
|---|---|
| Code-controlled transitions | Given the stored input, which step is permitted next? |
| Recorded-result reuse | Does resume use the previous model result or request another? |
| Restart recovery | Which persisted record restores an interrupted run? |
| Duplicate-effect prevention | How does the receiver recognize a repeated operation? |
A model can propose different wording while application code still rejects an unknown tool or routes a proposed comment to review. Conversely, repeatable mock text does not tell you whether a worker survives a restart.
For an inspected Python loop, the earlier Starkslab source-and-trace walkthrough provides the execution-level starting point. This checklist moves outward: it asks how the surrounding workflow governs that loop.
The accompanying original diagram marks the important branch after dispatch: if acknowledgement is missing, the outcome is unknown until the operation is reconciled. Keep that branch visible in your own design.
Does a deterministic workflow make an LLM return the same answer?
Deterministic routing governs which transition is allowed for the stored input; it does not make fresh model text identical. You can reuse a recorded result when the workflow supports it, or request a new result and validate it before continuing. Check the model-call boundary separately from routing, state persistence and tool execution.
Compare three execution patterns with the same agent task
The public Python workflow workshop describes the same PR-review core across three execution patterns, pinned here to commit 05784770d9d73d887f59808c50e1491c86dd11af. That controls the comparison's task shape while changing how work is invoked.
This is a documented architecture comparison. The named runtimes were not deployed or benchmarked for this article.
| Pattern | Dispatch boundary | State to inspect | Recovery owner | Application work that remains |
|---|---|---|---|---|
| In-process service | HTTP request runs the review | Saved review and execution records | Application | Restart handling, review gate and comment write |
| Queue-backed worker | Producer enqueues; worker consumes | Job, acknowledgement and review records | Queue and worker code | Retry/ack policy, review gate and comment write |
| Managed workflow tasks | Registered tasks invoked through SDK/API | Task results plus application records | Task runtime for configured retries | Action authority and receiver semantics |
The workshop README describes a deterministic mock option, so its model step can be exercised without a provider key. That is a documented facility, not a report that its test suite ran here.
Check the managed boundary
Render's current documentation describes managed task queuing, separate run instances and configurable retries. It also lists scheduled task triggering as requiring an external cron job. Those statements identify infrastructure ownership; they do not establish this application's comment deduplication or human authorization.
Comparison method
Hold the pinned PR-review task constant across the workshop's in-process, queue-backed and managed-task patterns. Compare dispatch, state, recovery ownership and application action work using the same criteria. This is a documented architecture comparison; named runtimes have no observed test score.
| Option | Dispatch boundary | State to inspect | Recovery owner | Application work remaining | Evidence status |
|---|---|---|---|---|---|
| In-process service | HTTP request runs the shared review Sources: [7] | Saved review and execution records Sources: [7] | Application Sources: [7] | Restart handling, review gate and comment write Sources: [7] | Documented design; runtime probes unverified Sources: [7] |
| Queue-backed worker | Producer enqueues; worker consumes Sources: [7] | Job, acknowledgement and review records Sources: [7] | Queue and worker code Sources: [7] | Retry/ack policy, review gate and comment write Sources: [7] | Documented design; runtime probes unverified Sources: [7] |
| Managed workflow tasks | Registered tasks invoked through SDK/API | Task results plus application records | Task runtime for configured retries | Action authority and receiver semantics | Documented design; runtime probes unverified |
Comparison limitations
This is a documented architecture comparison of one pinned PR-review task. No named runtime or hosted platform was executed or given an observed reliability score. The separate original SQLite fixture verifies only its own contract. Application review and receiver semantics require verification on the candidate.
When is a queue-backed worker enough for an LLM workflow?
A queue-backed worker can fit a workflow that needs execution outside the web request when you can own its acknowledgement, retry and recovery rules. The workshop's queue pattern makes those responsibilities explicit. Test redelivery and interrupted workers with the same input, then verify review authority and external actions independently of the queue.
The operator’s checklist: locate each control and its owner
For each candidate, record the enforcing code or service, the retained artifact and one failure probe. Use the same inputs and criteria throughout.
| Boundary | Artifact to request | Owner to name | Failure probe |
|---|---|---|---|
| Routing | Permitted steps and transition predicates | Application/runtime | Submit an unknown step |
| Tool access | Allowlist, input contract and dispatch result | Application/gateway | Propose a disallowed tool |
| Model result | Input/config revision and saved response | Application/storage | Resume after the response is saved |
| Run state | Step status and recoverable checkpoint | Runtime/storage | Restart the worker |
| Retry/concurrency | Attempt policy, claim/lease and acknowledgement | Runtime/queue/worker | Redeliver while work is claimed |
| Operation identity | Stable key and receiver receipt | Application/receiver | Lose acknowledgement after acceptance |
| Human review | Action digest, reviewer and policy version | Application | Change content after approval |
| Stop/inspection | Deadline, cancel state and correlated events | Application/runtime | Request cancellation during dispatch |
A prompt asking the model to respect a rule is not the code predicate that enforces it. A trace records a decision; an approval gate must determine whether execution can proceed.
Use Starkslab's framework control-surface guide for the wider map of framework, application, gateway and infrastructure responsibilities. This table adds the artifacts and probes needed to evaluate those responsibilities.
Durable execution: inspect the crash window around a write
The difficult crash window starts when the receiver may have accepted the write but the worker has not recorded its receipt. The local fixture uses separate SQLite files for those two records, so terminating the worker does not erase an accepted fake comment.
Its lost-response probe produced this sequence:
-
Persist the dispatch intent and stable operation key.
-
Commit one fake comment at the receiver.
-
Terminate the worker process before recording completion.
-
Start a new worker and look up the same operation.
-
Record the existing receipt; keep the effect count at one.
The resumed worker made no second dispatch attempt in this case. That result depends on the fake receiver's lookup contract, which the fixture deliberately supplies.
| What is known | Next action to evaluate |
|---|---|
| Definitely failed before acceptance | Re-check gates and retry within the budget |
| Receipt confirms acceptance | Record the known result |
| Acceptance cannot be established | Reconcile or escalate before another write |
When lookup was disabled, the fixture stopped at needs_reconciliation. Its negative control reused the comment with a new operation key and created two fake effects. Keep one key for retries of the same logical operation; a genuinely new operation needs its own identity.
LangGraph's persistence documentation provides another concrete check: MemorySaver and InMemorySaver lose their RAM checkpoints on process restart. A recorded model answer, a recoverable checkpoint and a receiver receipt solve different parts of this problem.
Evaluate workflow retries separately from receiver duplicate handling: a permitted retry still needs the correct logical operation identity.
Does durable execution prevent duplicate external actions?
A persisted checkpoint and a retry policy do not establish the receiver's duplicate-handling contract. In this fixture, a stable operation key and receiver lookup let a restarted worker confirm one accepted fake comment. The same payload with a new key created two effects. Verify those mechanisms on the actual receiver before relying on them.
Human approval: bind the decision to the action that executes
A deployed review record should identify the run, logical operation, input/content revision or digest, destination, reviewer, decision time and applicable policy version. At dispatch, validate the current action against that record.
The digest helps identify the reviewed payload. Authorization still requires application logic to establish who may approve that action and whether the decision remains valid.
In the local fixture, rejection caused no write. Editing the approved content or changing the policy version returned the run to needs_approval; an unrecognized fixture reviewer could not authorize it. These are observations of a test identity and local predicates, not a demonstration of real account authentication.
LangGraph documents a concrete pause/resume mechanism using a checkpointer, stable thread ID, interrupt() and Command. Its interrupted node starts again from the beginning on resume, so code before the interrupt runs again. Inspect any effect on that path before placing the gate.
Also separate an agent's review verdict from the human decision. A judge that outputs “approve” can help evaluate a PR; the application must still decide whether the proposed external action is authorized.
Is a human approval prompt enough to govern tool execution?
The application must enforce the decision at the dispatch boundary. Bind review to the destination and exact action, validate the reviewer and applicable policy, and require another decision when the action changes. LangGraph's interrupt mechanism can pause execution, but its resumed node reruns preceding code. Inspect the effect path around that pause.
Run failure probes before adopting the harness
The verification pack contains harness_probe.py, an original standard-library Python exercise. Its scripted proposal stands in for model output, and every comment is a synthetic record in a local fake receiver.
Run it from the unpacked directory with Python 3.10 or newer:
python3 harness_probe.py probe --output my-verification-run
Choose a new output directory. The script preserves results.json, environment.json, each case's events and the two SQLite records instead of overwriting an earlier capture.
Seven scenario groups
| Scenario | Observation in the local fixture |
|---|---|
| Valid completion | One confirmed fake comment |
| Invalid/disallowed action | Empty content and merge_pr each produced zero effects |
| Lost response | Stable-key lookup recovered one effect; absent lookup stopped for reconciliation |
| Worker restart | Recovery succeeded after a pre-effect crash and from a saved approval |
| Review changes | Rejection, edited content, policy drift and wrong reviewer prevented dispatch |
| Duplicate trigger/resume | Repeated delivery kept one effect |
| Budget/cancellation | Two-attempt limit and pre-dispatch stops held; cancellation after acceptance retained the effect |
Seventeen checks matched their expected outcomes on 2026-10-14, including the negative control that deliberately created two effects. Python was 3.14.3, SQLite 3.51.2, on macOS 27.0.1/arm64. The exact script SHA-256 and captured events are in the pack.
Know what the exercise establishes
The crash cases terminate an actual child process at selected committed boundaries. Deadline uses an injected logical clock, and cancellation is injected between receiver acceptance and worker acknowledgement.
These serial probes do not validate concurrent worker isolation. They also do not test LangGraph, Valkey, Render, a live model, GitHub or a remote deployment. Recreate the same cases on a pinned candidate before filling its worksheet with observed results.
Before you start
Python 3.10 or newer, a terminal, the accompanying original harness_probe.py, and a new writable output directory. The fixture uses only the standard library; no service, provider key or network is required.
Write the action contract
Record the pinned input, expected summary, approved destination, logical operation identity, attempt/deadline bounds and terminal states.
Expected result
The required state and effect for each normal, rejected and interrupted case is explicit.
Sources: [2]
Run the local fixture
From the unpacked verification pack, run python3 harness_probe.py probe --output my-verification-run. Use a fresh directory so earlier captures remain available.
Expected result
Seventeen records match their expected outcomes; the deliberate changed-key negative control records two effects.
Inspect persisted evidence
Read results.json, environment.json and each capture.json. Inspect the separate worker/receiver SQLite files for the lost-response case and compare before/after resume state.
Expected result
One accepted fake effect survives the process exit and is reconciled by the restarted worker.
Repeat equivalent probes on a pinned candidate
Recreate the same input and seven failure groups with the candidate's real storage, queue/runtime, review gate and receiver. Preserve its version, configuration, actual outputs and unresolved work.
Expected result
The worksheet contains observations of that candidate rather than transferred claims from the local fixture.
Record the selection decision
Name the owner and evidence level for every required boundary. Mark an unrun property unverified and keep the next test needed to resolve it.
Expected result
The choice follows the demonstrated contract and explicit remaining work.
Sources: [26]
Verify the result
The preserved local capture shows 17/17 expected outcomes. Its metadata pins Python 3.14.3, SQLite 3.51.2, macOS 27.0.1/arm64 and script SHA-256 1f8abb48539ae090bb003ebb51b44f80d21afa209dfe665426816d3beb633b4e. Hosted candidates remain unverified; the supplied lookup/deduplication and identity predicates belong to the fake receiver and original fixture.
Troubleshooting
A FileExistsError means the requested capture directory already exists; choose a new directory instead of overwriting it. A needs_reconciliation result means the fixture cannot establish the in-flight operation's outcome through its lookup interface. Read the effect record and retained operation identity before another dispatch. An edited payload or changed policy must return to review.
Command
python3 harness_probe.py probe --output verification-runActual output
{"case":"lost_response_after_effect","observed":{"attempts":1,"effects":1,"status":"completed"},"passed":true}Command
python3 harness_probe.py probe --output verification-runActual output
{"case":"negative_control_changed_operation_key","observed":{"attempts":0,"effects":2,"status":"approved"},"passed":true}Command
python3 harness_probe.py probe --output verification-runActual output
{"case":"cancel_after_receiver_acceptance","observed":{"attempts":1,"effects":1,"status":"completed_after_cancel"},"passed":true}Keep a harness decision worksheet you can revisit
Keep a decision record that survives your next framework change:
| Required boundary | Responsible layer | Source/config | Probe result | Remaining work |
|---|---|---|---|---|
| Recovery for the in-process pattern | Application | Pinned workshop design | Unverified | Exercise restart recovery |
| Queue acknowledgement and redelivery | Queue/worker | Same task and input | Unverified | Exercise duplicate delivery |
| Managed task retry | Task runtime | Current task settings | Unverified | Exercise configured retries |
| Authorized comment and duplicate handling | Application/receiver | Exact action and operation key | Locally observed only in the fixture | Verify the actual receiver contract |
An in-process runner can fit a bounded interactive task if its required recovery is supplied. A queue moves execution out of the request while adding acknowledgement and worker ownership. A managed task runtime can reduce that infrastructure work; verify the remaining application boundaries explicitly.
Choose against the contract you can demonstrate. If a necessary row is unverified, record the work that would establish it instead of converting a documentation feature into a numerical score.
The pack includes a reusable JSON worksheet. Return to the AI Agent Tools hub for the broader stack and workflow routes, including the separate coding-CLI harness lane.