Durable Execution for Agents with Temporal

AI agents are fundamentally different from traditional batch jobs. A traditional ETL pipeline is linear and bounded; if it fails, you retry the whole thing. An AI agent, however, is often a multi-step, stateful interaction involving external LLMs, database queries, and user inputs. If an agent crashes after calling an expensive LLM API but before saving the result, you have lost significant compute cost and potentially critical context.

Temporal provides a solution through durable execution. Unlike traditional frameworks where you manually serialize state to a database after each step, Temporal’s engine persists the entire workflow history automatically. This allows the system to reconstruct the agent’s state at any point in time, enabling true resilience against infrastructure failures, network partitions, and long-running inference tasks.

The Problem of Ephemeral State

Consider a typical AI agent workflow:

1. Retrieve user context from a vector database.
2. Call an LLM to generate a draft response.
3. Validate the response against safety guidelines.
4. Save the final response to the user’s profile.

If step 2 takes 30 seconds and the server hosting the code crashes at second 25, a standard asynchronous Python script using asyncio or Celery will lose its in-memory state. Standard scripts lose state, forcing developers to implement complex manual checkpointing to recover. This involves saving intermediate states to Redis or PostgreSQL after every step, handling partial completions, and managing race conditions. The complexity of implementing reliable manual state management often outweighs the benefits for simple workflows, yet it is essential for preventing data loss.

Temporal removes this burden. It treats the workflow code not as a script to be executed once, but as a deterministic state machine. The engine records every event (e.g., "Activity 'CallLLM' started", "Activity 'CallLLM' completed with result") in a persistent history store. As detailed in Temporal's documentation on durable execution, this history is immutable and append-only, serving as the single source of truth for the workflow's state.

The Replay Mechanism

The core innovation of Temporal is replay. When a workflow is paused due to an external event (like waiting for an LLM response), Temporal saves the current state of the workflow’s local variables and the history of events. If the process crashes or is restarted, Temporal does not resume from where it left off in memory. Instead, it replays the workflow code from the very beginning, using the stored history to drive the execution.

How Replay Works

1. Initial Run: The workflow code runs normally. When it encounters an external task (an Activity), Temporal schedules it and pauses the workflow, recording the event in history.
2. Crash/Restart: If the worker process dies, a new instance starts.
3. Replay: The new instance loads the history. It executes the workflow code line-by-line. When it hits the same Activity call, Temporal sees that the Activity has already been completed in the history. It returns the cached result without re-executing the external task.
4. Continuation: The workflow continues from where it left off, with all local variables restored to their state at the moment of the pause.

This mechanism ensures determinism. Because the replay uses the same code and the same history, the outcome is always identical. This is critical for AI agents, where non-deterministic behavior (e.g., calling an LLM twice) can lead to inconsistent state or double-charging.

Concrete Example

Consider a simple agent workflow in Python:

from temporalio import workflow
 
@workflow.defn
class AgentWorkflow:
    @workflow.run
    async def run(self, user_id: str) -> str:
        # Step 1: Fetch context
        context = await workflow.execute_activity(
            fetch_context,
            args=[user_id],
            start_to_close_timeout=datetime.timedelta(seconds=10)
        )
        
        # Step 2: Call LLM (Long-running)
        draft = await workflow.execute_activity(
            call_llm,
            args=[context],
            start_to_close_timeout=datetime.timedelta(minutes=5)
        )
        
        # Step 3: Save result
        await workflow.execute_activity(
            save_result,
            args=[draft],
            start_to_close_timeout=datetime.timedelta(seconds=10)
        )
        
        return draft

If the server crashes during the call_llm Activity:

• Without Temporal: The draft variable is lost. The next retry might call the LLM again, generating a different response (non-deterministic) or wasting money.
• With Temporal: On restart, the workflow replays. It sees that call_llm was already completed. It returns the cached draft value. The workflow proceeds to save_result. No LLM call is made. The state is perfectly preserved.

History Persistence and State

Temporal’s durability is achieved through Activities. An Activity is a function that performs non-deterministic work (e.g., HTTP requests, database writes, LLM calls). The workflow code is deterministic; the Activity code is not.

The Worker Loop

Temporal workers poll for tasks. When an Activity is scheduled, the worker executes it. If the worker crashes during execution, Temporal marks the task as "in progress" but not "completed." After a configurable timeout, Temporal reschedules the Activity to another available worker.

This creates a natural checkpointing mechanism. The "checkpoint" is the history record. You do not need to manually save state to a database after each step. The history serves the role of automatic checkpointing, avoiding confusion with frameworks requiring explicit checkpointing APIs. The history is the state.

Trade-offs

1. History Size: Every event is stored. For long-running workflows with many steps, the history can grow large. Temporal handles this via the Continue-As-New mechanism, which resets the workflow history, but you should design workflows to avoid unnecessary logging or large data payloads in history.
2. Replay Time: Replaying a workflow with thousands of steps can take time. However, for typical AI agent workflows (10–100 steps), replay is nearly instantaneous.

Resilience Patterns for Agents

Temporal provides built-in patterns for handling the inherent instability of AI systems.

Retry Policies

LLMs and APIs are flaky. Temporal allows you to define retry policies declaratively:

await workflow.execute_activity(
    call_llm,
    args=[context],
    retry_policy=RetryPolicy(
        initial_interval=datetime.timedelta(seconds=1),
        backoff_coefficient=2,
        maximum_interval=datetime.timedelta(seconds=10),
        maximum_attempts=3
    ),
    start_to_close_timeout=datetime.timedelta(minutes=5)
)

If the LLM call fails, Temporal automatically retries with exponential backoff. If it succeeds on the second attempt, the workflow continues seamlessly. The history records the failure and the success, ensuring the replay mechanism remains consistent.

Circuit Breakers

For external services that are consistently down, you can use circuit breakers to fail fast. You can implement circuit breaker patterns within your Activities, often using standard libraries, to prevent cascading failures.

Human-in-the-Loop

AI agents often require human approval for sensitive actions. Temporal supports signal and query mechanisms for human-in-the-loop workflows. You can pause the workflow and wait for a human to approve a decision:

approval = await workflow.wait_condition(lambda: self.approved)
if approval:
    await workflow.execute_activity(send_email, args=[draft])

The workflow remains in a "waiting" state until the human approves. If the server crashes during this wait, the workflow is still paused, and the history reflects that it is awaiting approval.

Conclusion

Durable execution is not just a fault-tolerance feature; it is a fundamental shift in how we build AI agents. By decoupling workflow logic from infrastructure stability, Temporal allows developers to focus on the agent’s logic rather than the plumbing of state management.

The key takeaway is determinism through replay. Every AI agent should be designed as a deterministic workflow, with non-deterministic operations encapsulated in Activities. This ensures that your agents are resilient, auditable, and capable of recovering from any failure without data loss or inconsistent state.

For backend developers and architects, this means moving away from custom state machines and manual checkpointing. Instead, leverage Temporal’s durable execution model to build agents that are truly resilient by design.