Workflow Durability and Reliability

Why Durability Changes Everything

Imagine a ten-step pipeline processing a batch of invoices: extraction, validation, model call, database write, notification. In a standard Python script, a crash at step 7 forces you back to step 1. The first six steps run again, creating serious issues if any of them wrote to a database or sent a message. With a durable workflow, the system resumes exactly at step 7. It skips the first six steps, avoids duplicating actions, and loses no data.

This capability relies on a specific mechanism called event sourcing. Understanding this concept is essential because it explains the constraints you will encounter later.

Event Sourcing: The Core Principle

Event sourcing records every workflow action as an immutable event in a log. The difference from a traditional approach comes down to one sentence: instead of storing only the current state (e.g., “in progress, step 7”), the platform keeps a complete history of operations and their results. A workflow execution with two activities leaves the following trace:

1. The workflow starts and receives input parameters → event recorded
2. The workflow calls Activity A → event recorded
3. Activity A completes with a result → event recorded
4. The workflow calls Activity B → event recorded
5. Activity B completes → event recorded
6. The workflow returns its final result → event recorded

The system persists each step before moving to the next. There is no window where an action occurs without the log knowing about it. This log becomes the single source of truth; the workflow state exists nowhere else.

Automatic Replay: Recovery from Failure

When a worker crashes, a new worker takes over, retrieves the event log, and replays the workflow code from the beginning. The word “replay” often surprises people: the code does restart from line one, but for every activity that already finished, the engine does not call your function again. Instead, it substitutes the recorded result from the log. The workflow advances at read speed until it hits the first unfinished activity. Only then does real work resume.

Activities that already succeeded are never re-executed during a replay. The example below makes this guarantee tangible for the most sensitive case: sending an email.

import mistralai.workflows as workflows

@workflows.activity()
async def send_email(recipient: str, content: str) -> dict:
    """This activity runs only once,
    even if the workflow replays multiple times."""
    response = await email_service.send(recipient, content)
    return {"status": "sent", "id": response.id}

@workflows.workflow.define(name="notification_workflow")
class NotificationWorkflow:
    @workflows.workflow.entrypoint
    async def run(self, recipient: str) -> dict:
        # If the workflow crashes AFTER sending the email,
        # the email will NOT be resent during replay.
        email_result = await send_email(recipient, "Your report is ready")

        # The workflow resumes here
        log_result = await log_notification(recipient, email_result)
        return log_result

Fault Tolerance: Covered Scenarios

The platform handles four types of incidents without requiring a single line of code from you. First, a worker crash: the Python process stops abruptly due to a memory error, a kill signal, or a server reboot. Another worker picks up the workflow exactly where it left off. Second, a transient network error: an API call fails due to a timeout. The retry policy automatically restarts the activity with exponential backoff, using increasing delays to avoid hammering a struggling service.

Third, errors in your own code: an exception raised in an activity retries a defined number of times based on the configured policy before failing permanently. Fourth, temporary unavailability of a third-party service. The effect here is dramatic: if a partner API goes down for ten minutes, your nightly pipeline no longer fails. Pending activities restart when the service returns, and processing finishes just ten minutes late.

The Determinism Constraint

Replay comes with a cost. For the log replay to faithfully reproduce the original execution, the workflow code (not the activities) must be deterministic. Given the same inputs, it must produce the same sequence of operations. If your workflow branches based on if datetime.now().hour < 12, a replay tonight might take the other branch, completely desynchronizing the log.

Therefore, you must never do the following in workflow code:

• Use datetime.now() → use workflow.now()
• Use uuid.uuid4() → use workflow.uuid4()
• Use random.random() → use workflow.random()
• Make network or I/O calls directly → move them into an activity

These three substitutions return values recorded during the first pass and replayed identically afterward. We will detail this constraint in Lesson 8, which focuses on determinism.

Comparison with Traditional Approaches

Pay close attention to the complexity column. A message queue seems simple to start, but its partial recovery forces you to write state management logic that the SDK provides out of the box.

Exactly-Once: A Logical Guarantee

The platform guarantees that your workflow logic executes exactly once. If an activity succeeds, the system records the result and never re-executes it. If it fails, the system retries it according to the configured policy.

The word “logical” is not decorative. At the network level, a call might actually send twice: your request to create a customer reaches the CRM, but the acknowledgment gets lost in transit. From your perspective, the activity failed and will retry, even though the customer already exists. This creates a need for idempotent activities, which produce the same result even if called multiple times with the same parameters. In practice, this means preferring a PUT on a stable ID over a POST that creates a new row with every call.

Key Takeaways

• Event sourcing records every step as an immutable event in a log
• Automatic replay lets you resume an interrupted workflow without re-executing finished activities
• The platform automatically handles crashes, network errors, and service outages
• Workflow code must be deterministic for replay to work
• Activities must be idempotent to ensure consistency during retries
• The execution guarantee is logical exactly-once