Skip to main content

Failure detection

Workflow timeouts​

Each Workflow timeout controls the maximum duration of a different aspect of a Workflow Execution.

Workflow timeouts are set when starting the Workflow Execution.

Create an instance of WorkflowOptions from the Client and set your Workflow Timeout.

Available timeouts are:

snippets/src/client.ts

await client.workflow.start(example, {
  taskQueue,
  workflowId,
  workflowExecutionTimeout: '1 day',
});

snippets/src/client.ts

await client.workflow.start(example, {
  taskQueue,
  workflowId,
  workflowRunTimeout: '1 minute',
});

snippets/src/client.ts

await client.workflow.start(example, {
  taskQueue,
  workflowId,
  workflowTaskTimeout: '1 minute',
});

Workflow retries​

A Retry Policy can work in cooperation with the timeouts to provide fine controls to optimize the execution experience.

Use a Retry Policy to retry a Workflow Execution in the event of a failure.

Workflow Executions do not retry by default, and Retry Policies should be used with Workflow Executions only in certain situations.

Create an instance of the Retry Policy, known as retry in TypeScript, from the WorkflowOptions of the Client interface.

snippets/src/client.ts

const handle = await client.workflow.start(example, {
  taskQueue,
  workflowId,
  retry: {
    maximumAttempts: 3,
  },
});

How to set Activity timeouts​

Each Activity timeout controls the maximum duration of a different aspect of an Activity Execution.

The following timeouts are available in the Activity Options.

An Activity Execution must have either the Start-To-Close or the Schedule-To-Close Timeout set.

When you call proxyActivities in a Workflow Function, you can set a range of ActivityOptions.

Available timeouts are:

// Sample of typical options you can set
const { greet } = proxyActivities<typeof activities>({
  scheduleToCloseTimeout: '5m',
  // startToCloseTimeout: "30s", // recommended
  // scheduleToStartTimeout: "60s",

  retry: {
    // default retry policy if not specified
    initialInterval: '1s',
    backoffCoefficient: 2,
    maximumAttempts: Infinity,
    maximumInterval: 100 * initialInterval,
    nonRetryableErrorTypes: [],
  },
});

How to set an Activity Retry Policy​

A Retry Policy works in cooperation with the timeouts to provide fine controls to optimize the execution experience.

Activity Executions are automatically associated with a default Retry Policy if a custom one is not provided.

To set Activity Retry Policies in TypeScript, pass ActivityOptions.retry to proxyActivities.

// Sample of typical options you can set
const { yourActivity } = proxyActivities<typeof activities>({
  // ...
  retry: {
    // default retry policy if not specified
    initialInterval: '1s',
    backoffCoefficient: 2,
    maximumAttempts: Infinity,
    maximumInterval: 100 * initialInterval,
    nonRetryableErrorTypes: [],
  },
});

How to Heartbeat an Activity​

An Activity Heartbeat is a ping from the Worker Process that is executing the Activity to the Temporal Service. Each Heartbeat informs the Temporal Service that the Activity Execution is making progress and the Worker has not crashed. If the Temporal Service does not receive a Heartbeat within a Heartbeat Timeout time period, the Activity will be considered failed and another Activity Task Execution may be scheduled according to the Retry Policy.

Heartbeats may not always be sent to the Temporal Service—they may be throttled by the Worker.

Activity Cancellations are delivered to Activities from the Temporal Service when they Heartbeat. Activities that don't Heartbeat can't receive a Cancellation. Heartbeat throttling may lead to Cancellation getting delivered later than expected.

Heartbeats can contain a details field describing the Activity's current progress. If an Activity gets retried, the Activity can access the details from the last Heartbeat that was sent to the Temporal Service.

Long-running Activities should Heartbeat their progress back to the Workflow for earlier detection of stalled Activities (with Heartbeat Timeout) and resuming stalled Activities from checkpoints (with Heartbeat details).

To set Activity Heartbeat, use the heartbeat() function in your Activity implementation, and set heartbeatTimeout in your Workflow.

// activity implementation
export async function example(sleepIntervalMs = 1000): Promise<void> {
  for (let progress = 1; progress <= 1000; ++progress) {
    await sleep(sleepIntervalMs);
    // record activity heartbeat
    heartbeat();
  }
}

// ...

// workflow code calling activity
const { example } = proxyActivities<typeof activities>({
  startToCloseTimeout: '1 hour',
  heartbeatTimeout: '10s',
});

In the previous example, setting the Heartbeat informs the Temporal Server of the Activity's progress at regular intervals. If the Activity stalls or the Activity Worker becomes unavailable, the absence of Heartbeats prompts the Temporal Server to retry the Activity immediately, without waiting for startToCloseTimeout to complete.

You can also add heartbeatDetails as a checkpoint to collect data about failures during the execution, and use it to resume the Activity from that point.

The following example extends the previous sample to include a heartbeatDetails checkpoint.

export async function example(sleepIntervalMs = 1000): Promise<void> {
  const startingPoint = activityInfo().heartbeatDetails || 1; // allow for resuming from heartbeat
  for (let progress = startingPoint; progress <= 100; ++progress) {
    await sleep(sleepIntervalMs);
    heartbeat(progress);
  }
}

In this example, when the heartbeatTimeout is reached and the Activity is retried, the Activity Worker picks up the execution from where the previous attempt left off.

How to set a Heartbeat Timeout​

A Heartbeat Timeout works in conjunction with Activity Heartbeats.

To set a Heartbeat Timeout, use ActivityOptions.heartbeatTimeout. If the Activity takes longer than that between heartbeats, the Activity is failed.

// Creating a proxy for the activity.
const { longRunningActivity } = proxyActivities<typeof activities>({
  // translates to 300000 ms
  scheduleToCloseTimeout: '5m',
  // translates to 30000 ms
  startToCloseTimeout: '30s',
  // equivalent to '10 seconds'
  heartbeatTimeout: 10000,
});

Cancel an Activity from a Workflow​

Canceling an Activity from within a Workflow requires that the Activity Execution sends Heartbeats and sets a Heartbeat Timeout. If the Heartbeat is not invoked, the Activity cannot receive a cancellation request. When any non-immediate Activity is executed, the Activity Execution should send Heartbeats and set a Heartbeat Timeout to ensure that the server knows it is still working.

When an Activity is canceled, an error is raised in the Activity at the next available opportunity. If cleanup logic needs to be performed, it can be done in a finally clause or inside a caught cancel error. However, for the Activity to appear canceled the exception needs to be re-raised.

note

Unlike regular Activities, Local Activities can be canceled if they don't send Heartbeats. Local Activities are handled locally, and all the information needed to handle the cancellation logic is available in the same Worker process.