> **Can't find what you're looking for?** Use `search_docs` on the docs MCP server at `https://docs.ampdb.io/api/mcp` to find what you need.
# Retries
Extraction depends on things that fail now and then: a provider that times out, a database connection that drops.
When a [job](/managing-amp/jobs) fails with an error a retry may fix, it moves to `ERROR`, and Amp runs it again for
you. Its **retry strategy** decides how many times, and how long it waits in between. This page explains how retries
work, and how to choose a strategy.
Errors a retry can't fix, like a dataset that doesn't exist anymore, move the job straight to `FATAL`, whatever its
strategy. See [Job Lifecycle](/managing-amp/jobs/lifecycle#failures).
## How retries work
The controller looks for jobs in `ERROR` once a minute. For each, it reads its retry strategy, then:
1. If the job has failed too often, as the strategy defines it, the controller moves it to `FATAL`.
2. Otherwise, if the job has waited in `ERROR` for its **backoff** delay, the controller schedules it again, on the
same worker. The job moves to `SCHEDULED`, then `RUNNING`.
3. Otherwise, it waits for the next check.
Since the checks are a minute apart, a retry starts up to a minute after its backoff has passed.
## Strategies
| Strategy | Retries |
| -------------------------- | ------------------------------------------------------------------------------------------ |
| `none` | Never: the job moves to `FATAL` on its first failure. |
| `bounded` | Until the job has failed `max_attempts` times within the last `window_secs`. Then `FATAL`. |
| `unless_stopped` (default) | Forever, until you stop the job. |
Unless you choose one, a job retries forever, with an exponential backoff: 1 second after its first failure, then 2,
4, 8, and so on, up to an hour.
`bounded` suits jobs that should fail loudly when something stays broken, so that your monitoring catches it.
`unless_stopped` suits jobs that should ride out outages on their own, like a deployment that keeps up with a chain.
## The failure window
A `bounded` strategy counts failures over a sliding window, `window_secs` long: an hour by default. Failures older than
the window stop counting. A job with `max_attempts` 5 therefore moves to `FATAL` once it has failed 5 times within an
hour, not after its fifth failure ever. A job that fails now and then, but less often than that, keeps running.
The window is at most 7 days (604,800 seconds), the history Amp reads failures from.
The `Attempt Index` a failed job's [status](/managing-amp/jobs/status#errors) shows isn't this count: it's how many
times the job has been scheduled, counting from 1, whenever it happened.
## Backoff
The backoff is how long a failed job waits before its retry. It grows with the failures in the window, which is the
`bounded` strategy's window, or an hour for `unless_stopped`:
| Backoff | Delay after the window's *n*-th failure |
| ------------------------- | --------------------------------------------------------------------------- |
| `fixed` | `base_delay_secs`, every time. |
| `exponential` (default) | `base_delay_secs` × `multiplier`*n* − 1, up to `max_delay_secs`. |
| `exponential_with_jitter` | Like `exponential`, plus a random extra of up to a quarter of the delay. |
With the defaults, a base delay of 1 second and a multiplier of 2, the delays are 1, 2, 4, 8, and 16 seconds after the
first five failures. Jitter spreads out the retries of jobs that failed at the same time, like every job reading from
a provider that went down, so that they don't all hit it again at once.
### Uptime shortens the backoff
A job that ran for a while before failing again gets some of its backoff back. Each step of the delay is paid back by
the time the job ran, starting from the largest: a job that ran for at least as long as the delay it owes moves down a
step, and so on. So a job that crashes as it starts backs off quickly to long delays, while one that fails after
hours of work retries almost at once.
For example, with the default backoff, a job that has failed 4 times in the window owes 8 seconds. If its last run
lasted 10 seconds, that pays back the 8, leaving 2, which isn't enough for the next step down, 4: the job waits
4 seconds. Had it run for 15 seconds, it would have paid back 8, 4, and 2, and waited 1 second.
Uptime only shortens the backoff. It never takes failures out of the window, so a `bounded` job that keeps failing,
however long it runs each time, still moves to `FATAL`. A job that fails before it starts running gets no uptime
credit.
## Choose a strategy
### With `ampctl`
The examples use the `MANIFEST_HASH` variable and `warehouse` provider of [Jobs](/managing-amp/jobs#prerequisites).
[`ampctl job create`](/managing-amp/jobs#create-a-job) and
[`ampctl dataset deploy`](/managing-amp/datasets#deploy-options) take these options:
| Option | Type | Default | Description |
| ------------------------------------------------- | ---------------------------------------------------- | ------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------- |
| `--retry-strategy ` | `none`, `bounded`, or `unless_stopped` | `unless_stopped`, with the default backoff | The strategy, like `--retry-strategy bounded`. The other options need it. |
| `--retry-max-attempts ` | integer, at least 1 | none | How many failures within the window make the job `FATAL`, like `--retry-max-attempts 5`. Required with `bounded`, and only allowed with it. |
| `--retry-window-secs ` | integer, 1 to 604800 | `3600` | How far back failures count, in seconds, like `--retry-window-secs 86400`. Only allowed with `bounded`. |
| `--retry-backoff ` | `fixed`, `exponential`, or `exponential_with_jitter` | `exponential` | How the delay grows, like `--retry-backoff exponential_with_jitter`. Not allowed with `none`. |
| `--retry-base-delay-secs ` | integer, at least 1 | `1` | The first delay, or every delay with `fixed`, in seconds, like `--retry-base-delay-secs 5`. Not allowed with `none`. |
| `--retry-multiplier ` | number, greater than 0 | `2.0` | What each delay is multiplied by, like `--retry-multiplier 1.5`. Not allowed with `none`. |
| `--retry-max-delay-secs ` | integer, at least `--retry-base-delay-secs` | none: the delay has no cap | The longest delay, in seconds, like `--retry-max-delay-secs 300`. Not allowed with `none`. |
When you choose a strategy, set `--retry-max-delay-secs` too: unlike the default strategy's, the delay of a strategy
you choose has no cap unless you give it one.
Fail permanently once a job has failed 5 times within an hour:
```bash
ampctl job create adbc acme eth_mainnet "$MANIFEST_HASH" --provider warehouse \
--retry-strategy bounded \
--retry-max-attempts 5 \
--retry-max-delay-secs 60
```
The same budget, counted over a day, so that brief outages never use it up:
```bash
ampctl job create adbc acme eth_mainnet "$MANIFEST_HASH" --provider warehouse \
--retry-strategy bounded \
--retry-max-attempts 5 \
--retry-window-secs 86400 \
--retry-max-delay-secs 600
```
Retry forever, waiting from 2 seconds up to 5 minutes, with jitter:
```bash
ampctl dataset deploy acme/eth_mainnet@1.0.0 \
--retry-strategy unless_stopped \
--retry-backoff exponential_with_jitter \
--retry-base-delay-secs 2 \
--retry-max-delay-secs 300
```
Never retry:
```bash
ampctl job create adbc acme eth_mainnet "$MANIFEST_HASH" --provider warehouse --retry-strategy none
```
`ampctl` checks the options before it sends anything. An option the strategy doesn't allow fails with a message that
names it, like `--retry-max-attempts is only valid when --retry-strategy=bounded`.
### With the Admin API
Requests that create jobs take a `retry_strategy` object:
[`POST /jobs`](/managing-amp/admin-api/jobs#create-a-job) and
[`POST /materialized-datasets/…`](/managing-amp/admin-api/materialized-datasets#deploy-a-dataset-version).
```json
{
"retry_strategy": {
"strategy": "bounded",
"max_attempts": 5,
"window_secs": 3600,
"backoff": {
"kind": "exponential",
"base_delay_secs": 1,
"multiplier": 2.0,
"max_delay_secs": 60
}
}
}
```
| Field | Type | Default | Description |
| ------------------------- | ---------------------------------------------------------- | --------------------------------------------- | ------------------------------------------------------------------------- |
| `strategy` | `"none"`, `"bounded"`, or `"unless_stopped"` | required | The strategy. |
| `max_attempts` | integer, at least 1 | required with `bounded` | How many failures within the window make the job `FATAL`. `bounded` only. |
| `window_secs` | integer, 1 to 604800 | `3600` | How far back failures count, in seconds. `bounded` only. |
| `backoff` | object | exponential, 1 second, ×2, up to 3600 seconds | How the delay grows. `bounded` and `unless_stopped` only. |
| `backoff.kind` | `"fixed"`, `"exponential"`, or `"exponential_with_jitter"` | required | The backoff. |
| `backoff.base_delay_secs` | integer, at least 1 | `1` | The first delay, or every delay with `fixed`, in seconds. |
| `backoff.multiplier` | number, greater than 0 | `2.0` | What each delay is multiplied by. Exponential backoffs only. |
| `backoff.max_delay_secs` | integer, at least `base_delay_secs` | none: no cap | The longest delay, in seconds. Exponential backoffs only. |
Two more shapes:
```json
{ "retry_strategy": { "strategy": "none" } }
```
```json
{ "retry_strategy": { "strategy": "unless_stopped", "backoff": { "kind": "fixed", "base_delay_secs": 30 } } }
```
A strategy the API can't read fails with `400 Bad Request` and `INVALID_BODY`.
A job's strategy is part of its descriptor, which [`ampctl job inspect`](/managing-amp/jobs#inspect-a-job) shows. To
change it, stop the job and create it again.
## Retries and triggers
A job that runs on a [trigger](/managing-amp/jobs/triggers) keeps its failures across runs: they count until they
leave the window, not until the next run. So a job that fails on every run eventually moves to `FATAL`, and its next
run starts with those failures still counted. See [Triggers](/managing-amp/jobs/triggers#triggers-and-retries).
## Limitations
* Uptime stands in for progress, which Amp doesn't measure. A job stuck on a provider that never answers looks
healthy until it fails, and one that does real work in a few seconds looks like it's crashing.
* Time a job spends in `SCHEDULED`, waiting for its worker, earns no uptime credit.
* A retry runs on the job's worker. While that worker is down, the job waits for it. See
[Troubleshooting Workers](/managing-amp/workers/troubleshooting#jobs-stay-scheduled).