> **Can't find what you're looking for?** Use `search_docs` on the docs MCP server at `https://docs.ampdb.io/api/mcp` to find what you need. # Job Lifecycle Every [job](/managing-amp/jobs) has a **status**, which says where it is in its life: waiting for a worker, running, finished, failed, or stopping. `ampctl job list`, `ampctl job inspect`, and the Admin API report it, and `ampctl job events` lists every status a job went through. This page explains each status, and what moves a job from one to the next. ## Statuses | Status | Finished | Meaning | | ---------------- | :------: | ---------------------------------------------------------------------------------------------------------------------------------- | | `SCHEDULED` | no | The job is waiting for its worker to start it. | | `RUNNING` | no | Its worker is running it. | | `COMPLETED` | yes | It finished its work, like a materialization that reached its end block. | | `ERROR` | no | It failed with an error a retry may fix, like a provider that didn't answer. It waits to be [retried](/managing-amp/jobs/retries). | | `FATAL` | yes | It failed for good: with an error a retry can't fix, or after spending its retries. | | `STOP_REQUESTED` | no | You asked to stop it, and its worker hasn't acted on it yet. | | `STOPPING` | no | Its worker is stopping it. | | `STOPPED` | yes | Its worker stopped it. | | `CANCELLED` | no | A job that runs on a [trigger](/managing-amp/jobs/triggers) gave up waiting for a retry, because its next run was due. | | `UNKNOWN` | no | A status this version of Amp doesn't know, which can happen while you [upgrade](/running-amp/upgrading) Amp. It shouldn't last. | A **finished** job is one that won't run again on its own. You can remove it with `ampctl job rm`, and creating a job with the same idempotency key schedules it again (see [IDs and idempotency keys](/managing-amp/jobs#ids-and-idempotency-keys)). The other statuses are **active**: `ampctl job list` shows them by default. A job that runs on a trigger is the exception: between runs it's `COMPLETED` or `FATAL`, and its trigger schedules it again when its next run is due. It only finishes for good when you stop it. ## What changes a job's status | From | To | When | | -------------------------------------------------- | ---------------- | ------------------------------------------------------------------------------------------------------------------------- | | (none) | `SCHEDULED` | The job is created: by deploying a dataset, or with `ampctl job create`. | | `SCHEDULED` | `RUNNING` | Its worker starts it. | | `RUNNING` | `COMPLETED` | It finishes its work. | | `SCHEDULED` or `RUNNING` | `ERROR` | It fails with an error a retry may fix, before it starts running or while it runs. | | `RUNNING` | `FATAL` | It fails with an error a retry can't fix, like a dataset that doesn't exist. | | `ERROR` | `SCHEDULED` | The controller retries it: its [retry strategy](/managing-amp/jobs/retries) has retries left, and its backoff has passed. | | `ERROR` | `FATAL` | Its retry strategy has no retries left. | | `SCHEDULED`, `RUNNING`, or `ERROR` | `STOP_REQUESTED` | You stop it. | | `COMPLETED`, `FATAL`, or `CANCELLED`, between runs | `STOP_REQUESTED` | You stop a job that runs on a trigger. | | `STOP_REQUESTED` | `STOPPING` | Its worker starts stopping it. | | `STOPPING` | `STOPPED` | Its worker has stopped it. | | `COMPLETED`, `FATAL`, or `CANCELLED` | `SCHEDULED` | The trigger of a job that runs on one fires. | | `ERROR` | `CANCELLED` | The trigger fires while the job waits for a retry: the retry is dropped, and the job is scheduled at once. | | `COMPLETED`, `STOPPED`, or `FATAL` | `SCHEDULED` | You create a job with the same idempotency key. | Workers record what happens while they run a job. The controller records the rest: new jobs, stop requests, retries, and trigger runs. It looks for failed jobs to retry once a minute, so a retry can start up to a minute after its backoff has passed. ## Stopping Stopping goes through three statuses. [`ampctl job stop`](/managing-amp/jobs#stop-a-job) moves the job to `STOP_REQUESTED` and tells its worker. The worker moves it to `STOPPING` while it stops the job's work, then to `STOPPED`. A job that wasn't running, like one waiting for a retry, goes straight through. The worker does the stopping, so a job whose worker is down stays in `STOP_REQUESTED`. When the worker starts again, it stops every job it was asked to stop while it was down. `STOPPED` is final: nothing schedules a stopped job again, except creating it again. ## Failures A job that fails moves to `ERROR` or `FATAL`, depending on the error: * `ERROR` is for failures that may go away: a provider that times out, a connection that drops, or a provider that isn't registered yet. The controller schedules the job again on the same worker once its backoff has passed, as its [retry strategy](/managing-amp/jobs/retries) says. When the strategy has no retries left, the job moves to `FATAL`. * `FATAL` is for failures a retry won't fix, like a dataset that doesn't exist anymore. Either way, the job's status shows the error: see [Job Status](/managing-amp/jobs/status#errors). ## Deployments A [deployment's](/managing-amp/jobs#deployment-jobs) parent job doesn't run anything itself. The controller sets its status from its children's: * `RUNNING` while any child is active. * `FATAL` once every child has finished and at least one is `FATAL`. * `COMPLETED` once every child has finished and none is `FATAL`. A child you stop counts as neither, so the parent stays `RUNNING` until its other children finish. Stopping the parent, with [`ampctl dataset stop`](/managing-amp/datasets#stop-a-deployment), stops its children too.