> **Can't find what you're looking for?** Use `search_docs` on the docs MCP server at `https://docs.ampdb.io/api/mcp` to find what you need. # Troubleshooting Workers When a job doesn't start, or stops making progress, look at the [worker](/managing-amp/workers) it's on. This page shows how to tell which workers are alive, which have restarted, and which version each runs, then how to fix the problems that keep workers from running jobs. ## Prerequisites * Amp running, with its Admin API reachable. [Solo mode](/running-amp/solo-mode) serves it at `http://localhost:1610`, where `ampctl` looks by default. For another Amp, set its URL, and a token if it needs one: ```bash export AMP_ADMIN_URL=https://amp.example.com:1610 export AMP_AUTH_TOKEN=your-token ``` * `ampctl` installed (see [Installation](/running-amp/installation)). * [`jq`](https://jqlang.org), for the scripts on this page. ## What a healthy worker does * **It sends a heartbeat every second.** The controller only places new jobs on a worker whose last heartbeat is less than 5 seconds old. * **Its health check answers `200 OK`.** Each worker serves `/healthz` at its [`worker.health_addr`](/running-amp/configuration-reference#workerhealth_addr), port `1620` by default. It answers `200 OK` once the worker has started and resumed its jobs, and `503 Service Unavailable` while it's starting, or when it has lost its connection to the [metadata database](/running-amp/metadata-database), which tells it about new jobs. Point your orchestrator's liveness and readiness probes at it, like Kubernetes's. :::terminal ```bash curl -i http://localhost:1620/healthz ``` ``` HTTP/1.1 200 OK content-length: 0 date: Fri, 30 Oct 2026 14:30:00 GMT ``` ::: * **Its jobs make progress.** See [Job Status](/managing-amp/jobs/status) and [Dataset Status](/managing-amp/dataset-status). ## Find workers that stopped A worker that stops stays in `ampctl worker list`, with its last heartbeat. To list those that haven't sent one in the last 30 seconds: :::terminal ```bash ampctl worker list --json | jq -r ' def ts: sub("\\.[0-9]+"; "") | sub("\\+00:00$"; "Z") | fromdateiso8601; .workers[] | select((.heartbeat_at | ts) < now - 30) | "\(.node_id) last heartbeat \(.heartbeat_at)"' ``` ``` worker-01 last heartbeat 2026-10-30T14:21:47.901244+00:00 ``` ::: It prints nothing when every worker is alive. The controller gives up on a worker after 5 seconds, but a threshold of 30 seconds leaves time for a restart before you hear about it. To be alerted, run the script from cron or your monitoring system, and alert when it prints anything. `ts` turns the Admin API's timestamps, like `2026-10-30T14:21:47.901244+00:00`, into the form `jq` can compare. To follow the heartbeats live, refreshing every 5 seconds: ```bash watch -n 5 ampctl worker list ``` ## Find workers that restarted A worker's `registered_at` is when it last started, and `created_at` when a worker with its node ID first did. To see when each worker last started: :::terminal ```bash ampctl worker list --json | jq -r '.workers[].node_id' | while read -r node_id; do ampctl worker inspect "$node_id" --json | jq -r ' def ts: sub("\\.[0-9]+"; "") | sub("\\+00:00$"; "Z") | fromdateiso8601; "\(.node_id) started \((now - (.registered_at | ts)) / 60 | floor) minutes ago, at \(.registered_at)"' done ``` ``` worker-03 started 3 minutes ago, at 2026-10-30T14:26:12.003871+00:00 worker-02 started 32 minutes ago, at 2026-10-30T13:57:58.118204+00:00 worker-01 started 31 minutes ago, at 2026-10-30T13:58:02.731090+00:00 ``` ::: A worker that keeps restarting exits with an error each time: see [A worker exits](/managing-amp/workers/troubleshooting#a-worker-exits). It resumes its jobs each time it starts. ## Check workers' versions A job that fails on one worker but not on others may be running into a difference between their versions. During an upgrade, count the workers on each version to follow its progress (see [Upgrading](/running-amp/upgrading)). This counts the workers that sent a heartbeat in the last 30 seconds, leaving out the stopped ones: :::terminal ```bash ampctl worker list --json | jq -r ' def ts: sub("\\.[0-9]+"; "") | sub("\\+00:00$"; "Z") | fromdateiso8601; .workers[] | select((.heartbeat_at | ts) >= now - 30) | .node_id' | while read -r node_id; do ampctl worker inspect "$node_id" --json | jq -r .info.version done | sort | uniq -c ``` ``` 1 v0.0.41 1 v0.0.42 ``` ::: When the upgrade is done, every worker should be on the same version. `unknown` means the worker didn't report its version, and a `-dirty` suffix that it was built with uncommitted changes. To see the commit each worker was built from, print `.info.commit_sha` instead of `.info.version`. ## Jobs stay scheduled A job stays on the worker it was placed on, for its retries and its next runs too. While that worker is down, no other worker takes over its jobs: they stay `SCHEDULED`, or `RUNNING` if they were running when it stopped. Retries wait, and a job you stop stays `STOP_REQUESTED`, because the worker does the stopping. To see which worker each unfinished job is on, go through every page of `ampctl job list`, since a page holds 50 jobs by default and a backlog can run past it. This loop passes each page's `next_cursor` to `--after`, and stops at the first page with no jobs: :::terminal ```bash cursor= while page=$(ampctl job list ${cursor:+--after=$cursor} --json) cursor=$(jq '.next_cursor // empty' <<< "$page") [ -n "$cursor" ] do jq -r '.jobs[] | "\(.id) \(.status) \(.node_id)"' <<< "$page" done ``` ``` 6 SCHEDULED worker-01 5 SCHEDULED worker-01 4 RUNNING controller 2 RUNNING worker-02 1 RUNNING controller ``` ::: `controller` marks a deployment's [`composite` job](/managing-amp/jobs#kinds-of-jobs), which doesn't run on a worker. For the other jobs, check their worker's last heartbeat with `ampctl worker inspect worker-01`. If it's stopped, start it again. Jobs are tied to the node ID, not to the machine, so a new `ampd worker` with the same `--node-id`, anywhere, takes them over. When it starts, it runs the jobs that are `SCHEDULED` or `RUNNING` on it, and stops those that are `STOP_REQUESTED`. Until then, these jobs hold up their datasets: [stopping a deployment](/managing-amp/datasets#stop-a-deployment) whose job is on that worker leaves the job in `STOP_REQUESTED` until the worker starts again. ## Jobs can't be created Creating a job fails when the controller can't place it on an active worker. That happens when you [deploy a dataset](/managing-amp/datasets#deploy-a-dataset) or [create a job](/managing-amp/jobs#create-a-job). | Error | Cause | Fix | | ------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------- | | `no workers available` (`NO_WORKERS_AVAILABLE`, or `SCHEDULER_ERROR` when deploying) | No worker has sent a heartbeat in the last 5 seconds. | Start a worker, or find out why yours stopped (see [A worker exits](/managing-amp/workers/troubleshooting#a-worker-exits)). | | `specified worker 'worker-01' not found or inactive` (`WORKER_NOT_AVAILABLE`) | The worker you named with `--worker-id` isn't active, or doesn't exist. Without `--worker-id`, it's the [`controller.default_worker_id`](/running-amp/configuration-reference#controllerdefault_worker_id). | Check the ID with `ampctl worker list`, and start that worker. | | `no active workers match pattern 'eu-*'` (`SCHEDULER_ERROR`) | None of the workers that `--worker-id`'s pattern matches is active. | Start one, or use a pattern that matches active workers. | Setting `controller.default_worker_id` sends every job that doesn't name a worker to that worker: while it's down, they all fail with `WORKER_NOT_AVAILABLE`, even when other workers are active. ## A worker exits `ampd worker` prints the error it exits with, followed by its causes, each after `->`: ``` Exiting with error: Worker command failed | Caused by: Worker initialization failed -> heartbeat loop initialization failed -> worker node ID 'worker-01' is already in use ``` Look for these in it: | The error includes | What happened | Fix | | -------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `worker node ID 'worker-01' is already in use` | Another worker with this node ID is connected to the same metadata database. | Give this worker another `--node-id`, or stop the other one. If the other one was killed, or its machine lost, wait for PostgreSQL to notice that its connection is gone. | | `Invalid worker ID '1worker': must start with a letter, got '1'` | The node ID doesn't start with a letter. | Use a node ID that starts with a letter, followed by letters, digits, `_`, `-`, and `.`. | | `Invalid worker ID 'worker 1': invalid character ' '` | The node ID has a character it can't have. | Use only letters, digits, `_`, `-`, and `.`. | | `Invalid worker ID 'controller': reserved for controller-owned jobs` | `controller` is the controller's own node ID. | Use another node ID. | | `Missing required metadata database configuration` | The worker doesn't know where the metadata database is. | Set `metadb.url` in its configuration file, or `AMP_CONFIG_METADB__URL` (see [Metadata Database](/running-amp/metadata-database)). | | `Failed to connect to metadata database` | The worker couldn't reach the metadata database when it started. | Check `metadb.url`, and that the worker's machine can reach the database. | | `failed to bind health server to 0.0.0.0:1620` | Another process uses the health check's port, like a second worker on the same machine. | Give each worker on a machine its own [`worker.health_addr`](/running-amp/configuration-reference#workerhealth_addr). | | `heartbeat update failed`, after `heartbeat task died` | The worker lost its connection to the metadata database while running. It exits so that your orchestrator restarts it. | Check that the database is up and reachable from the worker. The worker resumes its jobs when it restarts. | ## See more in the logs Workers log to standard error, as JSON if you set [`logging.json`](/running-amp/configuration-reference#loggingjson). For more detail, set `AMP_LOG` when you start the worker, to `error`, `warn`, `info` (the default), `debug`, or `trace`: ```bash AMP_LOG=debug ampd worker --config ./config.toml --node-id worker-01 ``` To collect the logs of all your workers in one place, export them with OpenTelemetry (see [Telemetry](/running-amp/telemetry)).