> **Can't find what you're looking for?** Use `search_docs` on the docs MCP server at `https://docs.ampdb.io/api/mcp` to find what you need.

# Distributed Mode

In distributed mode, each of Amp's [services](/running-amp/ampd#services) runs as its own `ampd` process: one
controller, and as many query servers and workers as you need. It's how you run Amp in production:

* **Scale each part on its own**: add query servers for more queries, and workers for more datasets.
* **Isolate failures**: a worker that runs out of memory doesn't take the query servers down with it.
* **Spread the load**: run the services on different machines, closer to their clients or data sources.

## How the services fit together

The services don't call each other. They share two things, which every one of them must reach:

* **The [metadata database](/running-amp/metadata-database)**, a PostgreSQL database with Amp's state. The controller
  assigns a job to a worker by recording it there, and the database notifies the worker.
* **[Storage](/running-amp/storage)**, where workers write datasets' Parquet files and query servers read them.

```
clients ───▶ query servers ──────┐
                                 ├──▶ metadata database (PostgreSQL)
operators ──▶ controller ────────┤
                                 └──▶ storage (S3, Google Cloud Storage, Azure, or a disk)
             workers ────────────────▶ both, and the chains' providers
```

Every service reads the same [configuration file](/running-amp/configuration), or at least the same database URL and
storage locations. Each one also reads its own section of it, like `[server]` for query servers.

## Prerequisites

* `ampd` on every machine that runs a service, from [Installation](/running-amp/installation), or Amp's
  [Docker image](/running-amp/installation#run-with-docker).
* A PostgreSQL 16 or newer database that every service can reach. Amp
  [connects to it without TLS](/running-amp/metadata-database#limitations), so keep it on a private network.
* Storage that every service can reach: a bucket in an object store, or directories on a disk, if every service runs on
  the same machine or mounts the same volume. See [Storage](/running-amp/storage).

## Deploy

This deploys every service on one machine, with a PostgreSQL server on `localhost` and storage in local directories. To
spread the services across machines, use the same configuration everywhere, with the database's address and an object
store instead.

::::steps
### Write the configuration

Create a directory for Amp, with the storage directories in it. Outside solo mode, `ampd` doesn't create them:

```bash
mkdir -p amp/data amp/providers amp/manifests
cd amp
```

Then write the configuration file. With `auto_migrate = false`, services check that the database is up to date when
they start, and leave its migrations to the next step:

```toml [config.toml]
data_dir = "data"
providers_dir = "providers"
manifests_dir = "manifests"

[metadb]
url = "postgresql://amp:amp@localhost:5432/amp"
auto_migrate = false
```

Set `AMP_CONFIG` so every command finds it:

```bash
export AMP_CONFIG=$PWD/config.toml
```

### Migrate the metadata database

`ampd migrate` creates Amp's tables, then exits:

```bash
ampd migrate
```

### Start the controller

In its own terminal, or as a service of your process manager:

```bash
ampd controller
```

### Start a query server

```bash
ampd server
```

### Start a worker

Give each worker a unique [ID](/running-amp/worker#node-ids). On one machine, give each its own health check port too,
since they'd all use `1620` otherwise:

```bash
ampd worker --node-id worker-01
```

```bash
AMP_CONFIG_WORKER__HEALTH_ADDR=0.0.0.0:1621 ampd worker --node-id worker-02
```

### Check the deployment

The controller lists the workers that are running:

:::terminal
```bash
ampctl worker list
```

```
Workers:
  worker-02 (last heartbeat: 2026-10-30T17:04:46.477839+00:00)
  worker-01 (last heartbeat: 2026-10-30T17:04:46.278755+00:00)
```
:::

Each service also answers its [health check](/running-amp/ampd#health-checks): `curl http://localhost:1602/healthz`
for the query server.
::::

Amp is ready for datasets. When `ampctl` runs on another machine than the controller, give it the Admin API's address
with `--admin-url` or `AMP_ADMIN_URL`, like `AMP_ADMIN_URL=http://amp-controller.internal:1610`.

## Run with Docker Compose

This Compose file runs the same deployment in containers: PostgreSQL, a container that migrates the database and
exits, a controller, a query server, and two workers. The services share their storage through a volume, which works
on a single Docker host. Across hosts, use an object store.

```toml [config.toml]
data_dir = "/var/lib/amp/data"
providers_dir = "/var/lib/amp/providers"
manifests_dir = "/var/lib/amp/manifests"

[metadb]
url = "postgresql://amp:amp@metadb:5432/amp"
auto_migrate = false

# Workers write the most, so give them more database connections.
[worker.metadb]
pool_size = 20
```

```yaml [compose.yaml]
x-amp: &amp
  image: ghcr.io/edgeandnode/amp:v0.0.42
  environment:
    AMP_CONFIG: /etc/amp/config.toml
  volumes:
    - ./config.toml:/etc/amp/config.toml:ro
    - amp-storage:/var/lib/amp
  restart: unless-stopped

services:
  metadb:
    image: postgres:16
    environment:
      POSTGRES_USER: amp
      POSTGRES_PASSWORD: amp
      POSTGRES_DB: amp
    volumes:
      - metadb:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD", "pg_isready", "-U", "amp"]
      interval: 2s
      retries: 30

  # Creates the storage directories and migrates the metadata database, then exits.
  migrate:
    <<: *amp
    entrypoint:
      ["sh", "-c", "mkdir -p /var/lib/amp/data /var/lib/amp/providers /var/lib/amp/manifests && exec /ampd migrate"]
    restart: "no"
    depends_on:
      metadb:
        condition: service_healthy

  controller:
    <<: *amp
    command: ["controller"]
    ports:
      - "1610:1610"
    depends_on:
      migrate:
        condition: service_completed_successfully

  server:
    <<: *amp
    command: ["server"]
    ports:
      - "1602:1602"
    depends_on:
      migrate:
        condition: service_completed_successfully

  worker-01:
    <<: *amp
    command: ["worker", "--node-id", "worker-01"]
    depends_on:
      migrate:
        condition: service_completed_successfully

  worker-02:
    <<: *amp
    command: ["worker", "--node-id", "worker-02"]
    depends_on:
      migrate:
        condition: service_completed_successfully

volumes:
  metadb:
  amp-storage:
```

Start it, and check its workers once it's up:

:::terminal
```bash
docker compose up -d
curl http://localhost:1610/workers
```

```json
{
  "workers": [
    { "node_id": "worker-02", "heartbeat_at": "2026-10-30T17:04:43.473520+00:00" },
    { "node_id": "worker-01", "heartbeat_at": "2026-10-30T17:04:43.275037+00:00" }
  ]
}
```
:::

To upgrade, change the image's tag and run `docker compose up -d` again: the `migrate` container migrates the database
before the services restart. See [Upgrading](/running-amp/upgrading).

## Scale

* **Query servers** keep no state of their own, so you can run as many as you need, behind a load balancer. Queries are
  gRPC calls, and streaming queries stay open for as long as the client reads them: use a load balancer that supports
  HTTP/2 and long-lived connections.
* **Workers** each run the jobs assigned to them. A new worker takes new jobs, but jobs that already run on another
  worker stay there, retries and periodic runs included. To choose where a job runs, see
  [Workers](/managing-amp/workers) and [`controller.default_worker_id`](/running-amp/controller#job-assignment).
* **The controller**: run one. It's only needed to manage Amp: while it's down, query servers keep answering queries and
  workers keep running the jobs they have, but the Admin API is unavailable and no job starts, periodic runs included.

Each service has its own [section of the configuration](/running-amp/configuration#per-service-settings), so you can
tune them apart, like a larger memory limit for query servers or more database connections for workers.

## Secure the deployment

`ampd` doesn't authenticate the requests it receives or encrypt its connections. Put that in front of it:

* **Keep the controller private.** Anyone who reaches the Admin API can manage Amp. Expose it only to operators, on a
  private network or through an authenticating proxy. When that proxy sends `amp-capabilities` metadata, the controller
  limits each request to the namespaces it allows: see [Admin API](/managing-amp/admin-api).
* **Expose query servers through a gateway** that authenticates clients, terminates TLS, limits their rate, and balances
  the load. To give each user a resource budget, see [Limits](/querying/limits).
* **Keep workers, the metadata database, and storage on private networks.** Workers only need to reach out: to the
  database, storage, and the chains' [providers](/providers).