> **Can't find what you're looking for?** Use `search_docs` on the docs MCP server at `https://docs.ampdb.io/api/mcp` to find what you need.

# Storage

Amp stores the tables of every dataset it extracts as [Parquet](https://parquet.apache.org) files, a compressed
format that stores each column apart, so queries only read the columns they need. They're Amp's data at rest: workers
write them, query servers read them, and you own them, on your disk or in your cloud's object store.

## Storage locations

Amp reads and writes three locations, set at the top of the [configuration](/running-amp/configuration):

| Setting                                                               | Default       | What Amp keeps there                                                                                           |
| --------------------------------------------------------------------- | ------------- | -------------------------------------------------------------------------------------------------------------- |
| [`data_dir`](/running-amp/configuration-reference#data_dir)           | `"data"`      | Datasets' Parquet files. Workers write them, and query servers read them.                                      |
| [`providers_dir`](/running-amp/configuration-reference#providers_dir) | `"providers"` | [Providers](/providers)' configuration, one TOML file each. See [Managing Providers](/managing-amp/providers). |
| [`manifests_dir`](/running-amp/configuration-reference#manifests_dir) | `"manifests"` | The [manifests](/datasets/manifests) of registered datasets. See [Manifest Storage](/managing-amp/manifests).  |

Each is a directory on disk, or an object store's URL:

| Storage                                       | Value                              |
| --------------------------------------------- | ---------------------------------- |
| [A local disk](/running-amp/storage#local-disk)                   | A path, like `"/var/lib/amp/data"` |
| [Amazon S3](/running-amp/storage#amazon-s3), or compatible        | `"s3://bucket/path"`               |
| [Google Cloud Storage](/running-amp/storage#google-cloud-storage) | `"gs://bucket/path"`               |
| [Azure Blob Storage](/running-amp/storage#azure-blob-storage)     | `"az://container/path"`            |

The three can be in different places, like the data in a bucket and the providers on disk. Every service must reach
the same ones. When services run on different machines, use an object store; it's also the better choice in
production, for its durability and size.

## Local disk

A path is relative to the configuration file's directory (see [Values](/running-amp/configuration#values) for a
relative `--config`). Without a configuration, solo mode uses `data`,
`providers`, and `manifests` in its [state directory](/running-amp/solo-mode#the-amp-directory), and creates them.

Other commands don't create directories: they must exist before a service starts, or it fails:

```
Exiting with error: Failed to load config | Caused by: Invalid object store URL at /etc/amp/config.toml -> invalid filesystem path: /etc/amp/data -> failed to canonicalize path `/etc/amp/data`: No such file or directory (os error 2)
```

Services share a disk only when they run on the same machine, or mount the same volume, like in the
[Docker Compose example](/running-amp/distributed-mode#run-with-docker-compose).

## Object storage

Amp authenticates to object stores with the standard environment variables of each cloud, which every service that
uses the store needs. The path after the bucket is optional, and lets several locations, or several Amp deployments,
share a bucket:

```toml [config.toml]
data_dir = "s3://acme-amp/production/data"
providers_dir = "s3://acme-amp/production/providers"
manifests_dir = "s3://acme-amp/production/manifests"
```

### Amazon S3

Use `s3://` URLs (`s3a://` works too). Set the credentials and region:

| Variable                             | Description                                                          |
| ------------------------------------ | -------------------------------------------------------------------- |
| `AWS_ACCESS_KEY_ID`                  | The access key's ID.                                                 |
| `AWS_SECRET_ACCESS_KEY`              | The access key's secret.                                             |
| `AWS_SESSION_TOKEN`                  | The session token, for temporary credentials.                        |
| `AWS_REGION` or `AWS_DEFAULT_REGION` | The bucket's region. `us-east-1` by default.                         |
| `AWS_ENDPOINT`                       | The endpoint of an S3-compatible store, like MinIO or Cloudflare R2. |
| `AWS_ALLOW_HTTP`                     | `true` to allow an endpoint without TLS, like a local MinIO.         |

Without an access key, Amp uses the credentials of its environment, like an EC2 instance's or an EKS pod's role. On a
machine that has none, requests fail after Amp gives up asking for them, with an error that includes
`Error performing PUT http://169.254.169.254/latest/api/token`.

For an S3-compatible store, set its endpoint:

```bash
export AWS_ENDPOINT=http://localhost:9000
export AWS_ALLOW_HTTP=true
export AWS_ACCESS_KEY_ID="$MINIO_ACCESS_KEY"
export AWS_SECRET_ACCESS_KEY="$MINIO_SECRET_KEY"
```

Without `AWS_ALLOW_HTTP`, requests to an `http://` endpoint fail with `HTTP error: builder error`.

### Google Cloud Storage

Use `gs://` URLs. Amp uses the first credentials it finds:

| Variable                         | Description                                                                                                                  |
| -------------------------------- | ---------------------------------------------------------------------------------------------------------------------------- |
| `GOOGLE_SERVICE_ACCOUNT_PATH`    | The path to a service account's key file.                                                                                    |
| `GOOGLE_SERVICE_ACCOUNT_KEY`     | A service account's key, as JSON.                                                                                            |
| `GOOGLE_APPLICATION_CREDENTIALS` | The path to [Application Default Credentials](https://cloud.google.com/docs/authentication/application-default-credentials). |

Without them, Amp uses the credentials of its environment, like a Compute Engine or GKE service account.

### Azure Blob Storage

Use `az://container/path` URLs (`azure://`, `abfs://`, `abfss://`, and `adl://` work too). Set the storage
account, and a way to authenticate to it:

| Variable                                                                            | Description                                  |
| ----------------------------------------------------------------------------------- | -------------------------------------------- |
| `AZURE_STORAGE_ACCOUNT_NAME`                                                        | The storage account.                         |
| `AZURE_STORAGE_ACCOUNT_KEY`                                                         | One of its access keys.                      |
| `AZURE_STORAGE_SAS_KEY`                                                             | A shared access signature, instead of a key. |
| `AZURE_STORAGE_CLIENT_ID`, `AZURE_STORAGE_CLIENT_SECRET`, `AZURE_STORAGE_TENANT_ID` | A service principal, instead of a key.       |

## How Amp organizes data

Under `data_dir`, each dataset has a directory, with one per table, and each table one per revision:

```
data/
└── acme/eth_mainnet/                 # The dataset's namespace and name
    └── logs/                         # A table
        └── 01a19d47-fdc9-71b3-883a-9fae41938c43/   # A revision
            ├── 024000000-4431ab5f879dda91.parquet  # Blocks from 24,000,000
            └── …
```

* **A revision** is one copy of a table's data. A table can have several, but queries read one: its active revision.
  Switching revisions changes which one the [metadata database](/running-amp/metadata-database) points to, without
  copying files. Revisions are named with an ID that sorts by creation time. See [Table
  Revisions](/managing-amp/table-revisions).
* **A file** holds a range of blocks. Its name starts with its first block, on 9 digits, followed by a random suffix.

Amp never changes a file: it writes new ones, merges small ones into new larger ones with
[compaction](/running-amp/storage/compaction), and deletes the ones it no longer needs with
[garbage collection](/running-amp/storage/garbage-collection).

The metadata database lists every file of every revision, with its metadata. Amp reads that list instead of listing
storage, which keeps query planning fast on object stores, where listing is slow. It also means:

* **Don't add, change, or delete files by hand.** Amp wouldn't see the new ones, and would fail to read the missing
  ones.
* **Back up storage and the database together.** A backup of the database only restores Amp while storage holds every
  file it lists, and compaction deletes files as it merges them. See
  [What a rollback needs](/running-amp/upgrading#what-a-rollback-needs).

## Parquet files

The `[writer]` section sets how workers write files. It applies to every job on a worker, and a change only applies to
the files written after the worker restarts:

| Setting                                                                           | Default      | Description                                                                                       |
| --------------------------------------------------------------------------------- | ------------ | ------------------------------------------------------------------------------------------------- |
| [`compression`](/running-amp/configuration-reference#writercompression)           | `"zstd(1)"`  | How files are compressed: see [Compression](/running-amp/storage#compression).                                        |
| [`bytes`](/running-amp/configuration-reference#writerbytes)                       | `2147483648` | The size a file grows to, in bytes: 2 GiB by default.                                             |
| [`rows`](/running-amp/configuration-reference#writerrows)                         | `0`          | The number of rows a file grows to. `0` is no limit.                                              |
| [`overflow`](/running-amp/configuration-reference#writeroverflow)                 | `1`          | How far past `bytes` and `rows` compaction may grow a file, as a multiple, like `1.5` or `"3/2"`. |
| [`max_row_group_mb`](/running-amp/configuration-reference#writermax_row_group_mb) | `128`        | The largest row group in a file, in MiB. A row group is the unit a query reads at once.           |

Raw dataset jobs close a file when it reaches `bytes`, or after
[`segment_flush_interval_secs`](/running-amp/configuration-reference#writersegment_flush_interval_secs), 600 seconds by
default. At a chain's head, they write small files often, which [compaction](/running-amp/storage/compaction) merges up
to `bytes` and `rows`. The section's other two settings size the
[metadata cache](/running-amp/storage/metadata-caching).

### Compression

Write `compression` as an algorithm. `zstd` takes a level in parentheses, like `"zstd(3)"`: higher levels make smaller
files, but take longer to write.

| Algorithm        | Levels             | Use it for                                      |
| ---------------- | ------------------ | ----------------------------------------------- |
| `zstd`           | 1 to 22, default 1 | Most cases: small files that read fast.         |
| `gzip`, `brotli` | none               | Readers that only support them.                 |
| `lz4`, `snappy`  | none               | The fastest writes and reads, for larger files. |
| `uncompressed`   | none               | The fastest writes, for the largest files.      |

A level for another algorithm, like `"gzip(6)"`, stops `ampd` from starting, with an error that includes:

```
unknown compression algorithm: gzip(6). Supported: zstd(N), lz4, gzip, brotli, snappy, uncompressed
```