> **Can't find what you're looking for?** Use `search_docs` on the docs MCP server at `https://docs.ampdb.io/api/mcp` to find what you need.

# Configuration Reference

Every setting `ampd` reads from its configuration file, by TOML table. To learn how the file works, how environment variables override it, and how to set a query engine setting once for every service, see [Configuration](/running-amp/configuration). The metadata database's settings, in `[metadb]`, are in [Metadata Database](/running-amp/metadata-database).

## Top level

### `data_dir`

Where the extracted datasets are stored.

* **Type**: string
* **Default**: `"data"`
* **Environment variable**: `AMP_CONFIG_DATA_DIR`

```toml
data_dir = "data"
```

### `providers_dir`

Path to a providers directory. Each provider is configured as a separate toml file in this directory.

* **Type**: string
* **Default**: `"providers"`
* **Environment variable**: `AMP_CONFIG_PROVIDERS_DIR`

```toml
providers_dir = "providers"
```

### `manifests_dir`

Path to a directory containing dataset manifest files.

* **Type**: string
* **Default**: `"manifests"`
* **Environment variable**: `AMP_CONFIG_MANIFESTS_DIR`

```toml
manifests_dir = "manifests"
```

## `[logging]`

Observability (shared by all services).

### `logging.json`

Output logs as JSON.

* **Type**: boolean
* **Default**: `false`
* **Environment variable**: `AMP_CONFIG_LOGGING__JSON`

```toml
[logging]
json = false
```

## `[writer]`

Writer/Parquet configuration (shared storage layer: the worker writes, every service reads the metadata cache settings).

### `writer.compression`

Compression algorithm. Supported: zstd(N), lz4, gzip, brotli, snappy, uncompressed.

* **Type**: string
* **Default**: `"zstd(1)"`
* **Environment variable**: `AMP_CONFIG_WRITER__COMPRESSION`

```toml
[writer]
compression = "zstd(1)"
```

### `writer.cache_size_mb`

Parquet metadata cache size in MB.

* **Type**: integer
* **Default**: `1024`
* **Environment variable**: `AMP_CONFIG_WRITER__CACHE_SIZE_MB`

```toml
[writer]
cache_size_mb = 1024
```

### `writer.cache_time_to_idle_secs`

Time-to-idle for parquet metadata cache entries, in seconds. Entries not accessed for this duration are evicted.

* **Type**: number
* **Default**: `120`
* **Environment variable**: `AMP_CONFIG_WRITER__CACHE_TIME_TO_IDLE_SECS`

```toml
[writer]
cache_time_to_idle_secs = 120
```

### `writer.max_row_group_mb`

Maximum row group size in MB.

* **Type**: integer
* **Default**: `128`
* **Environment variable**: `AMP_CONFIG_WRITER__MAX_ROW_GROUP_MB`

```toml
[writer]
max_row_group_mb = 128
```

### `writer.overflow`

Overflow multiplier: 1x target size, can use `"1.5"` for 1.5x, etc.

Accepts an integer, float, or fraction string (e.g. `1`, `1.5`, `"3/2"`).

* **Type**: integer, number, or string
* **Default**: `"1"`
* **Environment variable**: `AMP_CONFIG_WRITER__OVERFLOW`

```toml
[writer]
overflow = "1"
```

### `writer.bytes`

Target bytes per file.

* **Type**: integer
* **Default**: `2147483648`
* **Environment variable**: `AMP_CONFIG_WRITER__BYTES`

```toml
[writer]
bytes = 2147483648
```

### `writer.rows`

Target rows per file, 0 means no limit.

* **Type**: integer
* **Default**: `0`
* **Environment variable**: `AMP_CONFIG_WRITER__ROWS`

```toml
[writer]
rows = 0
```

### `writer.segment_flush_interval_secs`

Maximum wall-clock time before closing a segment, in seconds.

* **Type**: number
* **Default**: `600` (10 min)
* **Environment variable**: `AMP_CONFIG_WRITER__SEGMENT_FLUSH_INTERVAL_SECS`

```toml
[writer]
segment_flush_interval_secs = 600
```

## `[writer.compactor]`

Compaction settings.

### `writer.compactor.active`

Enable or disable the compactor.

* **Type**: boolean
* **Default**: `true`
* **Environment variable**: `AMP_CONFIG_WRITER__COMPACTOR__ACTIVE`

```toml
[writer.compactor]
active = true
```

### `writer.compactor.metadata_concurrency`

Maximum concurrent metadata operations.

* **Type**: integer
* **Default**: `2`
* **Environment variable**: `AMP_CONFIG_WRITER__COMPACTOR__METADATA_CONCURRENCY`

```toml
[writer.compactor]
metadata_concurrency = 2
```

### `writer.compactor.write_concurrency`

Limits concurrent compaction write operations; defaults to `2`.

* **Type**: integer
* **Allowed**: at least 1
* **Default**: `2`
* **Environment variable**: `AMP_CONFIG_WRITER__COMPACTOR__WRITE_CONCURRENCY`

```toml
[writer.compactor]
write_concurrency = 2
```

### `writer.compactor.min_interval`

Interval in seconds between compactor runs.

* **Type**: number
* **Default**: `1`
* **Environment variable**: `AMP_CONFIG_WRITER__COMPACTOR__MIN_INTERVAL`

```toml
[writer.compactor]
min_interval = 1
```

### `writer.compactor.gc_manifest_grace_period`

Grace period in seconds before compacted parent files become eligible for garbage collection.

* **Type**: number
* **Default**: `1800` (30 min)
* **Environment variable**: `AMP_CONFIG_WRITER__COMPACTOR__GC_MANIFEST_GRACE_PERIOD`

```toml
[writer.compactor]
gc_manifest_grace_period = 1800
```

### `writer.compactor.cooldown_duration`

Minimum time in seconds before a file can initiate a new compaction group. Prevents excessive I/O from rewriting large files too frequently.

* **Type**: number
* **Default**: `1024`
* **Environment variable**: `AMP_CONFIG_WRITER__COMPACTOR__COOLDOWN_DURATION`

```toml
[writer.compactor]
cooldown_duration = 1024
```

### `writer.compactor.skip_latest_segments`

Number of latest segments to skip during compaction. Avoids compacting recently-written data that may contain reorganized blocks.

* **Type**: integer
* **Default**: `3`
* **Environment variable**: `AMP_CONFIG_WRITER__COMPACTOR__SKIP_LATEST_SEGMENTS`

```toml
[writer.compactor]
skip_latest_segments = 3
```

### `writer.compactor.max_eager_bytes`

Files at or below this size in bytes are exempt from the cooldown (the "Live" tier). `0` disables the Live tier; the default is 100 KiB.

* **Type**: integer
* **Default**: `102400`
* **Environment variable**: `AMP_CONFIG_WRITER__COMPACTOR__MAX_EAGER_BYTES`

```toml
[writer.compactor]
max_eager_bytes = 102400
```

## `[location_membership]`

Location membership counter cache and change-log settings (shared: every service reads the counter, the controller prunes the log).

### `location_membership.cache_size_mb`

Membership id-set cache size in MB.

* **Type**: integer
* **Default**: `1024`
* **Environment variable**: `AMP_CONFIG_LOCATION_MEMBERSHIP__CACHE_SIZE_MB`

```toml
[location_membership]
cache_size_mb = 1024
```

### `location_membership.cache_time_to_idle_secs`

Time-to-idle for membership cache entries, in seconds. Entries not accessed for this duration are evicted.

* **Type**: number
* **Default**: `120`
* **Environment variable**: `AMP_CONFIG_LOCATION_MEMBERSHIP__CACHE_TIME_TO_IDLE_SECS`

```toml
[location_membership]
cache_time_to_idle_secs = 120
```

### `location_membership.max_delta_versions`

Longest run of change-log versions a reader replays before reading every file id instead, and the number of versions the pruner keeps behind each location's counter.

One value, because they are the same number: a version the reader will not replay is a version the log has no reason to hold.

* **Type**: integer
* **Allowed**: at least 1
* **Default**: `10000`
* **Environment variable**: `AMP_CONFIG_LOCATION_MEMBERSHIP__MAX_DELTA_VERSIONS`

```toml
[location_membership]
max_delta_versions = 10000
```

### `location_membership.log_prune_interval_secs`

How often the change log is pruned, in seconds.

Retention is by version, not by age, so this only sets how long deletable rows linger. Must be positive: a zero interval would spin the prune loop.

* **Type**: number
* **Default**: `3600` (1 hour)
* **Environment variable**: `AMP_CONFIG_LOCATION_MEMBERSHIP__LOG_PRUNE_INTERVAL_SECS`

```toml
[location_membership]
log_prune_interval_secs = 3600
```

## `[server]`

Query-server (`server`) settings.

### `server.flight_addr`

Arrow Flight RPC server address.

* **Type**: string
* **Default**: `"0.0.0.0:1602"`
* **Environment variable**: `AMP_CONFIG_SERVER__FLIGHT_ADDR`

```toml
[server]
flight_addr = "0.0.0.0:1602"
```

### `server.keep_alive_interval`

Keep-alive interval for the streaming server in seconds.

* **Type**: integer
* **Allowed**: at least 30
* **Default**: `30`
* **Environment variable**: `AMP_CONFIG_SERVER__KEEP_ALIVE_INTERVAL`

```toml
[server]
keep_alive_interval = 30
```

### `server.max_mem_mb`

How much memory the server can use in MB. 0 means unlimited.

* **Type**: integer
* **Default**: `0`
* **Environment variable**: `AMP_CONFIG_SERVER__MAX_MEM_MB`

```toml
[server]
max_mem_mb = 0
```

### `server.query_max_mem_mb`

Per-query memory limit in MB. 0 means unlimited per-query.

* **Type**: integer
* **Default**: `0`
* **Environment variable**: `AMP_CONFIG_SERVER__QUERY_MAX_MEM_MB`

```toml
[server]
query_max_mem_mb = 0
```

### `server.query_timeout_secs` (Experimental)

Limits batch query execution to this many seconds; `0` disables the timeout.

* **Type**: integer
* **Default**: `0`
* **Environment variable**: `AMP_CONFIG_SERVER__QUERY_TIMEOUT_SECS`

```toml
[server]
query_timeout_secs = 0
```

### `server.spill_location`

Paths for DataFusion temporary files for spill-to-disk.

* **Type**: array of strings
* **Default**: `[]`
* **Environment variable**: `AMP_CONFIG_SERVER__SPILL_LOCATION`

```toml
[server]
spill_location = ["/mnt/scratch/amp-spill"]
```

### `server.metadata_fetch_concurrency`

Maximum concurrent parquet metadata fetches during query planning.

* **Type**: integer
* **Allowed**: at least 1
* **Default**: `32`
* **Environment variable**: `AMP_CONFIG_SERVER__METADATA_FETCH_CONCURRENCY`

```toml
[server]
metadata_fetch_concurrency = 32
```

### `server.microbatch_max_interval`

Default max interval for streaming microbatches. In single-network mode this is a block count; in multi-network mode it is interpreted as seconds. Used as the default for streaming server queries when the client does not override via the SQL `SETTINGS` clause.

* **Type**: integer
* **Default**: `100000`
* **Environment variable**: `AMP_CONFIG_SERVER__MICROBATCH_MAX_INTERVAL`

```toml
[server]
microbatch_max_interval = 100000
```

## `[worker]`

Extraction-worker (`worker`) settings.

### `worker.health_addr`

Worker health check server address.

* **Type**: string
* **Default**: `"0.0.0.0:1620"`
* **Environment variable**: `AMP_CONFIG_WORKER__HEALTH_ADDR`

```toml
[worker]
health_addr = "0.0.0.0:1620"
```

### `worker.poll_interval_secs`

Polling interval for new blocks during dump in seconds.

* **Type**: number
* **Default**: `1.0`
* **Environment variable**: `AMP_CONFIG_WORKER__POLL_INTERVAL_SECS`

```toml
[worker]
poll_interval_secs = 1
```

### `worker.max_mem_mb`

How much memory the worker can use in MB. 0 means unlimited.

* **Type**: integer
* **Default**: `0`
* **Environment variable**: `AMP_CONFIG_WORKER__MAX_MEM_MB`

```toml
[worker]
max_mem_mb = 0
```

### `worker.query_max_mem_mb`

Per-query memory limit in MB. 0 means unlimited per-query.

* **Type**: integer
* **Default**: `0`
* **Environment variable**: `AMP_CONFIG_WORKER__QUERY_MAX_MEM_MB`

```toml
[worker]
query_max_mem_mb = 0
```

### `worker.spill_location`

Paths for DataFusion temporary files for spill-to-disk.

* **Type**: array of strings
* **Default**: `[]`
* **Environment variable**: `AMP_CONFIG_WORKER__SPILL_LOCATION`

```toml
[worker]
spill_location = ["/mnt/scratch/amp-spill"]
```

### `worker.metadata_fetch_concurrency`

Maximum concurrent parquet metadata fetches during query planning.

* **Type**: integer
* **Allowed**: at least 1
* **Default**: `32`
* **Environment variable**: `AMP_CONFIG_WORKER__METADATA_FETCH_CONCURRENCY`

```toml
[worker]
metadata_fetch_concurrency = 32
```

### `worker.microbatch_max_interval`

Max interval for microbatches. In single-network mode this is a block count. In multi-network mode it is interpreted as seconds because block numbers are not comparable across networks. Used for derived dataset dumps.

* **Type**: integer
* **Default**: `100000`
* **Environment variable**: `AMP_CONFIG_WORKER__MICROBATCH_MAX_INTERVAL`

```toml
[worker]
microbatch_max_interval = 100000
```

## `[worker.events]` (Experimental)

Worker event streaming configuration.

### `worker.events.enabled`

Enable/disable event emission.

* **Type**: boolean
* **Default**: `false`
* **Environment variable**: `AMP_CONFIG_WORKER__EVENTS__ENABLED`

```toml
[worker.events]
enabled = false
```

### `worker.events.event_queue_capacity`

In-memory queue capacity for best-effort worker events.

* **Type**: integer
* **Allowed**: at least 1
* **Default**: `1024`
* **Environment variable**: `AMP_CONFIG_WORKER__EVENTS__EVENT_QUEUE_CAPACITY`

```toml
[worker.events]
event_queue_capacity = 1024
```

### `worker.events.progress_interval`

Progress event emission interval. Progress events are emitted at most once per this interval when there is new progress.

* **Type**: number
* **Default**: `10` seconds
* **Environment variable**: `AMP_CONFIG_WORKER__EVENTS__PROGRESS_INTERVAL`

```toml
[worker.events]
progress_interval = 10
```

## `[worker.events.kafka]` (Experimental)

Kafka-specific configuration.

### `worker.events.kafka.brokers`

Kafka broker addresses (e.g., \["kafka-1:9092", "kafka-2:9092"]).

* **Type**: array of strings
* **Required** when the table is set
* **Environment variable**: `AMP_CONFIG_WORKER__EVENTS__KAFKA__BROKERS`

```toml
[worker.events.kafka]
brokers = ["kafka-1:9092", "kafka-2:9092"]
```

### `worker.events.kafka.topic`

Kafka topic name.

* **Type**: string
* **Default**: `"amp.worker.events"`
* **Environment variable**: `AMP_CONFIG_WORKER__EVENTS__KAFKA__TOPIC`

```toml
[worker.events.kafka]
topic = "amp.worker.events"
```

### `worker.events.kafka.partitions`

Number of partitions for the Kafka topic.

This should match the actual partition count of your Kafka topic. Used for consistent partition key hashing.

* **Type**: integer
* **Default**: `16`
* **Environment variable**: `AMP_CONFIG_WORKER__EVENTS__KAFKA__PARTITIONS`

```toml
[worker.events.kafka]
partitions = 16
```

### `worker.events.kafka.sasl_mechanism`

SASL authentication mechanism (optional).

Supported values: "PLAIN", "SCRAM-SHA-256", "SCRAM-SHA-512" If not set, no SASL authentication is used.

* **Type**: string
* **Default**: none
* **Environment variable**: `AMP_CONFIG_WORKER__EVENTS__KAFKA__SASL_MECHANISM`

```toml
[worker.events.kafka]
sasl_mechanism = "SCRAM-SHA-256"
```

### `worker.events.kafka.sasl_username`

SASL username (required if sasl_mechanism is set).

* **Type**: string
* **Default**: none
* **Environment variable**: `AMP_CONFIG_WORKER__EVENTS__KAFKA__SASL_USERNAME`

```toml
[worker.events.kafka]
sasl_username = "amp-worker"
```

### `worker.events.kafka.sasl_password`

SASL password (required if sasl_mechanism is set).

* **Type**: string
* **Default**: none
* **Environment variable**: `AMP_CONFIG_WORKER__EVENTS__KAFKA__SASL_PASSWORD`

```toml
[worker.events.kafka]
sasl_password = "your-password"
```

### `worker.events.kafka.tls_enabled`

Enable TLS encryption.

When enabled, connections to Kafka brokers use TLS.

* **Type**: boolean
* **Default**: `false`
* **Environment variable**: `AMP_CONFIG_WORKER__EVENTS__KAFKA__TLS_ENABLED`

```toml
[worker.events.kafka]
tls_enabled = false
```

### `worker.events.kafka.tls_ca_cert_path`

Path to a PEM-encoded CA certificate file for TLS verification (optional).

When set, the Kafka client will trust this CA certificate for verifying broker TLS connections, instead of using system root certificates. Required when connecting to brokers using self-signed or internal CA certificates.

* **Type**: string
* **Default**: none
* **Environment variable**: `AMP_CONFIG_WORKER__EVENTS__KAFKA__TLS_CA_CERT_PATH`

```toml
[worker.events.kafka]
tls_ca_cert_path = "/etc/amp/kafka-ca.pem"
```

## `[controller]`

Controller / Admin API (`controller`) settings.

### `controller.admin_api_addr`

Admin API server address.

* **Type**: string
* **Default**: `"0.0.0.0:1610"`
* **Environment variable**: `AMP_CONFIG_CONTROLLER__ADMIN_API_ADDR`

```toml
[controller]
admin_api_addr = "0.0.0.0:1610"
```

### `controller.default_worker_id`

Default worker ID for jobs without an explicit worker selector.

* **Type**: string
* **Default**: none
* **Environment variable**: `AMP_CONFIG_CONTROLLER__DEFAULT_WORKER_ID`

```toml
[controller]
default_worker_id = "worker-01"
```