> **Can't find what you're looking for?** Use `search_docs` on the docs MCP server at `https://docs.ampdb.io/api/mcp` to find what you need. # Compaction To keep up with a chain, a worker writes a small Parquet file every time new blocks arrive. A table with many small files is slower to query, since a query reads each one's metadata and opens each one. Compaction fixes that: in the background, it merges a table's adjacent small files into larger ones, up to a target size, while queries keep reading the table. Compaction is on by default, and runs inside the jobs that build [raw](/datasets/raw-datasets) and [derived](/datasets/derived-datasets) datasets, on their workers. You only need this page to tune it. ## How it works After a job writes a file, it starts a compaction pass on the table, unless one is already running, or the last one started less than [`min_interval`](/running-amp/storage/compaction#settings) ago. A pass: 1. Takes the table's files in block order, leaving out the newest ones ([`skip_latest_segments`](/running-amp/storage/compaction#settings)), whose blocks can still change if the chain reorganizes, which Amp calls a [reorg](/get-started/concepts/reorgs). 2. Groups adjacent files, from the oldest, as long as their total size stays under the target: `bytes` times `overflow`, and `rows` times `overflow` when `rows` is set. 3. Merges each group of two or more files into one new file. 4. Hands the merged files to [garbage collection](/running-amp/storage/garbage-collection), which deletes them once queries that may still read them have had time to finish. A failed merge is logged and counted in the [metrics](/running-amp/metrics#compaction-metrics), and doesn't stop the other groups. ### File ages To avoid rewriting the same data over and over, a file has to wait before it can start a group: * **Small files**, up to [`max_eager_bytes`](/running-amp/storage/compaction#settings) (100 KiB by default), can start a group right away. They're the many files written at the chain's head. * **Recent files**, larger than that and younger than [`cooldown_duration`](/running-amp/storage/compaction#settings) (about 17 minutes by default), can join a group that an older file started, but not start one. * **Older files** can start a group. A merged file is new, so it waits too before it's merged again. ### Sizes Compaction measures a file by the size of its data before compression. On disk, a file compressed with the default `zstd` is smaller than `bytes`: divide the size you want on disk by your tables' compression ratio to get `bytes`. The [metrics](/running-amp/metrics#compaction-metrics) give that ratio: `compaction_bytes_written_total` divided by `compaction_bytes_read_total`. A file that reaches the target on its own, like one a raw job closed when it reached `bytes`, is never merged again. ## Settings The `[writer.compactor]` section sets how compaction runs. It applies to every table on a worker, and a worker reads it when it starts, so restart workers after a change. | Setting | Default | Description | | ---------------------------------------------------------------------------------------------------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- | | [`active`](/running-amp/configuration-reference#writercompactoractive) | `true` | Runs compaction. `false` turns it off. | | [`min_interval`](/running-amp/configuration-reference#writercompactormin_interval) | `1.0` | The least time between two passes on a table, in seconds. | | [`skip_latest_segments`](/running-amp/configuration-reference#writercompactorskip_latest_segments) | `3` | How many of a table's newest files a pass leaves out, because a reorg can still replace them. | | [`cooldown_duration`](/running-amp/configuration-reference#writercompactorcooldown_duration) | `1024.0` | How long a file waits before it can start a group, in seconds. | | [`max_eager_bytes`](/running-amp/configuration-reference#writercompactormax_eager_bytes) | `102400` | Files up to this size, in bytes, don't wait. `0` makes every file wait. | | [`metadata_concurrency`](/running-amp/configuration-reference#writercompactormetadata_concurrency) | `2` | How many files' metadata a pass reads at once. | | [`write_concurrency`](/running-amp/configuration-reference#writercompactorwrite_concurrency) | `2` | How many groups a pass merges at once. | | [`gc_manifest_grace_period`](/running-amp/configuration-reference#writercompactorgc_manifest_grace_period) | `1800.0` | How long merged files stay in storage, in seconds, for queries that still read them. See [Garbage Collection](/running-amp/storage/garbage-collection). | The target size, and how merged files are written, come from [`[writer]`](/running-amp/storage#parquet-files), the same settings the jobs write their files with: | Setting | Default | Description | | --------------------------------------------------------------------------------- | ------------ | -------------------------------------------------------------------- | | [`bytes`](/running-amp/configuration-reference#writerbytes) | `2147483648` | The target size of files, in bytes, before compression. | | [`rows`](/running-amp/configuration-reference#writerrows) | `0` | The target number of rows of files. `0` is no limit. | | [`overflow`](/running-amp/configuration-reference#writeroverflow) | `1` | How far past the target a group may grow, as a multiple, like `1.5`. | | [`compression`](/running-amp/configuration-reference#writercompression) | `"zstd(1)"` | How files are compressed. | | [`max_row_group_mb`](/running-amp/configuration-reference#writermax_row_group_mb) | `128` | The largest row group in files, in MiB. | ## Tune compaction A table is well compacted when the part older than `cooldown_duration` is mostly files near the target size, and its number of small files stays the same as the table grows. Check it with the table's file sizes, from [`ampctl table histogram`](/clis/ampctl/table), and the [compaction metrics](/running-amp/metrics#compaction-metrics): a rising `failed_compactions`, or `compaction_bytes_read_total` growing much faster than the table, which means files are merged many times. * **Larger or smaller files**: set `bytes` and `rows` in `[writer]`. Lowering them doesn't split the files that are already larger: compaction only merges. Raw jobs also close the file they're writing when it reaches `bytes`. * **Files just over half the target don't merge**: with `overflow = 1`, two of them exceed the target together. A larger `overflow` lets them merge, into files above the target. * **Fewer rewrites**: a longer `cooldown_duration` rewrites files less often, but leaves more of them unmerged for longer. * **Small files piling up at the chain's head**: raise `max_eager_bytes`, so more of them merge right away. * **Reorgs**: make `skip_latest_segments` cover the chain's deepest reorg, in files. At the chain's head, a raw job writes one file per table each time new blocks arrive, checking for them every [`worker.poll_interval_secs`](/running-amp/configuration-reference#workerpoll_interval_secs). * **Load on storage**: lower `metadata_concurrency` and `write_concurrency`, or raise `min_interval`. For example, for files of about 512 MiB before compression, which may grow up to 4 times that to merge, with less load on storage: ```toml [config.toml] [writer] bytes = 536870912 overflow = 4 [writer.compactor] metadata_concurrency = 1 write_concurrency = 1 ``` To turn compaction off: ```toml [config.toml] [writer.compactor] active = false ``` ## Limitations * **Compaction only runs while the table's job runs.** A stopped dataset's tables aren't compacted, until it's deployed again. * **ADBC exports and Phaser datasets aren't compacted.** * **Sizes are estimates.** A group's size is the sum of its files' sizes, so a merged file can end up larger or smaller than the target once it's compressed again.