> **Can't find what you're looking for?** Use `search_docs` on the docs MCP server at `https://docs.ampdb.io/api/mcp` to find what you need.

# Catching Up

Status: Unstable.

The first time an [export](/managing-amp/pipelines) writes a table, it has the table's whole history to copy before it
can follow new blocks. This page explains where that export begins, how it crosses block ranges that have no rows
without spending time on them, and what you can set to make it faster.

## Prerequisites

The [start block example](/managing-amp/pipelines/catching-up#where-a-fresh-export-begins) needs what [Exports](/managing-amp/pipelines#prerequisites)
needs: Amp running with a worker, `ampctl`, `jq`, the `acme/eth_mainnet@1.0.0` dataset, and the `warehouse`
[destination](/managing-amp/pipelines/destinations). `warehouse` mustn't have exported `blocks` yet, since a start block
only applies to a fresh export.

## How an export walks a table

An export reads its source table in microbatches: block ranges of at most `worker.microbatch_max_interval` blocks
(100,000 by default, see the
[configuration reference](/running-amp/configuration-reference#workermicrobatch_max_interval)). For each microbatch, it
writes the range's rows to the destination, then records the range's last block in the destination's
[export state](/managing-amp/pipelines/destinations#export-state), in one transaction.

Many tables have a long stretch of blocks without rows at the start. A derived dataset of a contract's events, for
example, has nothing before the block that deployed the contract. Walking such a stretch 100,000 blocks at a time costs
a destination transaction for each microbatch, even with nothing to write, and on a chain with hundreds of millions of
blocks that's hours before the first row lands. Amp avoids that in three ways, all on by default.

## Where a fresh export begins

A table's export is fresh when the destination has no position recorded for it yet. Amp looks for the table's first
block with rows, reading the source's file statistics rather than its rows, and begins there. With a start block, it
begins at whichever is later:

| Start block set | Source has rows     | The export begins at                                                            |
| --------------- | ------------------- | ------------------------------------------------------------------------------- |
| No              | Yes, from block `F` | `F`                                                                             |
| Yes, `S`        | Yes, from block `F` | The later of `S` and `F`                                                        |
| No              | Not yet             | The start of the chain, crossing empty ranges [quickly](/managing-amp/pipelines/catching-up#crossing-empty-ranges) |
| Yes, `S`        | Not yet             | `S`                                                                             |

Set the start block with `--start-block` (`start_block` in the API) when you create the export. It only skips blocks: an
export never begins before the table's first rows.

```bash
HASH=$(ampctl dataset inspect acme/eth_mainnet@1.0.0 --json | jq -r .manifest_hash)
ampctl job create adbc acme eth_mainnet "$HASH" --provider warehouse --table blocks --start-block 24000050
```

The begin point only matters once. When the export starts again later, after a crash, a retry, or an export you stopped
and created again, it continues from the position the destination recorded, and ignores the start block.

With an end block too:

* **A start block past the end block** is refused when you create the export (`ADBC_START_AFTER_END_BLOCK`).
* **First rows past the end block**: the export doesn't skip ahead, since it would pass its end. It walks the range,
  which has no rows, and completes.

A table that combines several networks has no single first block, because their block numbers can't be compared. Its
export always begins at the start of each chain, and can't use start or end blocks: they make it fail.

:::warning
An export can't detect a reorg below the block it began at. If one ever replaces blocks before that point, rows below
it aren't corrected. This only matters for reorgs deeper than the distance between the chain's head and the table's
first rows, which is rare.
:::

## Crossing empty ranges

When the source's files show that a block range holds almost no data, at most 4 MiB, the export reads it as a single
microbatch, however many blocks it spans. A stretch of hundreds of millions of empty blocks then takes a few
microbatches instead of thousands. The microbatches go back to their usual size as soon as the data does.

This happens on its own, for every export of a table from a single network. Exports of tables that combine several
networks keep the usual microbatch size.

## Fewer commits on empty ranges

Under the `append` [reorg policy](/managing-amp/pipelines#reorg-policies), a microbatch with no rows doesn't write
anything to the destination right away. Amp holds its progress in memory and records it with the next microbatch that
has rows, so a run of empty microbatches costs one destination transaction instead of one each. It records held progress
without waiting for rows:

* once it has held progress for `state_heartbeat_secs` seconds,
* after 64 empty microbatches,
* and when the export reaches its end block.

A reorg drops held progress, since the reorg's replay covers those blocks again.

`state_heartbeat_secs` defaults to `worker.events.progress_interval`, 10 seconds. Set it per export:

| `ampctl job create adbc` flag    | API field              | Type               | Default                                        | Description                                                                                                                               |
| -------------------------------- | ---------------------- | ------------------ | ---------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------- |
| `--state-heartbeat-secs SECONDS` | `state_heartbeat_secs` | integer, 0 or more | `worker.events.progress_interval` (10 seconds) | How long progress over empty blocks may go unrecorded in the destination, like `--state-heartbeat-secs 60`. `0` records every microbatch. |

Held progress only ever covers blocks without rows, so nothing is lost when an export crashes while holding it: the next
run walks those blocks again, quickly. Rows are never held back.

The `replace` and `soft-delete` policies record every microbatch.

## What you see

* **A fresh export's first rows arrive within seconds** when its table has rows: it begins at them directly.
* **Progress jumps** across empty ranges, rather than advancing 100,000 blocks at a time.
* **`amp_export_state` can trail by up to `state_heartbeat_secs`** while an export crosses blocks without rows, so its
  `updated_at` doesn't change on every microbatch. A table that stays empty for long has its row updated at that rate,
  which you can use as a heartbeat.

## Limitations

* **The fast crossing only applies to a single network.** Tables that combine several networks keep the usual
  microbatch size, and begin at the start of each chain.
* **The 4 MiB threshold is fixed.** A range made of thousands of tiny files still pays to open each one.
* **Reorgs below a fresh export's first block go undetected**, as described [above](/managing-amp/pipelines/catching-up#where-a-fresh-export-begins).
* **Only the destination's bookkeeping is held back**, so the export state signals nothing over a source that produces
  no blocks at all. The job's status still shows whether it's running.