> **Can't find what you're looking for?** Use `search_docs` on the docs MCP server at `https://docs.ampdb.io/api/mcp` to find what you need.

# Resuming

Status: Unstable.

An [export](/managing-amp/pipelines) can stop at any time: its worker restarts or crashes, it fails and waits to be
retried, or you stop it and create it again. When it starts again, it continues each table right after the last block
it wrote, without skipping a block or writing one twice. If a [reorg](/get-started/concepts/reorgs) replaced blocks it
had already written while it was stopped, it notices, and corrects them before going on. Nothing needs setting up.

## Where an export continues

Every time an export writes a microbatch of rows, it also records, in the destination's
[`amp_export_state`](/managing-amp/pipelines/destinations#export-state) table and in the same transaction, the
microbatch's last block and that block's hash:

| Column                 | What it holds                                                 |
| ---------------------- | ------------------------------------------------------------- |
| `last_committed_block` | The last block written to the table, for one source network.  |
| `resume_hash`          | That block's hash, as `0x` followed by 64 hexadecimal digits. |

When the export starts, it reads these back for each table and network, and continues from the next block. A table with
nothing recorded is a fresh export, which begins as
[Catching Up](/managing-amp/pipelines/catching-up#where-a-fresh-export-begins)
describes.

Since the position lives in the destination, it survives what happens on Amp's side, like a new worker or an export you
stopped and created again. It also means the rows and their position can't disagree, on PostgreSQL. On Snowflake, once
its exports work, they can, after a crash: see [Exactly-Once Delivery](/managing-amp/pipelines/exactly-once).

## Reorgs while an export was stopped

Before going on, the export checks that the block it recorded, with its hash, is still part of the chain the source
dataset holds:

* **It is.** Nothing changed under the export, and it continues from the next block.
* **It isn't.** A reorg replaced that block while the export was stopped. The export walks back to the last block both
  chains share, and writes the blocks after it again, as a reorg. The destination tables are corrected according to the
  export's [reorg policy](/managing-amp/pipelines#reorg-policies), exactly as for a reorg that happens while the export
  runs, before any new block is exported.

Under the `append` policy, correcting means adding the new rows: the rows from the replaced blocks stay, as they do for
any reorg under that policy.

## Resuming without a hash

A position recorded by a version of Amp from before `resume_hash` existed has no hash. For a table from a single
network, the export then continues from the next block without the check above, and the worker logs a warning:

```text
resumed without chain-continuity verification
```

A reorg that happened while that export was stopped goes undetected. The warning appears once per table: the next
microbatch records a hash, and every later start is checked.

A table that combines several networks always has a hash for each network. If one is missing, the position can't be
checked, and the export fails rather than trust it.

## When the position can't be trusted

The export fails, without retrying, when what it reads from `amp_export_state` doesn't make sense. Each of these means
the table was edited by hand, or the export's tables or networks changed:

| Error                                                            | Cause                                                                   |
| ---------------------------------------------------------------- | ----------------------------------------------------------------------- |
| `invalid resume hash recorded for table "…"`                     | A `resume_hash` isn't a block hash.                                     |
| `stored network key "…" is not a valid network id`               | A `network` isn't a network's name.                                     |
| `multi-network resume requires every network's watermark hash`   | A table that combines several networks has a position without a hash.   |
| `table "…" records networks [...] where the plan declares [...]` | The networks recorded for a table aren't the ones its source table has. |

To recover, put back the rows Amp wrote, or start the table over: stop the export, drop the destination table, delete
its rows from `amp_export_state` (the ones with its `target_schema` and `target_table`), and start the export again.
See [Exactly-Once Delivery](/managing-amp/pipelines/exactly-once#remove-them) for an example.

## Upgrading Amp

The `resume_hash` column, and the per-network `network` column before it, are added to existing `amp_export_state`
tables on their own. Each time an export connects, Amp adds whichever of them is missing: `resume_hash` as a nullable
column, and `network` with an empty default, so existing rows stay as they are. Exports running on several workers at
once can do this together safely.

After an upgrade, each table's first start resumes without a hash, with the warning above, and its first microbatch
records one.

Going back to an older version is safe too: it ignores the column. Its exports record positions without a hash, so the
next upgrade resumes once without the check. Under the default `upsert` [export state
mode](/managing-amp/pipelines/destinations#export-state), the older version updates the block but leaves the old hash,
and the next upgrade sees them disagree and replays one reorg that didn't happen. That only happens under the `append`
reorg policy, where it adds a copy of the replayed rows.

## Limitations

* **A completed export isn't checked again.** An export that reached its end block does nothing when it starts again,
  so a reorg below its end block afterwards isn't corrected.
* **A fresh export's first block isn't checked.** A reorg below where an export began goes undetected (see
  [Catching Up](/managing-amp/pipelines/catching-up#where-a-fresh-export-begins)).
* **Finding where chains split needs the replaced blocks.** The export finds the last shared block in the source
  dataset's copies of the replaced blocks. If [garbage collection](/running-amp/storage/garbage-collection) already
  removed them, it replays from the replaced block's own height instead, which can miss earlier replaced blocks.