> **Can't find what you're looking for?** Use `search_docs` on the docs MCP server at `https://docs.ampdb.io/api/mcp` to find what you need. # Exports Status: Unstable. An export keeps a dataset's tables copied into a database you already run, like PostgreSQL or Snowflake. It exports every row the [dataset](/get-started/concepts/datasets) holds, then keeps following it, writing new blocks' rows as Amp extracts them. Use one when the tools that read your data, like dashboards, apps, or your team's SQL, work with that database rather than with Amp's [query server](/running-amp/query-server). An export is a [job](/managing-amp/jobs) of kind `adbc`, which a [worker](/managing-amp/workers) runs. It reads each source table a block range at a time and writes the range's rows to a [destination](/managing-amp/pipelines/destinations), a database registered with Amp, together with how far it got. Each table goes into the destination's `data_schema`, under the source table's name. Amp keeps the export's position in the destination itself, in a table called `amp_export_state`, so an export that stops for any reason continues where it left off (see [Resuming](/managing-amp/pipelines/resuming)). ## Prerequisites * Amp running, with a worker. [Solo mode](/running-amp/solo-mode) runs one, and serves the [Admin API](/managing-amp/admin-api) at `http://localhost:1610`. * [`ampctl`](/clis/ampctl/job), for the `ampctl` examples. It sends its requests to `http://localhost:1610` unless you pass `--admin-url` or set `AMP_ADMIN_URL`. * The ADBC driver for your database, which `ampd` loads when an export starts (see [Install ADBC drivers](/running-amp/installation#install-adbc-drivers)). * A deployed dataset to export. The examples use `acme/eth_mainnet@1.0.0`, with blocks 24,000,000 to 24,000,099, which the [Quickstart](/get-started/quickstart) deploys. An export copies the rows Amp has extracted so far, then follows new ones. * A PostgreSQL database registered as the destination `warehouse`, with its `data_schema` set to `eth`. To set one up, follow [Destinations](/managing-amp/pipelines/destinations). * `jq`, and for the steps that read the destination, `psql`, with the database's connection URL in an environment variable: ```bash export WAREHOUSE_URL=postgresql://amp:your-password@localhost:5432/analytics ``` ## Export a dataset ::::steps ### Find the dataset's manifest hash An export takes the [manifest hash](/datasets/versioning) of the dataset version to export, rather than its version. Look it up: ```bash HASH=$(ampctl dataset inspect acme/eth_mainnet@1.0.0 --json | jq -r .manifest_hash) ``` The export is pinned to that manifest hash. When `1.0.0`, or a revision like `latest`, later points somewhere else, the export keeps exporting the one it started with. ### Create the export Export `acme/eth_mainnet@1.0.0`'s `blocks` and `transactions` tables through `warehouse`, up to block 24,000,099, the last block the Quickstart extracted: :::code-group ```bash [ampctl] ampctl job create adbc acme eth_mainnet "$HASH" \ --provider warehouse \ --table blocks \ --table transactions \ --end-block 24000099 ``` ```bash [Admin API] curl -X POST http://localhost:1610/jobs \ -H 'Content-Type: application/json' \ -d "$(jq -n --arg hash "$HASH" '{ kind: "adbc", dataset_namespace: "acme", dataset_name: "eth_mainnet", manifest_hash: $hash, table_names: ["blocks", "transactions"], end_block: "24000099", adbc_target: {provider_name: "warehouse"} }')" ``` ::: ```text ✓ Job 57 created ``` The Admin API answers `202 Accepted`, with the job's ID: `{"job_id":57}`. Leave out the end block to keep the export running, following the dataset as Amp extracts new blocks. ### Follow its progress :::code-group ```bash [ampctl] ampctl job status 57 ``` ```bash [Admin API] curl http://localhost:1610/jobs/57/status ``` ::: The job's status is `RUNNING` while it exports, and `COMPLETED` once every table has reached its end block (see [Job Lifecycle](/managing-amp/jobs/lifecycle)). `ampctl` also prints `No tables written yet.` for every export: that line is about the tables a job writes in Amp, and an export writes into its destination instead. How far each table got is in the destination, in `amp_export_state` (see [Export state](/managing-amp/pipelines/destinations#export-state)): :::terminal ```bash psql "$WAREHOUSE_URL" -tAc 'SELECT target_table, network, last_committed_block FROM amp.amp_export_state ORDER BY 1' ``` ```text blocks|mainnet|24000099 transactions|mainnet|24000099 ``` ::: ### Check the destination Amp created `blocks` and `transactions` in the `eth` schema, with the source tables' columns. Query them like any other table: :::terminal ```bash psql "$WAREHOUSE_URL" -tAc 'SELECT min(_block_num), max(_block_num) FROM eth.blocks' ``` ```text 24000000|24000099 ``` ::: `_block_num` is the block each row comes from, a [reserved column](/datasets/manifests#reserved-columns). ### Stop it An export without an end block runs until you stop it: :::code-group ```bash [ampctl] ampctl job stop 57 ``` ```bash [Admin API] curl -X PUT http://localhost:1610/jobs/57/stop ``` ::: Its status then moves through `STOP_REQUESTED` and `STOPPING` to `STOPPED` (see [Job Lifecycle](/managing-amp/jobs/lifecycle#stopping)). Stopping an export that has already finished, like the one above once it's `COMPLETED`, succeeds without changing its status. The destination tables, and what Amp wrote in them, stay where they are. Creating the same export again later continues each table from the position Amp recorded in the destination. :::: ## Export options `ampctl job create adbc NAMESPACE NAME MANIFEST_HASH` takes the dataset's namespace, name, and manifest hash, then these options. The Admin API takes them as fields of the JSON body of `POST /jobs`, with `kind` set to `adbc`. | `ampctl` flag | Body field | Type | Default | Description | | -------------------------------------------- | ---------------------------------------------------- | ---------------------------------------------------------------- | ----------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `NAMESPACE NAME MANIFEST_HASH` | `dataset_namespace`, `dataset_name`, `manifest_hash` | string | required | The dataset version to export, by its manifest hash, like `acme eth_mainnet "$HASH"`. | | `--provider PROVIDER` | `adbc_target.provider_name` | string | required | The [destination](/managing-amp/pipelines/destinations) to export to, by name, like `--provider warehouse`. Its kind, `adbc-postgres` or `adbc-snowflake`, picks the driver. | | `--provider-version REQ` | `adbc_target.version` | version requirement | any version | A [version](/managing-amp/pipelines/destinations#versions) the destination must have, like `--provider-version '^1'`. | | `--table TABLE` | `table_names` | string, repeatable / array of strings | every table of the dataset | A source table to export. Repeat the flag for several, like `--table blocks --table transactions`. | | `--start-block BLOCK` | `start_block` | integer | the table's first block with rows | The block a table's export begins at the first time, like `--start-block 24000050`. See [Catching Up](/managing-amp/pipelines/catching-up#where-a-fresh-export-begins). | | `--end-block BLOCK` | `end_block` | integer, a decimal string in the API | none: the export keeps following the dataset | The block the export stops at, like `--end-block 24000099`. The job is `COMPLETED` once every table has exported it. | | `--reorg-policy POLICY` | `adbc_target.reorg_policy` | `append`, `replace`, or `soft-delete` (`soft_delete` in the API) | `append` | What happens to rows from blocks a reorg replaces. See [Reorg policies](/managing-amp/pipelines#reorg-policies). | | `--state-heartbeat-secs SECONDS` | `state_heartbeat_secs` | integer, 0 or more | `worker.events.progress_interval` (10 seconds) | How long progress over blocks without rows may go unrecorded in the destination. See [Catching Up](/managing-amp/pipelines/catching-up#fewer-commits-on-empty-ranges). | | `--worker-id WORKER` | `worker_id` | worker ID or glob pattern | the controller's default worker, or any active worker | The [worker](/managing-amp/workers) to run the export on, like `--worker-id worker-01`. | | `--retry-strategy` and the `--retry-*` flags | `retry_strategy` | see [Retries](/managing-amp/jobs/retries) | retry until stopped | How Amp retries the export after a failure. | An export has no trigger: it starts when you create it. Every `ampctl` command also takes `--admin-url`, `--auth-token`, and `--json`. With `--json`, `job create` prints the job's ID and the request it sent. ### Creating an export again Amp tells exports apart by their dataset version, tables, and destination. Creating an export that's already active returns its job when the other options are the same too, and fails with `409 ACTIVE_JOB_CONFLICT` when they aren't. Once it has stopped or completed, creating it again schedules the same job again, under the same ID, and it continues each table from the position Amp recorded in the destination. ### Changing an export An export's options are fixed while its job is active. To change them, [stop it](/managing-amp/pipelines#stop-it) and wait until its status is `STOPPED`, then create it again with the new options. An export that has already finished, `COMPLETED` or `FATAL`, needs no stopping: create it again right away. The job continues each table from the position Amp recorded in the destination, so nothing is exported twice or skipped, and `--start-block` only applies to tables that have no position yet. Two changes need new destination tables, in a destination with another `data_schema`: * **A different reorg policy.** `replace` and `soft-delete` add columns to the tables they create, and a table created under another policy doesn't have them. * **A different dataset version with other columns.** Amp checks the existing tables' columns before writing. A change to the destination's own configuration, like its password, reaches an export the next time its job starts: on its next retry, or when you stop it and create it again. ## Reorg policies A reorg replaces blocks the chain had already produced with different ones (see [Reorgs](/get-started/concepts/reorgs)). When the source dataset's rows for those blocks change, the export writes the block range again. The reorg policy decides what happens to the rows already in the destination for the replaced blocks: | Policy | Rows from replaced blocks | Columns Amp adds | | --------------------------------------------- | ------------------------------------------------------------------------- | ------------------------------------------ | | `append` | Stay where they are. The new rows are added beside them. | None | | `replace` | Deleted. The new rows take their place. | `_amp_commit_id` | | `soft-delete` (`ampctl`), `soft_delete` (API) | Kept, with `_amp_invalidated_by` set. The new rows are added beside them. | `_amp_commit_id` and `_amp_invalidated_by` | * **`append`** is the default, and the cheapest. Under it, a table can hold rows for the same block from both versions of the chain, so use it when you only export finalized blocks, or when whatever reads the table can cope with that. * **`replace`** keeps each table equal to the source. * **`soft-delete`** also keeps the history: to read the current rows, filter on `_amp_invalidated_by IS NULL`. The added columns tie rows to the export's commits in the destination's [export state table](/managing-amp/pipelines/destinations#export-state): `_amp_commit_id` is the commit that wrote the row, and `_amp_invalidated_by` the commit of the reorg that replaced it, or `NULL` while the row is current. A source table can't have columns with these names. `replace` and `soft-delete` have requirements the export only checks when it starts, failing if they're not met: * **PostgreSQL only.** On Snowflake the export fails with `reorg policy Replace requires transactional ingest, which the adbc-snowflake driver lacks` (or `SoftDelete`), or, in the default `export_state_mode`, with the next error first. * **A destination whose `export_state_mode` is `append` or `upsert_retry`**, not the default `upsert`. Otherwise the export fails with `reorg policy Replace requires an Append-shaped export state mode, got "upsert"`. See [Export state](/managing-amp/pipelines/destinations#export-state). * **Tables from a single network.** For a table that combines several networks, the export fails with `reorg policy Replace tracks commit ids per batch and cannot serve multi-network table "…"`. ## Errors Creating an export can fail with these codes, in the `error_code` field of the Admin API's response. `ampctl` prints the response's message. | Code | Status | Cause and fix | | -------------------------------- | ------ | --------------------------------------------------------------------------------------------------------------------- | | `INVALID_BODY` | 400 | The body isn't a valid request: check the field names and types under [Export options](/managing-amp/pipelines#export-options). | | `ADBC_PROVIDER_NOT_FOUND` | 400 | No destination is registered under that name. Register the [destination](/managing-amp/pipelines/destinations) first. | | `ADBC_PROVIDER_KIND_UNSUPPORTED` | 400 | The destination's kind isn't one Amp can export to: `adbc-postgres` or `adbc-snowflake`. | | `ADBC_VERSION_REQ_UNSATISFIED` | 400 | The destination's version doesn't meet `--provider-version`. The message gives its version. | | `ADBC_START_AFTER_END_BLOCK` | 400 | `--start-block` is past `--end-block`. | | `NO_WORKERS_AVAILABLE` | 400 | No worker is running. Start one, then try again. | | `WORKER_NOT_AVAILABLE` | 400 | The worker `--worker-id` names isn't active. | | `ACTIVE_JOB_CONFLICT` | 409 | The same export is already active (see [Creating an export again](/managing-amp/pipelines#creating-an-export-again)). | Amp doesn't check the dataset or its tables when you create an export. The job fails when it starts, with an error code in its status like `TABLE_NOT_FOUND` (a `--table` isn't a table of the dataset) or `NO_ACTIVE_REVISION` (the dataset isn't deployed). The [Admin API reference](/managing-amp/admin-api/jobs#create-a-job) lists every code of `POST /jobs`. ## Limitations * **Every export to Snowflake fails at v0.0.42**, without writing a row (see [Register a Snowflake destination](/managing-amp/pipelines/destinations#register-a-snowflake-destination)). * **Snowflake destinations only support `append`**, and once exports work, can write a microbatch's rows twice after a crash (see [Exactly-Once Delivery](/managing-amp/pipelines/exactly-once)). * **Tables are written to the destination's `data_schema`, under their source names.** You can't choose another schema or name per export. Use a destination with another `data_schema` to write the same tables twice. * **Nothing keeps two exports out of one table.** Two exports with different table lists, or different dataset versions, can both write a table they share. Give each destination table a single export. * **Progress isn't in the job's status.** Read it from the destination's `amp_export_state`, which can trail by up to `state_heartbeat_secs` while the export crosses blocks without rows. * **Start and end blocks don't work for tables that combine several networks**, whose block numbers can't be compared: their export fails.