> **Can't find what you're looking for?** Use `search_docs` on the docs MCP server at `https://docs.ampdb.io/api/mcp` to find what you need. # Upgrading A new version of Amp can change the tables of its [metadata database](/running-amp/metadata-database), the PostgreSQL database where Amp keeps its state. So an upgrade has three steps: install the new version, migrate the database to it, then restart every service on it. Your datasets' data carries over: a newer Amp reads the Parquet files an older one wrote, so you don't extract them again. The reverse isn't true: an older Amp may not read what a newer one wrote, and won't start on a database a newer one migrated. To go back, you [restore a backup](/running-amp/upgrading#roll-back) you took before the upgrade. ## Before you upgrade * **Read the release notes** of each version between yours and the new one, on [Amp's releases page](https://github.com/edgeandnode/amp/releases), and the [version-specific notes](/running-amp/upgrading#version-specific-notes) below. * **Plan to back up Amp.** A backup is the only way to [roll back](/running-amp/upgrading#roll-back), and it has to cover your datasets' files too: see [What a rollback needs](/running-amp/upgrading#what-a-rollback-needs). The steps below take one. * **Try the upgrade on a staging deployment first**, with a copy of your production database, so you see how long the migrations take and whether they fail before production does. ## What a rollback needs Amp's state is in two places that must agree: the metadata database, which lists every file of every table, and [storage](/running-amp/storage), which holds the files. A backup of the database only restores Amp while storage still holds every file it lists. Workers delete files as they run: [compaction](/running-amp/storage/compaction) merges small files into larger ones, and [garbage collection](/running-amp/storage/garbage-collection) deletes the merged ones once their grace period ends, 30 minutes by default. Restore a backup from before that, and its tables list files that no longer exist: their jobs fail, and so do queries that read those blocks. So back up one of two ways: * **Copy storage with the database**, while Amp is stopped, and restore both. That's how you back up [solo mode](/running-amp/upgrading#upgrade-solo-mode), which keeps both in its `.amp` directory. * **Turn compaction off** before you back up the database, and keep it off until you're sure you'll keep the new version. Nothing deletes a file the backup lists then, as long as you don't [prune](/running-amp/storage/garbage-collection#delete-a-tables-abandoned-files), truncate, or delete [table revisions](/managing-amp/table-revisions) either. That's how you back up [distributed mode](/running-amp/upgrading#upgrade-distributed-mode), whose storage is usually too large to copy, and solo mode with storage outside `.amp`. To turn compaction off, set `active = false` in the [`[writer.compactor]`](/running-amp/storage/compaction#settings) section, or set `AMP_CONFIG_WRITER__COMPACTOR__ACTIVE=false`, which overrides the file. While compaction is off, tables that grow at a chain's head collect small files, which slows their queries until compaction is back on. ## Upgrade solo mode [Solo mode](/running-amp/solo-mode) migrates its database when it starts, so you only need to install the new version and restart it: ::::steps ### Stop solo mode Press Ctrl+C where `ampd solo` runs, and wait for it to exit. ### Back up the `.amp` directory Solo mode's [`.amp` directory](/running-amp/solo-mode#the-amp-directory) holds both its database and its datasets' files, so a copy of it, taken while `ampd` is stopped, is a backup of both. Copy it to a new directory, named after the time: ```bash cp -R .amp ~/amp-backup-$(date +%Y%m%d-%H%M%S) ``` Don't copy it to a directory that exists, like an older backup's: `cp` would put the copy inside it, and a [rollback](/running-amp/upgrading#roll-back-solo-mode) would restore the older backup. If `.amp` doesn't hold everything, back up the rest too: * **With [your own PostgreSQL](/running-amp/solo-mode#use-your-own-postgresql)**, `.amp` doesn't hold the database. [Back it up](/running-amp/upgrading#back-up-the-database) too, while solo mode is stopped. * **With storage outside `.amp`**, like a bucket, the copy doesn't hold your datasets' files. Once you've copied `.amp`, [turn compaction off](/running-amp/upgrading#what-a-rollback-needs) in `.amp/config.toml`, so that solo mode keeps the files the copy lists. Where the section is already there, change its `active` instead of adding the section again: ```toml [.amp/config.toml] [writer.compactor] active = false ``` Or set `AMP_CONFIG_WRITER__COMPACTOR__ACTIVE=false` where you run `ampd solo`. Once you're sure you'll keep the new version, remove both settings, and restart solo mode. ### Install the new version `ampup install` installs the latest release and switches to it. Give it a version to install that one instead, like `ampup install v0.0.42`: ```bash ampup install ``` Then check the version in use: :::terminal ```bash ampd --version ``` ``` ampd v0.0.42 ``` ::: ### Start solo mode ```bash ampd solo ``` It applies the new version's migrations, then starts as usual. :::: ## Upgrade distributed mode In [distributed mode](/running-amp/distributed-mode), turn compaction off and back up the database, then migrate the database once and restart each service: :::steps ### Turn compaction off So that the backup's files stay in storage until you're sure you'll keep the new version. In every worker's configuration. Where the section is already there, change its `active` instead of adding the section again: ```toml [config.toml] [writer.compactor] active = false ``` Or set `AMP_CONFIG_WRITER__COMPACTOR__ACTIVE=false` in its environment. Then restart the workers, on the version you have. A worker that restarts with the same ID resumes its jobs. ### Back up the database With `AMP_CONFIG_METADB__URL` set to the database's URL, and a `pg_dump` at least as new as your PostgreSQL server: ```bash pg_dump --format=custom --file="amp-$(date +%Y%m%d).dump" "$AMP_CONFIG_METADB__URL" ``` ### Install the new version On each machine that runs a service, with `ampup install`, or in your deployment, by changing the tag of Amp's [Docker image](/running-amp/installation#run-with-docker), like `ghcr.io/edgeandnode/amp:v0.0.42`. ### Migrate the metadata database Run the new version's `ampd migrate`, with the same configuration as your services. It applies the migrations the database lacks, then exits: ```bash ampd --config config.toml migrate ``` It exits with status `0` when the database is up to date, including when it already was, and `1` on an error. Services with [`auto_migrate`](/running-amp/metadata-database#auto_migrate) on, the default, would also migrate the database when they restart. Running `ampd migrate` first lets you see a failure while your services still run. ### Restart the services Restart the controller, the query servers, and the workers on the new version, with compaction still off. Do it soon after migrating: services still on the old version keep running, but can't start again, since an older `ampd` refuses the migrated database. ### Check the deployment Check that each service is up, and that the workers are back, as in [Check the deployment](/running-amp/distributed-mode#check-the-deployment): ```bash curl http://localhost:1610/workers ``` ### Turn compaction back on Once you're sure you'll keep the new version, remove both settings from the workers' configuration, and restart them. Compaction then merges the small files that piled up. From then on, the backup can't restore Amp anymore, since garbage collection deletes files it lists. ::: With [Docker Compose](/running-amp/distributed-mode#run-with-docker-compose), once you've backed up, change the image's tag and run `docker compose up -d`: its `migrate` container migrates the database before the services restart. With another orchestrator, like Kubernetes, run `ampd migrate` as a one-off task, like a Kubernetes Job, before you roll out the services. ## Roll back Migrations only go forward, so to go back to the version you had, restore the backup you took before the upgrade. Amp goes back to its state at the time: what changed since, like datasets you registered, is lost. ### Roll back solo mode 1. Stop solo mode. 2. Replace the `.amp` directory with the copy you took before the upgrade, the newest `~/amp-backup-…` directory: ```bash rm -rf .amp && cp -R ~/amp-backup-20261030-154500 .amp ``` 3. With your own PostgreSQL, restore its backup too: recreate the database, then restore the backup into it, as in [Roll back distributed mode](/running-amp/upgrading#roll-back-distributed-mode). 4. Switch back to the version you had, like `ampup use v0.0.41`. 5. Start solo mode. With storage outside `.amp`, the copy's `config.toml` predates turning compaction off, so compaction runs again, unless you turned it off with environment variables: then remove them too. Every file the restored database lists is still in storage. ### Roll back distributed mode You restore the backup into an empty database. Restoring it over the migrated one fails, and can leave the database half restored, with tables of both versions. You need a PostgreSQL administrator's URL to another database on the same server, like `postgres`: ```bash export PG_ADMIN_URL=postgresql://postgres:$POSTGRES_PASSWORD@db.internal:5432/postgres ``` :::steps ### Stop every service The controller, the query servers, and the workers. ### Recreate the database Drop Amp's database, then create it again, empty, owned by the user Amp connects as. Here, both are named `amp`, like in `postgresql://amp:amp@db.internal:5432/amp`: ```bash psql "$PG_ADMIN_URL" -c "DROP DATABASE amp" -c "CREATE DATABASE amp OWNER amp" ``` ### Restore the backup ```bash pg_restore --single-transaction --dbname="$AMP_CONFIG_METADB__URL" amp-20261030.dump ``` It prints nothing when it succeeds. It restores everything or nothing: at the first error, it stops, prints it, and leaves the database empty. Fix the cause, then run it again. Don't start the services until it succeeds: on an empty database, they'd start Amp over, without your state. ### Switch back to the version you had Like `ampup use v0.0.41`, or the tag of the image you had. ### Start the services Remove both settings that turned compaction off from the workers' configuration first: compaction can run again, since every file the restored database lists is still in storage. ::: The restored database doesn't list the files the new version wrote, so Amp doesn't read them, and its jobs extract those blocks again. The files stay in storage: Amp only deletes files it lists. ## Version-specific notes ### v0.0.37: managed PostgreSQL connects as `postgres` Since v0.0.37, the PostgreSQL server that solo mode [manages](/running-amp/solo-mode#managed-postgresql) has a `postgres` user, which solo mode connects as. Before, its only user was named after your login on your machine. Solo mode reuses the database it created before, so after an upgrade from an older version, it doesn't start: ``` Exiting with error: Solo command failed | Caused by: Failed to connect to metadata database -> Error connecting to metadata db: error returned from database: role "postgres" does not exist at line 710 ``` This only affects solo mode with managed PostgreSQL. To fix it, either: * **Start over**, which deletes Amp's state, like registered datasets and jobs. Solo mode creates a new database on its next start: ```bash rm -rf .amp/metadb ``` * **Keep your state** by adding the `postgres` user to the database, with PostgreSQL's own programs, while `ampd` is stopped: ```bash DATADIR=$PWD/.amp/metadb pg_ctl -D "$DATADIR" -o "-k $DATADIR -h ''" -w start psql "host=$DATADIR dbname=postgres" -c "CREATE ROLE postgres SUPERUSER LOGIN;" pg_ctl -D "$DATADIR" -w stop ``` ## Troubleshooting The errors you can get when a service connects to a database of another version, or when a migration fails, are in [Metadata Database](/running-amp/metadata-database#troubleshooting). ### `permission denied for schema public` ``` Exiting with error: Migrate command failed | Caused by: Failed to run migrations -> Error running migrations -> while executing migrations: error returned from database: permission denied for schema public at line 2812 -> error returned from database: permission denied for schema public at line 2812 -> permission denied for schema public at line 2812 ``` The database user can't create tables. Make it the owner of the database, as a PostgreSQL administrator, then migrate again: ```sql ALTER DATABASE amp OWNER TO amp; ``` ### `invalid filesystem path` ``` Exiting with error: Failed to load config | Caused by: Invalid object store URL at /etc/amp/config.toml -> invalid filesystem path: /etc/amp/data -> failed to canonicalize path `/etc/amp/data`: No such file or directory (os error 2) ``` `ampd migrate` checks that the [storage](/running-amp/storage) directories in its configuration exist, like the services do, even though it doesn't use them. Create them, or run it where they exist, like in a container that mounts them.