> **Can't find what you're looking for?** Use `search_docs` on the docs MCP server at `https://docs.ampdb.io/api/mcp` to find what you need. # ampctl trace `ampctl trace` shows where the time goes when Amp runs a query or a [job](/managing-amp/jobs). Amp records each one as a trace: a tree of spans, each a step of the work, like looking up a table's files, fetching a block from a [provider](/providers), or writing a file, with when it started and how long it took. Amp exports its traces with [OpenTelemetry](/running-amp/telemetry), and `ampctl trace` reads them back from a Jaeger-compatible API, like [Jaeger](https://www.jaegertracing.io) or [VictoriaTraces](https://docs.victoriametrics.com/victoriatraces/). Its reports turn traces into flamegraphs: charts where each span is a box as wide as the time it took, stacked on the span that started it. Use them when a query is slower than you expect, or a job extracts or transforms blocks slower than you need. They tell you whether the time goes into reading data, looking up files, waiting for the provider, or writing files, and so what to change. ## Prerequisites * A running Amp that exports its traces to Jaeger or another Jaeger-compatible API. [Telemetry](/running-amp/telemetry) explains how: you start `ampd` with `OTEL_EXPORTER_OTLP_ENDPOINT` set to the address that receives them. * `ampctl`, [installed](/clis/ampctl#install). `ampctl trace` reads from Jaeger, not from the Admin API, so it doesn't need to [connect to Amp](/clis/ampctl#connect-to-amp), and takes none of the [global flags](/clis/ampctl#global-flags). The examples report on the [Quickstart](/get-started/quickstart)'s `acme/eth_mainnet@1.0.0` dataset, on [`acme/eth_transfers@1.0.0`](/datasets/derived-datasets#the-erc-20-transfers-dataset), a derived dataset you create on top of it, and on a query of the second. ## Connect to Jaeger `ampctl trace` reads from the Jaeger API at `http://localhost:16686`, where Jaeger serves it, with its UI, by default. To read from another, set `JAEGER_URL` for every command in your shell, or pass `--jaeger-url` to one command, which takes precedence: ```bash export JAEGER_URL=http://jaeger.internal:16686 ``` For VictoriaTraces, include the path of its Jaeger API, like `https://traces.example.com/select/jaeger`. [Telemetry](/running-amp/telemetry#export-to-victoriatraces) shows how to export Amp's traces to it. `ampctl trace` reads Jaeger's HTTP API, which Jaeger 2.20 and earlier serve, but not Jaeger 2.21 and later: with those, every command fails with `✗ Jaeger API returned 404 Not Found: 404 page not found`. Run Jaeger 2.20, like `jaegertracing/jaeger:2.20.0`, or VictoriaTraces. If the API asks for a username and a password (HTTP basic authentication), set `JAEGER_AUTH` to `user:password`. There's no flag for it: ```bash export JAEGER_AUTH="amp:$JAEGER_PASSWORD" ``` Without it, commands fail with `✗ Jaeger API returned 401 Unauthorized:`. A value without a colon fails before anything is sent, with a message that names a `--basic-auth` flag that doesn't exist; it means `JAEGER_AUTH`: ``` ✗ --basic-auth must be user:password ``` When nothing answers at the URL, commands fail with [exit code](/clis/ampctl#exit-codes) 1: ``` ✗ searching http://localhost:16686/api/traces?service=ampd.server&limit=20 → error sending request for url (http://localhost:16686/api/traces?service=ampd.server&limit=20) → client error (Connect) → tcp connect error → Connection refused (os error 61) ``` Check that Jaeger is running, and that the URL and port are its API's. When Jaeger answers with an error, commands fail with its status and response, like `✗ Jaeger API returned 404 Not Found: …`. ## Where Amp's traces are Amp exports each trace under a service, named after the `ampd` command that runs the work: | Command | Service | | -------------------------------------------- | ----------------- | | [`ampd solo`](/running-amp/solo-mode) | `ampd.solo` | | [`ampd server`](/running-amp/query-server) | `ampd.server` | | [`ampd worker`](/running-amp/worker) | `ampd.worker` | | [`ampd controller`](/running-amp/controller) | `ampd.controller` | | `ampd migrate` | `ampd.migrate` | `ampctl trace` looks in the services of a [distributed deployment](/running-amp/distributed-mode): `ampd.server`, `ampd.worker`, and `ampd.controller`. Reports never look in `ampd.solo`, so with solo mode, like the Quickstart's, they find nothing: ``` ✗ no traces found matching filters ``` Instead, find the trace with [`ampctl trace search --service ampd.solo`](/clis/ampctl/trace#ampctl-trace-search), save it with [`ampctl trace fetch`](/clis/ampctl/trace#ampctl-trace-fetch), and report on the file with `--file`. Each report's section below shows both ways. ## Filters `ampctl trace search` and the reports take the same flags to pick traces: `--filter` on their tags, and `--after` and `--before` on their times. ### Tags Spans carry tags, which say what they worked on. These identify a query or a job: | Span | Tag | Value | | ----------- | -------------------------- | ---------------------------------------------------------------------------------------------------------------------------- | | `do_get` | `request_id` | The request's ID: the value of its `cf-ray` header, which Cloudflare adds, or else a random one. On the outer `do_get` span. | | `do_get` | `query` | The query's SQL, exactly as the query server received it. On the inner `do_get` span. | | `execute` | `job_id` | The job's ID, like `2`. Each attempt of a job is a trace of its own, with its `execute` span at the root. | | `execute` | `dataset` | The dataset and its manifest hash, like `acme/eth_mainnet@b9e2ebd92c0752ba8a61a233f9cc5d3f4f8b601b1b789a882a5dd8ece9bca56b`. | | `run_range` | `start_block`, `end_block` | The first and last block of the range a raw dataset's job extracted, like `24000000`. | | `run_range` | `job_id` | The ID of the job that extracted it, as on its `execute` span. | `--filter key=value` keeps the traces with a span whose tag `key` is exactly `value`. Repeat it to require several tags. Jaeger and VictoriaTraces match all of them, and the `--operation` of a search, against one span at a time, so filter on a tag of the span you search for. `--operation execute` with a `--filter` on `dataset` finds a job's traces, but `--operation run_range` with the same filter finds none, because `dataset` is on the job's `execute` span, not on `run_range`. A query has two `do_get` spans, one inside the other, and each has one of its two tags. So a search that filters on both `query` and `request_id` prints `No traces found.`, even for that query. Filter on one of them. The same goes for reports, which search for the spans they start from: their `--filter` works with `query` or `request_id` for queries, and `job_id`, `start_block`, and `end_block` for raw datasets. The spans a derived dataset's report starts from have no `job_id`, so the `--filter job_id=…` of `ampctl trace --help`'s example finds nothing; to report on one derived dataset's job, [find its trace](/clis/ampctl/trace#ampctl-trace-search) by its `execute` span, and report on the file. A filter without `=` fails with exit code 1. ### Times `--after` and `--before` keep the traces with spans that started after, or before, a time. They take either: * A time relative to now: a number followed by `s`, `m`, `h`, or `d`, like `30m` for 30 minutes ago. * A date and time in ISO 8601, like `2026-10-30T21:00:00Z` or `2026-10-30T17:00:00-04:00`. Without an offset, it's read as UTC. `--before` only applies with `--after`; alone, it's ignored. Without `--after`, Jaeger decides how far back to look. Any other value fails with exit code 1. ## `ampctl trace report` Writes a report on queries or jobs: two flamegraphs, and the data behind them. ``` ampctl trace report query [OPTIONS] ampctl trace report raw-dataset [OPTIONS] ampctl trace report derived-dataset [OPTIONS] ``` Each subcommand reports on one kind of work. It searches Jaeger for the spans that do that work, or reads them from a file, and keeps them and the spans they started. It leaves out the rest, like a job's setup and the time it waits between two rounds of work, so a report adds up the time the work took: | Subcommand | Reports on | Spans it starts from | Services it searches | | --------------------------------------------------------- | ------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------- | --------------------------------------------------- | | [`query`](/clis/ampctl/trace#ampctl-trace-report-query) | Queries | `do_get` | `ampd.server` | | [`raw-dataset`](/clis/ampctl/trace#ampctl-trace-report-raw-dataset) | Extracting [raw datasets](/datasets/raw-datasets) | `run_range` | `ampd.worker` | | [`derived-dataset`](/clis/ampctl/trace#ampctl-trace-report-derived-dataset) | Materializing [derived datasets](/datasets/derived-datasets) | `execute_microbatch`, `next_microbatch_range`, `write`, `close`, `register`, and `send_location_change_notif` | `ampd.worker`, and `ampd.controller` for `register` | The three take the same flags: | Flag | Type | Default | Description | | --------------------------------- | ------------------ | ------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `--file ` | path | none | Reads the spans from a file that [`ampctl trace fetch`](/clis/ampctl/trace#ampctl-trace-fetch) or a report saved, instead of searching Jaeger, like `--file eth_mainnet.json.gz`. A name ending in `.gz` is read as gzipped. It can't be combined with `--filter`, `--after`, or `--before`. | | `--filter ` | string, repeatable | none | Keeps only the spans with this [tag](/clis/ampctl/trace#tags), like `--filter "query=$QUERY"`. | | `--after ` | [time](/clis/ampctl/trace#times) | none | Keeps only the traces with spans that started after this time, like `--after 1h`. | | `--before ` | [time](/clis/ampctl/trace#times) | none | Keeps only the traces with spans that started before this time, like `--before 2026-10-30T21:00:00Z`. Only applies with `--after`. | | `--limit ` | integer | `20` | How many traces to read for each kind of span it starts from, the most recent ones, like `--limit 1`. | | `-o`, `--output-dir ` | path | `.` | The directory to write the report to, created if it doesn't exist, like `-o query_report`. | | `--jaeger-url ` | URL | `$JAEGER_URL`, or `http://localhost:16686` | The Jaeger API to read from. See [Connect to Jaeger](/clis/ampctl/trace#connect-to-jaeger). | Without `--file`, a report merges every trace it finds into one, so a report on the last hour of queries adds up all of them. Narrow it with `--filter`, `--after`, and `--limit` to the query or the stretch of a job you want to see. ### The report's files A report writes four files, named after its subcommand (`query`, `raw_dataset`, or `derived_dataset`), and replaces those it wrote there before: | File | What it holds | | --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `query_wallclock.svg` | The wall-clock flamegraph: how long each span took from start to end, waiting included. | | `query_busy.svg` | The busy-time flamegraph: how long each span actually worked, without the time it spent waiting, like for the network, the disk, or other spans. | | `query_folded.txt` | The wall-clock flamegraph's data, as folded stacks: one line per stack of spans, then the microseconds its top span took by itself. Tools like [speedscope](https://www.speedscope.app) read it. | | `query_trace.json.gz` | The spans the report kept, in Jaeger's JSON format, gzipped. Pass it to `--file` to report on them again. | ### Read a report Open the flamegraphs in a browser. The bottom box, `all`, is the total time. Each box above it is a span, sitting on the span that started it, as wide as its share of the time. Hover over a box to see its time in microseconds (μs) and its percentage of the total, click it to zoom in, and use Search, at the top right, to highlight the spans whose names match. The colors don't mean anything. The widest boxes are where the time goes. Compare the two flamegraphs to tell what kind of time it is: * **About as wide in both**: the span spent its time working, like decoding blocks or aggregating rows. Give it less to do, like a query that reads fewer blocks or columns, or give Amp more CPU. * **Wide in the wall-clock flamegraph, narrow in the busy-time one**: the span spent its time waiting, for the provider, for [storage](/running-amp/storage), or for the [metadata database](/running-amp/metadata-database). Look at what it waits for. The wall-clock flamegraph adds up spans that run at the same time, so its total can be longer than the work took: in a raw dataset's report, each table's writer runs during the whole extraction, alongside the fetching. A span's time by itself is its time minus that of the spans it started. To list the spans that took the most time by themselves, add up the last span of each line of the folded stacks: :::terminal ```bash awk -F';' '{n = $NF; sub(/.* /, "", n); sub(/ [0-9]+$/, "", $NF); t[$NF] += n} END {for (s in t) print t[s], s}' query_report/query_folded.txt | sort -rn | head -5 ``` ``` 3176 resolve_batch_snapshot 2618 DataSourceExec 1863 execute_query 1205 RepartitionExec 1092 execute_plan ``` ::: Span names come from Amp's code, but most say what they do. Each report's section below describes the ones you'll see most. ### Errors * `✗ no traces found matching filters`: Jaeger has nothing that matches in the services the report searches. With solo mode, report on a file instead (see [Where Amp's traces are](/clis/ampctl/trace#where-amps-traces-are)). Otherwise, widen `--after`, and check that you filter on a [tag the report's spans have](/clis/ampctl/trace#tags), and not on both `query` and `request_id`. * `✗ no root spans found in trace`, after `Filtered to 0 spans`: the file holds none of the spans this report starts from, like a query's trace passed to `ampctl trace report raw-dataset`. Use the subcommand for the work it holds. * `error: the argument '--file ' cannot be used with '--after '`: a file's spans can't be filtered. Filter when you search for the trace instead. This exits with code 2. * `thread 'main' (6908454) panicked at bin/ampctl/src/cmd/trace/flamegraph.rs:193:43: end byte index 60 is not a char boundary`: `ampctl trace report query` labels each query with its first 60 bytes, and fails when they end inside a character that takes several bytes, like `é`. It has written the JSON file, but no flamegraph. Report on a version of the query without that character in its first 60. ## `ampctl trace report query` Reports on how the query server ran queries: finding the files a query reads, planning it, and running its plan. ``` ampctl trace report query [OPTIONS] ``` It takes the [report flags](/clis/ampctl/trace#ampctl-trace-report), and starts from the query server's `do_get` spans, one for each query it runs. To report on one query, filter on its SQL: the `query` tag holds it exactly as the query server received it. Keep the query in a variable, so you run and filter on the same text: ```bash QUERY='SELECT evm_decode_hex(token) AS token, count(*) AS transfers FROM "acme/eth_transfers@1.0.0".transfers GROUP BY token ORDER BY transfers DESC LIMIT 5' ampsql "$QUERY" ``` When Amp runs its own [query server](/running-amp/query-server), Jaeger has the query's trace under `ampd.server`, and the report finds it: :::terminal ```bash ampctl trace report query --filter "query=$QUERY" -o query_report ``` ``` Fetched 64 spans from Jaeger Filtered to 63 spans (roots: ["do_get"]) Wrote trace to query_report/query_trace.json.gz Wrote wallclock flamegraph to query_report/query_wallclock.svg Wrote busy-time flamegraph to query_report/query_busy.svg Wrote folded stacks to query_report/query_folded.txt ``` ::: It fetched every span of the query's trace, and kept the `do_get` spans and those they started. Each time you run the query, Jaeger gets another trace that matches, and the report adds them up. Add `--limit 1` to report on the latest only. With solo mode, search `ampd.solo` for the trace, save it, and report on the file: :::terminal ```bash ampctl trace search --service ampd.solo --operation do_get --filter "query=$QUERY" --limit 1 ``` ``` 293f144ff185204451ddc15da908cbcb | 2026-10-30 21:23:50 UTC | 14.3ms | 64 spans | http.server.request ``` ::: ```bash TRACE_ID=$(ampctl trace search --service ampd.solo --operation do_get --filter "query=$QUERY" --limit 1 | cut -d' ' -f1) ampctl trace fetch --trace-id "$TRACE_ID" -o top_tokens.json.gz ampctl trace report query --file top_tokens.json.gz -o query_report ``` The flamegraphs show the query as these spans: * `do_get (SELECT evm_decode_hex(token) AS token, count(*) AS transfers)`: the query, labeled with its first 60 characters. `execute_query`, on it, is all of its work. * `resolve_batch_snapshot`, and the spans on it like `attach`: finding, in the metadata database, the [table revisions](/managing-amp/table-revisions) and the files that hold the blocks the query reads. * `optimize`: planning the query. * `execute_plan`, and the spans on it whose names end in `Exec`: running the query's plan, one span for each of its steps. `DataSourceExec` reads the tables' Parquet files, `AggregateExec` groups rows, `SortExec` sorts them, and `RepartitionExec` spreads them across threads. This query took 14 ms: 4.4 ms to find its files, and 7.3 ms to run its plan, 3 ms of which to read the files. Its two flamegraphs are about as wide, so it spent little time waiting. When a query is slow: * **Most of its time is in `DataSourceExec`**: it reads more data than it needs. Filter on `block_num` to read fewer blocks, and select only the columns you need. See [SQL](/querying/sql). * **Most of its time is in finding its files**: the table may have many small files, which a job writes as it follows the chain. [Compaction](/running-amp/storage/compaction) merges them. Compare the busy-time flamegraph to tell whether the metadata database is slow to answer. A query whose SQL spans several lines keeps its line breaks in its label, which splits its lines in `query_folded.txt`: tools that read the file then misread them. The flamegraphs aren't affected. ## `ampctl trace report raw-dataset` Reports on how workers extracted a raw dataset: fetching blocks from the provider, turning them into rows, and writing the rows to files. ``` ampctl trace report raw-dataset [OPTIONS] ``` It takes the [report flags](/clis/ampctl/trace#ampctl-trace-report), and starts from the worker's `run_range` spans, one for each range of blocks a job extracts. When Amp runs its own [workers](/running-amp/worker), Jaeger has their spans under `ampd.worker`, and `ampctl trace report raw-dataset --after 1h -o raw_report` finds those of the last hour. With solo mode, search `ampd.solo` for the traces that extracted a range: :::terminal ```bash ampctl trace search --service ampd.solo --operation run_range --after 1h ``` ``` d234225d0760da8ad05f8e0408b2ad15 | 2026-10-30 21:13:59 UTC | 30981.1ms | 621 spans | execute 73f4abd85745b8a89b0cb394c8d5d0c2 | 2026-10-30 21:12:56 UTC | 1831.9ms | 45 spans | execute ``` ::: Each attempt of a job is a trace of its own. Here, an attempt failed after 1.8 seconds, and the job [retried](/managing-amp/jobs/retries): the trace with 621 spans is the one that extracted the blocks. To list the traces of one job, search for its `execute` span by the ID of its `materialize-raw` job, which [`ampctl job list --status all`](/clis/ampctl/job) shows: ```bash ampctl trace search --service ampd.solo --operation execute --filter job_id=2 ``` Save the trace, and report on it. `--limit 1` keeps the latest attempt, the one that extracted the blocks here: ```bash TRACE_ID=$(ampctl trace search --service ampd.solo --operation execute --filter job_id=2 --limit 1 | cut -d' ' -f1) ampctl trace fetch --trace-id "$TRACE_ID" -o eth_mainnet.json.gz ampctl trace report raw-dataset --file eth_mainnet.json.gz -o raw_report ``` The flamegraphs show the extraction as these spans: * `run_range [24000000..24000099]`: the range of blocks, with its first and last block. * `fetch_block`: waiting for the provider to return a block, one span per block. * `process_block`: turning a block into rows. * `table_writer`: writing a table's rows to Parquet files, one for each of the dataset's tables. `write`, `close`, and `finalize_parquet`, on it, write and finish each file. In the wall-clock flamegraph, this job spent 30.5 seconds in `fetch_block`, and its three table writers ran for those 30.5 seconds too. In the busy-time one, the whole range took 63 ms of work: 15 ms to turn blocks into rows, 47 ms to write files, and less than 1 ms to fetch blocks. So the job spent its time waiting for the provider. When `fetch_block` takes most of the wall-clock time: * Let the job fetch more blocks at once, as far as the provider's rate limits allow. An `evm-rpc` provider fetches as many as its `concurrent_request_limit`, up to eight ([EVM RPC](/providers/evm-rpc)). For a Solana dataset, deploy it with a higher `--parallelism` ([`ampctl dataset deploy`](/clis/ampctl/dataset#--parallelism-parallelism)). * Use a faster provider, or one closer to Amp. See [Providers](/providers). When `table_writer` is wide in the busy-time flamegraph too, writing files is what's slow: check the [storage](/running-amp/storage) the dataset writes to. ## `ampctl trace report derived-dataset` Reports on how workers materialized a derived dataset: running its SQL over each new range of blocks, and writing and registering the files. ``` ampctl trace report derived-dataset [OPTIONS] ``` It takes the [report flags](/clis/ampctl/trace#ampctl-trace-report). A derived dataset's job runs its SQL in rounds, each over the blocks its dependencies added since the last, called a microbatch. The report starts from the spans each round ends: `execute_microbatch` and `next_microbatch_range` on the worker, `write` and `close` for each file, `register`, on the worker and on the [controller](/running-amp/controller), and `send_location_change_notif` on the worker. A job that follows its dependencies' new blocks never ends, and neither does its trace. The report only needs the spans each round ends, so when Amp runs its own workers, `ampctl trace report derived-dataset --after 10m` reports on the rounds of the last 10 minutes. With solo mode, search `ampd.solo` for the traces with a microbatch: :::terminal ```bash ampctl trace search --service ampd.solo --operation execute_microbatch --after 1h ``` ``` d87edaddd663ea6128f14d2f51c6c10a | 2026-10-30 21:11:00 UTC | 210028.4ms | 144 spans | execute ``` ::: This job ran for 3.5 minutes, most of them waiting for `acme/eth_mainnet@1.0.0` to have the blocks it transforms. The report leaves the waiting out. Save the trace, and report on it: ```bash TRACE_ID=$(ampctl trace search --service ampd.solo --operation execute_microbatch --after 1h --limit 1 | cut -d' ' -f1) ampctl trace fetch --trace-id "$TRACE_ID" -o eth_transfers.json.gz ampctl trace report derived-dataset --file eth_transfers.json.gz -o derived_report ``` The flamegraphs show the job's work as these spans: * `execute_microbatch`: running the dataset's SQL over one microbatch. On it, `execute_plan` and the spans whose names end in `Exec` are the steps of its plan, as in a [query's report](/clis/ampctl/trace#ampctl-trace-report-query). * `next_microbatch_range`: choosing the blocks of the next microbatch. * `write` and `close`: writing a Parquet file, and finishing it. * `register`: recording a file in the metadata database, so queries read it. `send_location_change_notif`, on it, tells the query server that the table has a new file. This job's work took 45 ms: 33 ms running its SQL, 29 ms of which reading `acme/eth_mainnet@1.0.0`'s files, and 6 ms registering files, which took less than 1 ms in the busy-time flamegraph. When a derived dataset falls behind: * **Most of the time is in `execute_microbatch`**: its SQL is slow. Read it as a query's report, and make the SQL read less. See [Derived Datasets](/datasets/derived-datasets). * **Most of the time is in `write`, `close`, or `register`, but not in the busy-time flamegraph**: the job waits for [storage](/running-amp/storage) or the metadata database. ## `ampctl trace search` Lists recent traces, to find the one to save or report on. ``` ampctl trace search [OPTIONS] ``` | Flag | Type | Default | Description | | --------------------------- | ------------------ | ----------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------ | | `--operation ` | string | none | Lists only traces with a span of this name, like `--operation do_get`. | | `--service ` | string, repeatable | `ampd.server`, `ampd.worker`, `ampd.controller` | The [service](/clis/ampctl/trace#where-amps-traces-are) to search, like `--service ampd.solo`. Repeat it to search several. | | `--filter ` | string, repeatable | none | Lists only traces with a span that has this [tag](/clis/ampctl/trace#tags), like `--filter job_id=2`. | | `--after ` | [time](/clis/ampctl/trace#times) | none | Lists only traces with spans that started after this time, like `--after 1h`. | | `--before ` | [time](/clis/ampctl/trace#times) | none | Lists only traces with spans that started before this time, like `--before 2026-10-30T21:00:00Z`. Only applies with `--after`. | | `--limit ` | integer | `20` | How many traces to list for each service, the most recent ones, like `--limit 1`. | | `--jaeger-url ` | URL | `$JAEGER_URL`, or `http://localhost:16686` | The Jaeger API to read from. See [Connect to Jaeger](/clis/ampctl/trace#connect-to-jaeger). | To list the traces of the query in [the query report's example](/clis/ampctl/trace#ampctl-trace-report-query), with `QUERY` set as there: :::terminal ```bash ampctl trace search --service ampd.solo --operation do_get --filter "query=$QUERY" --after 1h ``` ``` c40971fbd6291b32a0d21c7198b6c033 | 2026-10-30 21:14:49 UTC | 14.8ms | 64 spans | http.server.request 293f144ff185204451ddc15da908cbcb | 2026-10-30 21:23:50 UTC | 14.3ms | 64 spans | http.server.request 63e2f340ca779e312ddff50d877a609d | 2026-10-30 21:23:27 UTC | 23.1ms | 69 spans | http.server.request ``` ::: Each line is a trace, here one for each time the [query](/clis/ampctl/trace#ampctl-trace-report-query) ran in the last hour: its ID, the time its first span started (in UTC), how long that span took, how many spans the trace has, and the first span's name. The lines aren't always in order of time. A query's first span is `http.server.request`, and a job's is `execute`. Running a query takes two requests, so it makes two traces: one to plan it, and one to run it, which has the `do_get` spans that query reports start from. Without `--operation`, a search also lists the traces of Amp's own background work, like each worker's heartbeat (`update_heartbeat`). When nothing matches, it prints `No traces found.`, and exits with code 0. ## `ampctl trace fetch` Saves a trace from Jaeger to a file, to report on it, or to keep it after Jaeger has deleted it. ``` ampctl trace fetch [OPTIONS] --trace-id TRACE_ID --output OUTPUT ``` | Flag | Type | Default | Description | | --------------------------- | ------ | ------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------- | | `--trace-id ` | string | required | The trace's ID, as [`ampctl trace search`](/clis/ampctl/trace#ampctl-trace-search) prints it, like `--trace-id d234225d0760da8ad05f8e0408b2ad15`. | | `-o`, `--output ` | path | required | The file to write, like `-o eth_mainnet.json.gz`. A name ending in `.gz` is gzipped. Its directory must exist. | | `--jaeger-url ` | URL | `$JAEGER_URL`, or `http://localhost:16686` | The Jaeger API to read from. See [Connect to Jaeger](/clis/ampctl/trace#connect-to-jaeger). | ```bash ampctl trace fetch --trace-id d234225d0760da8ad05f8e0408b2ad15 -o eth_mainnet.json.gz ``` It saves every span of the trace, in Jaeger's JSON format, and replaces the file if it exists. A report keeps only the spans it needs, and saves them the same way. ### Errors * `✗ Jaeger API returned 404 Not Found: {"data":null,"total":0,"limit":0,"offset":0,"errors":[{"code":404,"msg":"trace not found"}]}`: Jaeger has no trace with that ID. Check the ID, and that Jaeger still keeps the trace. VictoriaTraces answers with a similar body. * `✗ No such file or directory (os error 2)`: the output file's directory doesn't exist. Create it first.