> **Can't find what you're looking for?** Use `search_docs` on the docs MCP server at `https://docs.ampdb.io/api/mcp` to find what you need. # Telemetry Amp reports what it's doing in three ways, so you can tell whether it's healthy and find out why when it isn't: * **[Logs](/running-amp/telemetry#logs)**: each event, like a job starting or a request failing, as a line of text. Every service writes them. * **[Traces](/running-amp/telemetry#traces)**: how long each operation took, like a query or a job's step, and the operations it was made of. * **[Metrics](/running-amp/telemetry#metrics)**: counts and measurements over time, like queries per second or rows extracted. Amp sends traces and metrics with [OpenTelemetry](https://opentelemetry.io), the standard most observability tools accept, like Grafana, Datadog, and Honeycomb. It only sends them when you [tell it where](/running-amp/telemetry#traces-and-metrics). ## Logs Each service writes its logs to its standard error. A line has the time, in UTC, the level, the operation it happened in, the part of Amp it comes from, and the message, followed by its fields: ``` 2026-10-30T20:40:54.508569Z INFO ampd::solo_cmd: Controller Admin API running at 0.0.0.0:1610 2026-10-30T20:43:18.161151Z ERROR connect_pool_with_config:run_migrations: amp_metadb::db::conn: error=Error running migrations ``` When standard error is a terminal, the levels are in color. To keep logs, collect them from standard error, like `journald` and Kubernetes do, or redirect it to a file: `ampd solo 2>> ampd.log`. ### Log levels `AMP_LOG` sets how much Amp logs. From the least to the most: | Level | What Amp logs | | ------- | ------------------------------------------------------------------------------------ | | `error` | Failures: a request, a job, or a service that failed. | | `warn` | Failures, and problems Amp works around, like a request it retries. | | `info` | The default. Warnings, and what each service does: starting, running jobs, and more. | | `debug` | Details, for investigating a problem. | | `trace` | Every detail. Very verbose. | ```bash AMP_LOG=debug ampd solo ``` `AMP_LOG` applies to Amp's own logs. The libraries Amp uses, like its PostgreSQL driver, only log errors. To set the level of one part of Amp, or of a library, set `RUST_LOG` to that part's name, as it appears in its log lines, then `=` and the level. Separate several with commas. They override `AMP_LOG` for those parts: ```bash # Amp at info, its metadata database code at debug, and the sqlx library at warn RUST_LOG=amp_metadb=debug,sqlx=warn ampd solo ``` ### JSON logs For log collectors that parse JSON, like Loki, Elasticsearch, or Datadog, write each line as a JSON object: ```toml [config.toml] [logging] json = true ``` Or set `AMP_CONFIG_LOGGING__JSON=true`. Each line is then: ``` {"timestamp":"2026-10-30T20:42:38.644243Z","level":"INFO","fields":{"message":"Running migrations on metadata database..."},"target":"amp_metadb","span":{"name":"connect_pool_with_config"},"spans":[{"name":"connect_pool_with_config"}]} ``` | Field | Description | | ----------- | -------------------------------------------------------------- | | `timestamp` | When it happened, in UTC. | | `level` | `ERROR`, `WARN`, `INFO`, `DEBUG`, or `TRACE`. | | `fields` | The message, in `message`, and the event's other fields. | | `target` | The part of Amp the line comes from, the name `RUST_LOG` uses. | | `span` | The operation it happened in, with its fields. | | `spans` | Every operation it happened in, from the outermost. | ### Operation timings To log how long each operation took, set `AMP_LOG_SPAN_EVENT`: * `close` logs a line when an operation ends, with the time it spent working (`time.busy`) and waiting (`time.idle`). * `full` also logs when each operation starts, and each time it resumes and pauses. :::terminal ```bash AMP_LOG_SPAN_EVENT=close ampd --config config.toml migrate ``` ``` 2026-10-30T20:42:38.788013Z INFO connect_pool_with_config:connect: amp_metadb::db::conn: close time.busy=4.32ms time.idle=18.3ms 2026-10-30T20:42:38.788052Z INFO connect_pool_with_config: amp_metadb: Running migrations on metadata database... 2026-10-30T20:42:38.798839Z INFO connect_pool_with_config:run_migrations: amp_metadb::db::conn: close time.busy=1.02ms time.idle=9.74ms 2026-10-30T20:42:38.798859Z INFO connect_pool_with_config: amp_metadb: Migrations completed successfully ``` ::: It makes the logs much longer. To follow operations across services, [export traces](/running-amp/telemetry#traces) instead. ## Traces and metrics Amp exports traces and metrics with OTLP, OpenTelemetry's protocol, to a backend that accepts it, or to an [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/) that forwards them. It doesn't export logs: [collect them](/running-amp/telemetry#logs) from standard error. It's off until you set where to send them, with OpenTelemetry's standard environment variables. Each service reads them when it starts: ```bash export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 ``` ### Try it locally Grafana's [`otel-lgtm`](https://github.com/grafana/docker-otel-lgtm) image runs a complete observability stack in one container: an OTLP collector, Prometheus for metrics, Tempo for traces, and Grafana to explore them. You need Docker, and Amp [installed](/running-amp/installation) with [solo mode's prerequisites](/running-amp/solo-mode#prerequisites). ::::steps #### Start the stack ```bash docker run -d --name lgtm -p 3000:3000 -p 4317:4317 -p 4318:4318 -p 9090:9090 grafana/otel-lgtm ``` It listens for OTLP on ports `4317` (gRPC) and `4318` (HTTP), serves Grafana on port `3000`, and Prometheus on port `9090`. #### Start Amp ```bash export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 export OTEL_METRIC_EXPORT_INTERVAL=5000 # every 5 seconds, rather than 60 ampd solo ``` #### Run a query In another terminal, one without the `OTEL_*` variables, since `ampsql` [hangs](/running-amp/telemetry#ampsql-hangs) with them: ```bash ampsql "SELECT 1" ``` #### See its metrics After a few seconds, Prometheus has the query in `query_count_total`: :::terminal ```bash curl -s 'http://localhost:9090/api/v1/query?query=query_count_total' ``` ```json { "status": "success", "data": { "resultType": "vector", "result": [ { "metric": { "__name__": "query_count_total", "job": "amp/ampd.solo", "service_name": "ampd.solo", "service_namespace": "amp", "service_version": "v0.0.42" }, "value": [1793393112.423, "1"] } ] } } ``` ::: #### Explore in Grafana Open [http://localhost:3000](http://localhost\:3000), then **Explore**. Choose **Prometheus** for metrics, or **Tempo** for traces, and search for the service `ampd.solo`. :::: ### Environment variables Giving any of these `OTEL_EXPORTER_OTLP_*` variables a value turns exporting on: | Variable | Default | Description | | ------------------------------------------------ | ----------------------- | ------------------------------------------------------------------------------------------------------------------------ | | `OTEL_EXPORTER_OTLP_ENDPOINT` | `http://localhost:4318` | Where to export, `http://localhost:4317` by default with gRPC. Over HTTP, Amp adds `/v1/traces` and `/v1/metrics` to it. | | `OTEL_EXPORTER_OTLP_PROTOCOL` | `http/protobuf` | `http/protobuf` or `http/json`, usually on port `4318`, or `grpc`, usually on port `4317`. | | `OTEL_EXPORTER_OTLP_HEADERS` | none | Headers to send with each export, like a backend's API key: `x-api-key=$API_KEY`. Separate several with commas. | | `OTEL_EXPORTER_OTLP_TIMEOUT` | `10000` | How long an export may take, in milliseconds. | | `OTEL_RESOURCE_ATTRIBUTES` | none | Attributes to add to everything a service exports. See [Resource attributes](/running-amp/telemetry#resource-attributes). | | `OTEL_TRACES_SAMPLER`, `OTEL_TRACES_SAMPLER_ARG` | `parentbased_always_on` | Which traces to export. See [Sampling](/running-amp/telemetry#sampling). | | `OTEL_METRIC_EXPORT_INTERVAL` | `60000` | How often to export metrics, in milliseconds. | The first four apply to traces and metrics. To set one for traces or metrics only, add `TRACES_` or `METRICS_` after `OTLP_`, like `OTEL_EXPORTER_OTLP_METRICS_HEADERS`: it overrides the shared one. A signal's own endpoint is a full URL, like `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=http://localhost:4318/v1/traces`. Setting only one signal's variables exports only that signal. Amp doesn't compress exports: leave `OTEL_EXPORTER_OTLP_COMPRESSION` unset, or the service doesn't start. ### Resource attributes Everything a service exports carries attributes that say where it comes from: | Attribute | Value | | --------------------- | ------------------------------------------------------------------------------- | | `service.namespace` | `amp` | | `service.name` | The command's [service name](/running-amp/telemetry#service-names), like `ampd.server` | | `service.version` | The version of `ampd`, like `v0.0.42` | | `service.instance.id` | Workers only: the worker's [ID](/running-amp/worker#node-ids), like `worker-01` | Add your own with `OTEL_RESOURCE_ATTRIBUTES`, like the environment or the region. Amp's values win over the same attributes set there, and over `OTEL_SERVICE_NAME`. Backends turn them into labels. Prometheus, for example, has `job="amp/ampd.server"` and `service_name="ampd.server"`. ### Service names Each `ampd` command exports under its own service name, which is how you find a service's traces and metrics in a backend: | Command | Service name | | ----------------------------------------------------------- | ----------------- | | [`ampd solo`](/running-amp/solo-mode) | `ampd.solo` | | [`ampd server`](/running-amp/query-server) | `ampd.server` | | [`ampd worker`](/running-amp/worker) | `ampd.worker` | | [`ampd controller`](/running-amp/controller) | `ampd.controller` | | [`ampd migrate`](/running-amp/metadata-database#migrations) | `ampd.migrate` | Solo mode exports everything under `ampd.solo`, whether its query server, its controller, or its worker did the work. ### Traces A trace follows one operation, like a query, a request to the [Admin API](/managing-amp/admin-api), or a step of a job, through each service it involves. Amp traces the operations it logs at its [log level](/running-amp/telemetry#log-levels), so lowering `AMP_LOG` to `warn` drops most of them. When a client sends the W3C `traceparent` header with a query or an Admin API request, Amp continues the client's trace rather than starting its own, so you can follow a request from your application into Amp. To turn traces into flame graphs of a query or a job, use [`ampctl trace`](/clis/ampctl/trace) with a Jaeger-compatible backend. It searches `ampd.server`, `ampd.worker`, and `ampd.controller`, not `ampd.solo`, so its reports don't find solo mode's traces on their own. [Where Amp's traces are](/clis/ampctl/trace#where-amps-traces-are) shows how to report on them. #### Export to VictoriaTraces [VictoriaTraces](https://docs.victoriametrics.com/victoriatraces/) takes traces over OTLP, but not metrics, so set only the traces endpoint. It's a full URL, ending in `/insert/opentelemetry/v1/traces`. On VictoriaTraces's default port, `10428`: ```bash export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=http://localhost:10428/insert/opentelemetry/v1/traces ``` `ampctl trace` then reads them from `JAEGER_URL=http://localhost:10428/select/jaeger`. When VictoriaTraces asks for a username and a password (its `-httpAuth.*` flags), they protect its OTLP endpoint too: send the ones you give `ampctl trace`'s [`JAEGER_AUTH`](/clis/ampctl/trace#connect-to-jaeger) in an `Authorization` header, with the space in it written `%20`: ```bash export OTEL_EXPORTER_OTLP_TRACES_HEADERS="Authorization=Basic%20$(printf 'amp:%s' "$JAEGER_PASSWORD" | base64)" ``` #### Sampling Every worker records a heartbeat each second, and every query and job step is traced, so a busy deployment exports many traces. To keep a share of them, sample by trace: ```bash # Keep 10% of traces, and every trace a client already sampled export OTEL_TRACES_SAMPLER=parentbased_traceidratio export OTEL_TRACES_SAMPLER_ARG=0.1 ``` | `OTEL_TRACES_SAMPLER` | What Amp exports | | -------------------------- | ------------------------------------------------------------------------------------------------- | | `parentbased_always_on` | The default. Every trace, except those a client's `traceparent` says it didn't sample. | | `parentbased_traceidratio` | The share of traces in `OTEL_TRACES_SAMPLER_ARG`, from `0` to `1`, and the ones a client sampled. | | `parentbased_always_off` | Only the traces a client sampled. | | `always_on` | Every trace. | | `traceidratio` | The share of traces in `OTEL_TRACES_SAMPLER_ARG`, whatever the client decided. | | `always_off` | None. To export metrics only, set only the `OTEL_EXPORTER_OTLP_METRICS_*` variables instead. | ### Metrics The [Metrics Reference](/running-amp/metrics) lists every metric, with its type, its unit, and its attributes. Each service exports its metrics every `OTEL_METRIC_EXPORT_INTERVAL`, from when it started: they reset when it restarts. Amp's own metrics are named like Prometheus metrics, like `query_count_total`, and keep their names in most backends. Its HTTP metrics follow OpenTelemetry's names, like `http.server.request.duration`, which backends may translate: Prometheus stores that one as `http_server_request_duration_seconds`. ### Grafana dashboards Amp's repository has Grafana dashboards for its services, in [`grafana/dashboards`](https://github.com/edgeandnode/amp/tree/f27102fd59608258dc58a6dbbaac880f124364ae/grafana/dashboards): | Dashboard | What it shows | | -------------------------- | ------------------------------------------------------------------------------------------------ | | Amp Home | One figure per area, linking to the other dashboards. | | Amp Overview | The whole system's health: queries, extraction, providers, and errors. | | Query Service | Query servers: batch and streaming queries, their latency, errors, memory, and metadata caching. | | Raw Dataset Operations | A raw dataset's extraction: its progress against the chain's head, its provider, and its files. | | Derived Dataset Operations | A derived dataset: its tables' progress, the memory and size of each step. | | ADBC Export Jobs | Exports to external databases. | | Worker Health | Workers: their load, and their use of the metadata database. | | Logs | Each service's logs, from VictoriaLogs. | To use them, import their JSON files in Grafana, in **Dashboards**, **New**, **Import**. They read metrics from a Prometheus data source whose name starts with `amp-`, so add one first, pointing to your Prometheus, like `amp-prometheus`. Their **Cluster** filter reads a `cluster` label, which your metrics pipeline can add to tell deployments apart. Without it, they show every deployment's metrics together. Some panels read Kubernetes metrics, for deployments that run Amp there. ## Troubleshooting ### `unsupported OTLP traces protocol` ``` Exiting with error: Failed to initialize monitoring | Caused by: Reason: unsupported OTLP traces protocol `udp`; supported values are `grpc`, `http/protobuf`, and `http/json` ``` An `OTEL_EXPORTER_OTLP_*PROTOCOL` variable has a value Amp doesn't support. Set it to `grpc`, `http/protobuf`, or `http/json`. ### `unsupported compression algorithm` ``` Exiting with error: Failed to initialize monitoring | Caused by: unsupported compression algorithm 'gzip compression requested but gzip-http feature not enabled' ``` Amp can't compress exports. Unset `OTEL_EXPORTER_OTLP_COMPRESSION`, and the variables like it for one signal. ### Nothing arrives When exports fail, the service keeps running, and logs an error: ``` ERROR amp_telemetry::telemetry::traces: failed to flush OpenTelemetry tracing provider error=Operation failed: errs: [Err(InternalFailure("HTTP export failed: network error"))] error_source=[] ``` * **Check that the endpoint is reachable** from where the service runs. In a container, `localhost` is the container itself. * **Check that the protocol matches the port.** Collectors usually take gRPC on `4317` and HTTP on `4318`, and Amp sends HTTP unless `OTEL_EXPORTER_OTLP_PROTOCOL` is `grpc`. * **Check that you set the variables for the service**, not only in your shell: a process manager or a container only passes the ones you configure. Restart the service after changing them. * **Wait for the interval.** Metrics arrive every 60 seconds by default. ### `ampsql` hangs v0.0.42's `ampsql` prints nothing and never exits when an `OTEL_EXPORTER_OTLP_*` variable that [turns exporting on](/running-amp/telemetry#environment-variables) has a value in its environment, including the ones for traces or metrics only, like [VictoriaTraces's](/running-amp/telemetry#export-to-victoriatraces). Only `ampsql` does this: `ampd` and `ampctl` work with them. Run `ampsql` from a shell without them, or unset them all for it: ```bash env $(env | grep -oE '^OTEL_EXPORTER_OTLP_[A-Z_]+' | sed 's/^/-u /') ampsql "SELECT 1" ```