> **Can't find what you're looking for?** Use `search_docs` on the docs MCP server at `https://docs.ampdb.io/api/mcp` to find what you need. # Metrics Reference Every metric Amp exports when you [export metrics](/running-amp/telemetry#traces-and-metrics), grouped by the part of Amp that records it. To use them, see [Telemetry](/running-amp/telemetry#metrics). ## Types | Type | What it measures | | ------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- | | Counter | A total that only goes up, like queries run. Graph its rate. Most counters' names end in `_total`. | | Gauge | A value at the time it was recorded, like the latest block written. An observable gauge is read each time Amp exports metrics. | | UpDownCounter | A value that goes up and down as things start and end, like queries running. | | Histogram | The distribution of values, like query durations, for percentiles and averages. Prometheus stores it as `_bucket`, `_sum`, and `_count` series. | Most histograms use OpenTelemetry's default buckets, which stop at 10,000: larger values all land in the last one, so percentiles can't go past it. That affects durations in milliseconds over 10 seconds, like `query_duration_milliseconds`, and `query_memory_peak_bytes` and `streaming_microbatch_rows`. The histograms whose values go higher, like the microbatch, provider, and archive ones, have buckets that fit them. Each service records its own metrics, from when it starts: they reset when it restarts. A job running again doesn't reset them: its retries and later runs continue the same series until its worker restarts. To see a deployment's total, sum them across services. Names are as Amp exports them. Most backends keep them, except the [HTTP metrics](/running-amp/metrics#http-server-metrics)' names, which follow OpenTelemetry's conventions: Prometheus stores `http.server.request.duration` as `http_server_request_duration_seconds`. ## Attributes Besides each service's [resource attributes](/running-amp/telemetry#resource-attributes), like `service.name`, many metrics have these: | Attribute | Description | | --------------- | ------------------------------------------------------------------------------------------------------------------------------------- | | `dataset` | The dataset, as `namespace/name`, like `acme/eth_mainnet`. | | `manifest_hash` | The hash of the dataset version's [manifest](/datasets/manifests), which identifies the version. | | `job_id` | The [job](/managing-amp/jobs#ids-and-idempotency-keys)'s ID. Its retries and later runs keep it, so it doesn't identify a single run. | | `table` | The table. | | `location_id` | The ID of the table's [revision](/managing-amp/table-revisions) being written. | | `provider` | The [provider](/providers)'s name. | | `network` | The chain's network, like `mainnet`. | ## HTTP server metrics The [query server](/running-amp/query-server) and the [controller](/running-amp/controller) record each HTTP request, with OpenTelemetry's names and attributes for HTTP: `http.request.method`, `http.response.status_code`, and `http.route`, the route the request matched, if any. The query server's Arrow Flight calls are HTTP requests too, with the route `/arrow.flight.protocol.FlightService/{*rest}`. | Metric | Type | Description | | -------------------------------- | ------------- | ----------------------------------------------------------------------------- | | `http.server.request.duration` | Histogram | How long requests took, in seconds. | | `http.server.active_requests` | UpDownCounter | Requests in progress. Only has `http.request.method`. | | `http.server.request.body.size` | Histogram | The size of request bodies, in bytes, for requests that state it. | | `http.server.response.body.size` | Histogram | The size of response bodies, in bytes, when Amp knows it before sending them. | To count requests, use the count of `http.server.request.duration`: `http_server_request_duration_seconds_count` in Prometheus. ## Query metrics The query server records each query. `query_count_total` and `query_errors_total` count both [batch](/querying/batch-queries) and [streaming](/querying/streaming-queries) queries, and the others describe batch queries only: [streaming query metrics](/running-amp/metrics#streaming-query-metrics) describe streaming ones. A query that reads several datasets is recorded once for each, with its `dataset` and `manifest_hash`. A query that reads none, like `SELECT 1`, is recorded without them, and so are some others. | Metric | Type | Description | | ----------------------------- | --------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | | `query_count_total` | Counter | Queries that started running. One that fails before is only counted in `query_errors_total`. | | `query_duration_milliseconds` | Histogram | How long queries took to plan and run. | | `query_rows_returned_total` | Counter | Rows returned to clients. | | `query_bytes_egress_total` | Counter | The size of the results returned to clients, in memory, before they're encoded to send. | | `query_errors_total` | Counter | Queries that failed, by `error_code` (see below). | | `query_memory_peak_bytes` | Histogram | The most memory each query reserved, once for each dataset it reads, without `dataset` or `manifest_hash`. See [Memory](/running-amp/query-server#memory). | `error_code` is Amp's error code, like `PLANNING_ERROR`, or, for a batch query that fails after returning rows, the error's message. Not every failure is counted: a query that names a table that doesn't exist, for example, isn't, and neither is a streaming query that fails after it started. ## Streaming query metrics The query server records each [streaming query](/querying/streaming-queries), once for each dataset it reads, with its `dataset` and `manifest_hash`. A streaming query that reads no dataset isn't recorded. A streaming query sends new rows in batches as blocks arrive. | Metric | Type | Description | | -------------------------------------------- | ------------- | -------------------------------------------------------------------------------- | | `streaming_queries_started_total` | Counter | Streaming queries started. | | `streaming_queries_completed_total` | Counter | Streaming queries that ended, whether they finished, failed, or the client left. | | `streaming_queries_active` | UpDownCounter | Streaming queries running on this query server. | | `streaming_query_lifetime_milliseconds` | Histogram | How long streaming queries ran. | | `streaming_microbatch_rows` | Histogram | The rows in each batch sent. | | `streaming_microbatch_duration_milliseconds` | Histogram | How long each batch took, from its start to its end. | | `streaming_rows_sent_total` | Counter | Rows sent. | | `streaming_bytes_sent_total` | Counter | The size of the rows sent, in memory. | ## Microbatch metrics Streaming queries, [derived dataset](/datasets/derived-datasets) jobs, and [export jobs](/managing-amp/pipelines) process new blocks in steps called microbatches, sized to fit their memory. These metrics show how Amp sizes them. Each has a `context` attribute: * `flight`: streaming queries. They also have `dataset` and `manifest_hash` when the query reads one dataset. * `materialize_table_derived`: derived dataset jobs. * `materialize_table_adbc`: export jobs. Jobs also have `table`: the dataset, the first 7 characters of its manifest hash, and the table, like `acme/eth_mainnet@b9e2ebd.blocks`. They don't have `job_id`, so jobs of the same kind that process the same table share a series. | Metric | Type | Description | | --------------------------------------------------- | --------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `microbatch_estimated_source_bytes` | Histogram | How much compressed data each microbatch was estimated to read. | | `microbatch_source_bytes_budget` | Histogram | How much data the next microbatch may read. Only recorded with a memory limit, once Amp has calibrated it. | | `microbatch_range_resolution_duration_milliseconds` | Histogram | How long Amp took to choose the next microbatch's blocks, including when there were none yet. Also has `network_mode`, `single` or `multi` networks, and `status`: `ready`, `pending`, `complete`, or `error`. | | `microbatch_selected_interval_blocks` | Histogram | How many blocks each microbatch covered, for queries over one network. | | `microbatch_selected_interval_seconds` | Histogram | How much time each microbatch covered, for queries over several networks. | | `microbatch_peak_reserved_bytes` | Histogram | The most memory each microbatch reserved. | | `microbatch_memory_pool_size` | Histogram | The memory limit of each query or job. Only recorded with a limit. | | `microbatch_first_batch_duration_milliseconds` | Histogram | How long each microbatch took to produce its first rows. Not recorded for microbatches without rows. | | `microbatch_drain_duration_milliseconds` | Histogram | How long each microbatch took to produce all its rows, including the time its consumer took to read them. | ## Materialization metrics Workers record these as they build [raw](/datasets/raw-datasets) and [derived](/datasets/derived-datasets) datasets' tables. Each has `dataset`, `manifest_hash`, `job_id`, and `table`, and the ones about what was written have `location_id` too. | Metric | Type | Description | | ---------------------------------- | --------- | ---------------------------------------------------------------- | | `rows_ingested_total` | Counter | Rows written. | | `files_written_total` | Counter | Parquet files written. | | `bytes_written_total` | Counter | The size of the rows written, before compression. | | `write_calls_total` | Counter | Batches of rows written. | | `latest_block_number` | Gauge | The number of the latest block written. Raw datasets only. | | `latest_segment_timestamp_seconds` | Gauge | The time of the latest block written, as a Unix timestamp. | | `latest_segment_delay_seconds` | Gauge | How far behind the current time the latest block written is. | | `dump_duration_milliseconds` | Histogram | How long each run of the job took to complete. No `location_id`. | | `dump_errors_total` | Counter | Runs that failed. No `location_id`. | ## Export metrics Workers record these as [export jobs](/managing-amp/pipelines) export tables to [PostgreSQL or Snowflake](/managing-amp/pipelines/destinations). Each has `dataset`, `manifest_hash`, `job_id`, and `table`. | Metric | Type | Description | | ----------------------------------- | --------- | --------------------------------------------------------------- | | `adbc_export_duration_milliseconds` | Histogram | How long each table's export took, when it succeeded. | | `adbc_export_errors_total` | Counter | Exports that failed. | | `adbc_rows_exported_total` | Counter | Rows exported. Amp doesn't record it yet, so it doesn't appear. | ## Compaction metrics Workers record these as they [compact](/running-amp/storage/compaction) tables. Each has `dataset`, `manifest_hash`, `job_id`, and `table`, and the ones that count bytes and files have `location_id` too. | Metric | Type | Description | | ---------------------------------- | --------- | ------------------------------ | | `compaction_files_read_total` | Counter | Files read, to merge them. | | `compaction_bytes_read_total` | Counter | Bytes read. | | `compaction_bytes_written_total` | Counter | Bytes written. | | `compaction_duration_milliseconds` | Histogram | How long each compaction took. | | `successful_compactions` | Counter | Compactions that succeeded. | | `failed_compactions` | Counter | Compactions that failed. | To see how much compaction shrinks data, divide the rate of `compaction_bytes_written_total` by that of `compaction_bytes_read_total`. ## Garbage collection metrics Workers record these as they [collect](/running-amp/storage/garbage-collection) tables' files. Each has `dataset_namespace`, `dataset_name`, `manifest_hash`, and `table`. | Metric | Type | Description | | ----------------------------- | ------- | ----------------------------------------------------------------------------------------- | | `gc_expired_files_found` | Counter | Expired files found. It counts each time a file is found, so it isn't the number waiting. | | `gc_metadata_entries_deleted` | Counter | Files removed from the metadata database's list. | | `gc_files_deleted` | Counter | Files deleted from storage. | | `gc_files_not_found` | Counter | Files that were already gone from storage. | ## Data store metrics Every service records these as it reads tables' files and their metadata. ### Parquet metadata cache The [Parquet metadata cache](/running-amp/storage/metadata-caching#parquet-metadata): | Metric | Type | Description | | ----------------------------------- | ------- | ------------------------------------------------------------- | | `parquet_footer_cache_hits_total` | Counter | Lookups found in the cache. | | `parquet_footer_cache_misses_total` | Counter | Lookups not in the cache, fetched from the metadata database. | | `parquet_footer_cache_usage_bytes` | Gauge | The cache's size, recorded at each lookup. | ### Metadata database pool Each service's connections to the [metadata database](/running-amp/metadata-database), to size its [pool](/running-amp/metadata-database#pool_size): | Metric | Type | Description | | --------------------------------------- | ---------------- | -------------------------------------------------------------------------------------------------------------------------------------------- | | `metadb_pool_connections` | Observable gauge | The pool's connections, by `state`: `idle` or `active`. | | `metadb_pool_acquire_wait_milliseconds` | Histogram | How long starting a transaction took, including waiting for a connection or opening one, by `outcome`: `acquired`, `timed_out`, or `failed`. | | `metadb_pool_acquire_timeouts_total` | Counter | Transactions that waited longer than [`acquire_timeout_secs`](/running-amp/metadata-database#acquire_timeout_secs) for a connection. | Requests that use a connection without starting a transaction aren't recorded. `metadb_pool_acquire_wait_milliseconds`'s buckets go past the default `acquire_timeout_secs`, 5 seconds, so timeouts land in the last ones. ### File lists The [cache of each table's files](/running-amp/storage/metadata-caching#file-lists). Their `subsystem` attribute is the part of Amp that asked: `raw_materialize`, `raw_freshness_tracker`, `compaction`, `derived_materialize`, `adbc_export`, `query_server`, `admin_api`, or `unknown`. | Metric | Type | Description | | -------------------------------------- | --------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `membership_resolve_total` | Counter | Times a service found a table's files, by `subsystem` and `outcome` (see below). Failures aren't counted. | | `membership_cache_usage_bytes` | Gauge | The cache's size, recorded each time a service finds a table's files. | | `file_ids_query_total` | Counter | Times a service read a table's whole list from the metadata database, by `subsystem` and `status`: `ok` or `error`. A read that didn't get a connection isn't counted. | | `file_ids_query_rows_total` | Counter | Files those reads returned, by `subsystem` and `status`. | | `file_ids_query_duration_milliseconds` | Histogram | How long those reads took, without waiting for a connection, by `subsystem` and `status`. | | `membership_count_mismatch_total` | Counter | Reads where the database's count of a table's files didn't match the files it listed, by `subsystem`. | `membership_resolve_total`'s `outcome`: | Outcome | How the service found the files | | ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ | | `served_from_cache` | From its cache, after checking with the database that the version hadn't changed. | | `updated_from_log` | From its cache, updated with the change log. | | `reread_not_cached` | It read the whole list, because it wasn't cached: it was never read, the cache dropped it to make room, or it went unused. Expected after a start. | | `reread_log_too_long` | It read the whole list: there were too many changes to read from the log at once. Expected after truncating a revision, which changes every file. | | `reread_log_incomplete` | It read the whole list: its version was more than `max_delta_versions` behind. Expected now and then; a rising rate means services fall behind faster. | | `reread_files_changed` | It read the whole list again: a file was deleted while it read the files' details. | | `reread_cache_ahead` | It read the whole list: its cached version was newer than the database's, which happens when the database is restored from a backup. | | `revision_gone` | The revision no longer exists. | The cache hit rate is the share of lookups the cache answered without reading the list: `served_from_cache` over every outcome (see the [example](/running-amp/metrics#example-queries)). When `membership_cache_usage_bytes` stays at [`cache_size_mb`](/running-amp/storage/metadata-caching#file-list-settings), in bytes, and `reread_not_cached` rises, the cache is too small. `file_ids_query_*` with `subsystem="unknown"` means a part of Amp didn't say who it was: report it. Any rate of `membership_count_mismatch_total` means the metadata database's tables disagree, which is worth investigating. ## Controller metrics The controller prunes the [change log of tables' files](/running-amp/storage/metadata-caching#file-lists) every [`log_prune_interval_secs`](/running-amp/configuration-reference#location_membershiplog_prune_interval_secs). | Metric | Type | Description | | ------------------------------- | ------- | ----------------------------------------- | | `membership_pruned_rows_total` | Counter | Rows deleted from the log. | | `membership_prune_cycles_total` | Counter | Prunes run, by `status`: `ok` or `error`. | `membership_pruned_rows_total` only goes up when there was something to delete, so it often stays flat. `membership_prune_cycles_total` goes up at each prune, so it tells you pruning runs: watch it over a window longer than `log_prune_interval_secs`. ## Provider request metrics Requests to [providers](/providers). Each has `provider`, `network`, and `method`, the request's method, like `eth_getBlockByNumber`. | Metric | Type | Description | | ---------------------------------------- | --------- | ----------------------- | | `provider_requests_total` | Counter | Requests completed. | | `provider_request_duration_milliseconds` | Histogram | How long requests took. | | `provider_request_errors_total` | Counter | Requests that failed. | ## Chain head metrics The latest block of the chain, as the provider reports it, for [Solana](/providers/solana). Each has `provider` and `network`. Amp learns of new blocks in two ways, which the `observer` attribute names: `subscription`, which the endpoint pushes new blocks to, and `poll`, which asks for the latest block when the subscription goes quiet. A steady trickle of `poll` alongside `subscription` means the subscription keeps going quiet; only `poll`, after a run of `subscription`, means the endpoint stopped accepting it. | Metric | Type | Description | | ------------------------------------------ | --------- | ---------------------------------------------------------------------------------------------------------------- | | `chain_head_block_number` | Gauge | The number of the latest block. | | `chain_head_advance_interval_seconds` | Histogram | The time between new blocks, by `observer`. | | `chain_head_advance_slots` | Histogram | How many blocks the head moved at once, by `observer`. | | `chain_head_observations_total` | Counter | Times Amp learned of the head, by `observer` and `outcome`: `advanced` or `behind`. | | `chain_head_poll_failures_total` | Counter | Requests for the latest block that failed. After several, Amp waits longer between them, so the rate levels off. | | `chain_head_subscription_reconnects_total` | Counter | Times the subscription ended and Amp reconnected. | ## eth_call metrics Calls of the [`eth_call`](/querying/functions/eth-call) function, which calls contracts from SQL. Each has `network` and `provider`. | Metric | Type | Description | | ------------------------------- | --------- | ------------------------------------------ | | `eth_call_requests_total` | Counter | Calls. | | `eth_call_latency_milliseconds` | Histogram | How long calls took, including retries. | | `eth_call_errors_total` | Counter | Calls that failed, by `class` (see below). | | `eth_call_retries_total` | Counter | Retries. | `class` is one of: * `rpc_error_non_retryable`: the endpoint returned an error that retrying can't fix. * `rpc_error_retry_exhausted`: the call still failed after its retries. * `network_not_configured`, `provider_creation_failed`: Amp couldn't find or start a provider for the network. These have `provider="unknown"`. ## Extraction metrics How workers turn the blocks providers return into rows, for the [Solana provider](/providers/solana). Each has `provider`, `network`, and `source`, where the block came from: `archive` or `rpc`. The first six, about turning blocks into rows, also have `phase`: `backfill` before the chain's head, or `live` at it. | Metric | Type | Description | | ---------------------------------------- | ------------- | ---------------------------------------------------------------------------------------- | | `provider_extract_duration_seconds` | Histogram | The processing time to turn a block into rows, without waiting. | | `provider_extract_queue_wait_seconds` | Histogram | How long blocks waited to be turned into rows. | | `provider_blocks_extracted_total` | Counter | Blocks turned into rows. | | `provider_extract_in_flight` | UpDownCounter | Blocks being turned into rows. | | `provider_extract_bytes` | Histogram | The size of each block, as received. Solana's archive only. | | `provider_block_age_seconds` | Histogram | How old blocks were when they became rows. Solana's RPC only. | | `provider_ordered_completed_undelivered` | UpDownCounter | Blocks ready, waiting for earlier blocks so they're written in order. | | `provider_ordered_delivery_lag_seconds` | Histogram | How long ready blocks waited for earlier ones. | | `provider_ordered_head_of_line_gap` | Histogram | How far ahead of the block awaited the furthest ready block was, in blocks. | | `provider_ordered_starved_requests` | Histogram | Requests Amp could have sent but didn't, because too many blocks were waiting. RPC only. | | `provider_ordered_window` | Gauge | The most ready blocks that may wait. | | `provider_read_ahead_stall_seconds` | Histogram | How long extraction waited for the job to take a block. | | `provider_read_ahead_depth` | Gauge | Blocks extracted ahead of the job. | | `provider_read_ahead_capacity` | Gauge | The most blocks that may be extracted ahead. | | `provider_stream_active_seconds` | Histogram | How long each block took to produce, once the job asked for it. | | `provider_stream_blocked_seconds` | Histogram | How long extraction waited for the job to ask for the next block. | ## Solana metrics The [Solana provider](/providers/solana) records these besides the ones above. Each has `provider` and `network`. ### Archive Reading Old Faithful archives, one file per epoch. Each has `epoch`. | Metric | Type | Description | | ----------------------------------------------------- | ------------- | ---------------------------------------------------------------------------------- | | `provider_archive_files_processed_total` | Counter | Archive files read. | | `provider_archive_download_duration_seconds` | Histogram | How long each whole-file download took. | | `provider_archive_download_bytes` | Histogram | The size of each whole-file download. | | `provider_archive_download_errors_total` | Counter | Downloads that failed. | | `provider_archive_reader_in_flight` | UpDownCounter | Parts of files being read at once. | | `provider_archive_reader_duration_seconds` | Histogram | How long each part took to read and decode. | | `provider_archive_reader_bytes` | Histogram | The compressed size of each part. | | `provider_archive_reader_first_block_seconds` | Histogram | How long each part took to produce its first block. | | `provider_archive_reader_blocks` | Histogram | Blocks decoded from each part. | | `provider_archive_emit_loop_duration_seconds` | Histogram | How long each decoded part took to hand over to extraction. | | `provider_archive_emit_yield_blocked_seconds` | Histogram | How much of that time was spent waiting for extraction. | | `provider_archive_emit_slots_total` | Counter | Slots handed over to extraction. | | `provider_archive_split_planning_duration_seconds` | Histogram | How long Amp took to split each file into parts to read at once. | | `provider_archive_split_ranges` | Histogram | The parts each epoch's file was split into. | | `provider_archive_split_fallbacks_total` | Counter | Epochs read in one part, because Amp couldn't split them, by `reason` (see below). | | `provider_archive_boundary_validation_failures_total` | Counter | Split points Amp checked and rejected. | | `provider_archive_stream_errors_total` | Counter | Errors reading files, by `kind` and `fatal` (see below). | * **`reason`**: `unsupported`, `index_unavailable`, or `no_boundaries`. * **`kind`**: the step that failed, like `file_stream`, `car_read`, `decode_task`, `slot_out_of_range`, `fill_truncated_logs`, or `rows`. * **`fatal`**: `true` when the error stopped the read, `false` when Amp recovered. ### RPC Requests to a Solana RPC endpoint. Each has `method`. | Metric | Type | Description | | ----------------------------------------------- | ------------- | ---------------------------------------------------------------------------------------- | | `provider_rate_limit_wait_milliseconds` | Histogram | How long requests waited for the provider's rate limit. | | `provider_requests_in_flight` | UpDownCounter | Requests sent, waiting for a response. | | `provider_response_ttfb_milliseconds` | Histogram | How long the endpoint took to start responding, by `http_version`. | | `provider_response_body_duration_milliseconds` | Histogram | How long responses took to receive, once started, by `http_version`. | | `provider_response_parse_duration_milliseconds` | Histogram | How long responses took to parse, by `http_version`. | | `provider_response_bytes` | Histogram | The size of responses, uncompressed, by `http_version`. | | `provider_responses_total` | Counter | Responses, by `http_version`. | | `provider_requests_throttled_total` | Counter | Responses refusing a request for exceeding the endpoint's rate limit, by `http_version`. | | `provider_request_failures_total` | Counter | Requests that failed, by `kind` (see below). | * **`http_version`**: `http/1.1`, `h2`, `h3`, `http/1.0`, `http/0.9`, or `unknown`. * **`kind`**: why the request failed, like `timeout`, `transport`, `status`, `body`, `parse`, `skipped_slot`, or `rpc_error`. ### Archive host limits Amp adapts how many requests it sends archive hosts at once, lowering the limit when a host refuses requests. Requests for the archive's indexes are the control plane, and requests for its blocks the data plane. These metrics have no `epoch`, and Amp reads them each time it exports metrics. | Metric | Type | Description | | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------ | ------------------------------------------------------------------------------------- | | `provider_archive_host_limit_control`, `provider_archive_host_limit_data` | Observable gauge | The current limit of requests at once, for each plane. | | `provider_archive_host_in_flight_control`, `provider_archive_host_in_flight_data` | Observable gauge | Requests in progress, for each plane. | | `provider_archive_host_rate_limit_events_total` | Observable counter | Times a host refused a request as over its rate limit (HTTP 429), lowering the limit. | | `provider_archive_host_rate_limit_events_root_cid_total`, `provider_archive_host_rate_limit_events_index_total`, `provider_archive_host_rate_limit_events_car_frame_total`, `provider_archive_host_rate_limit_events_car_stream_total` | Observable counter | The same, by the kind of request. | | `provider_archive_host_stream_drop_events_total` | Observable counter | Downloads the host dropped partway, which lowered the limit. | | `provider_archive_host_stream_drop_bytes_total` | Observable counter | Bytes received before those drops. | ## Example queries In [PromQL](https://prometheus.io/docs/prometheus/latest/querying/basics/), for Prometheus and compatible backends: ```txt # Queries per second, by dataset sum by (dataset) (rate(query_count_total[5m])) # 95th percentile query duration, in seconds histogram_quantile(0.95, sum by (dataset, le) (rate(query_duration_milliseconds_bucket[5m]))) / 1000 # Failed queries per query started sum(rate(query_errors_total[5m])) / sum(rate(query_count_total[5m])) # Streaming queries running, across query servers sum by (dataset) (streaming_queries_active) # Rows written per second, by table sum by (dataset, table) (rate(rows_ingested_total[5m])) # How far behind the chain each table is, in seconds max by (dataset, table) (latest_segment_delay_seconds) # How much compaction shrinks data sum(rate(compaction_bytes_written_total[5m])) / sum(rate(compaction_bytes_read_total[5m])) # Files garbage collection deletes per second sum by (dataset_namespace, dataset_name, table) (rate(gc_files_deleted[5m])) # Share of each service's metadata database connections in use sum by (service_name) (metadb_pool_connections{state="active"}) / sum by (service_name) (metadb_pool_connections) # Share of provider requests that fail sum by (provider, network) (rate(provider_request_errors_total[5m])) / sum by (provider, network) (rate(provider_requests_total[5m])) # Parquet metadata cache hit rate sum(rate(parquet_footer_cache_hits_total[5m])) / (sum(rate(parquet_footer_cache_hits_total[5m])) + sum(rate(parquet_footer_cache_misses_total[5m]))) # File-list cache hit rate sum(rate(membership_resolve_total{outcome="served_from_cache"}[5m])) / sum(rate(membership_resolve_total[5m])) # Admin API requests per second that fail, by route sum by (http_route) (rate(http_server_request_duration_seconds_count{service_name="ampd.controller", http_response_status_code=~"5.."}[5m])) ```