> **Can't find what you're looking for?** Use `search_docs` on the docs MCP server at `https://docs.ampdb.io/api/mcp` to find what you need. # Limits A query server can limit how long batch queries run, how much memory queries use, and how much each user queries in total, so that no query or user takes more than their share. A query that goes over a limit fails, with an error saying which. By default, those limits are off: whoever runs the query server sets them, in [its configuration](/running-amp/configuration) or, for each user, in a gateway in front of it. One limit is always on, and can't be changed: a query's SQL has to fit in [4 MiB](/querying/limits#query-size). | Limit | Applies to | Set by | Error status | | ----------------------------------------- | ---------------------------------------------------------- | ------------------------------------ | -------------------- | | [Execution deadline](/querying/limits#execution-deadline) | Each batch query | `server.query_timeout_secs` | `DEADLINE_EXCEEDED` | | [Server memory](/querying/limits#memory) | All the queries running on a query server, together | `server.max_mem_mb` | `RESOURCE_EXHAUSTED` | | [Query memory](/querying/limits#memory) | Each batch query, and each microbatch of a streaming query | `server.query_max_mem_mb` | `RESOURCE_EXHAUSTED` | | [Per-user limits](/querying/limits#per-user-limits) | Everything a user has queried | A gateway, in each request's headers | `RESOURCE_EXHAUSTED` | | [Query size](/querying/limits#query-size) | Each query's SQL | Fixed | `OUT_OF_RANGE` | ## Prerequisites * To set the deadline and memory limits: a query server you run, like [solo mode](/running-amp/solo-mode)'s, and its configuration file (see [Configuration](/running-amp/configuration)). Restart it after changing them. * To set per-user limits: a gateway in front of the query server that authenticates users (see [authentication](/querying/connecting#authentication)). * To run the examples, the [example datasets](/querying/connecting#example-datasets). The per-user example also uses the Python client from [Arrow Flight](/querying/arrow-flight#write-a-client), saved as `flight_query.py`. ## Execution deadline (Experimental) `query_timeout_secs` stops batch queries that run longer than a number of seconds: ```toml [config.toml] [server] query_timeout_secs = 30 ``` * `0`, the default, means no deadline. * The clock starts when the query starts running, in `DoGet` (see [how a query runs](/querying/arrow-flight#how-a-query-runs)). Planning the query doesn't count. * Streaming queries have no deadline. * Amp stops a query between steps of its work, so a long step, like a slow call to a [JavaScript function](/querying/functions/javascript), can run past the deadline. * The clock only runs while the client reads the result: a client that stops reading keeps the query open past its deadline. A query that runs past it fails with gRPC status `DEADLINE_EXCEEDED`. With `query_timeout_secs = 1`, a query that pairs each of the Ethereum example dataset's transactions with each of its logs and blocks, 383 billion combinations, fails: :::terminal ```bash ampsql 'SELECT COUNT(DISTINCT t.gas_used * l.log_index + b.gas_used) AS n FROM "acme/eth_mainnet".transactions t, "acme/eth_mainnet".logs l, "acme/eth_mainnet".blocks b' ``` ```text ✗ query failed → gRPC status error → code: 'Deadline expired before operation could complete', message: "query execution deadline exceeded" ``` ::: To fit a query within the deadline, have it read fewer blocks, by [filtering on them](/querying/batch-queries#filter-by-block). To get a large result over time instead, run it as a [streaming query](/querying/streaming-queries). ## Memory Queries need memory to sort, group, and join rows. Two settings cap it, in MiB: ```toml [config.toml] [server] max_mem_mb = 4096 query_max_mem_mb = 1024 ``` * `max_mem_mb` caps the memory of all the queries running on the query server at once. Each query server has its own: several query servers don't share it. * `query_max_mem_mb` caps each batch query, and each microbatch of a streaming query. Amp sizes a stream's microbatches to fit in it (see [microbatch size](/querying/streaming-queries#microbatch-size)). * `0`, the default for both, means no limit. * They cap the memory Amp's query engine sets aside to run queries, not all the memory the query server uses. A query that needs more memory than it can get fails with gRPC status `RESOURCE_EXHAUSTED`, and a message with the limit it hit. With `query_max_mem_mb = 1`: :::terminal ```bash ampsql 'SELECT COUNT(DISTINCT input) AS inputs FROM "acme/eth_mainnet".transactions' ``` ```text ✗ query failed → gRPC status error → code: 'Some resource has been exhausted', message: "query execution error | Caused by: Resources exhausted: memory limit of 1024.0 KB exceeded: Resources exhausted: Memory Exhausted while SpillPool (DiskManager is disabled)" ``` ::: To fit a query in less memory, have it read fewer blocks or columns. For a streaming query, lower its [`microbatch_max_interval`](/querying/sql#settings). A query that failed because others were using the server's memory may succeed when they're done. ## Per-user limits A hosted Amp can cap how much each of its users queries in total: memory, time, and data read and sent. A gateway in front of the query server sends the user's limits with each request. Amp stores what each user has used, and gives each new query what's left. A request has per-user limits when it has both of these headers: 1. `authorization`, with a JWT whose `sub` claim is the user's ID. 2. `amp-capabilities`, with a `resource.limit` capability: ```json { "capabilities": [ { "name": "resource.limit", "scope": { "memory": 1073741824, "query_duration": 30000, "network_in": 1048576, "network_out": 104857600, "disk_in": 1073741824, "disk_out": 1073741824 } } ] } ``` All six fields are required: | Field | Unit | What counts | | ---------------- | ------------ | ------------------------------------------------------------------------------- | | `memory` | bytes | The most memory each batch query, or each microbatch of a stream, used at once. | | `query_duration` | milliseconds | The time each batch query, or each microbatch of a stream, ran. | | `network_in` | bytes | The tickets the user sent with `DoGet`. | | `network_out` | bytes | The results Amp sent the user. | | `disk_in` | bytes | The data queries read from storage. | | `disk_out` | bytes | The data queries wrote to disk when they ran out of memory. | * `memory: 0` means no per-user memory limit, but `0` for any other field leaves no budget at all. * The limits are totals, over all of a user's queries, and their usage never resets. To give a user more, the gateway sends higher limits. * Amp checks the budgets while a query runs. A query that goes over fails then, and what it used still counts. * A request whose `amp-capabilities` has no `resource.limit` fails, when it has a user. A request without a user has no per-user limits. * Each query server checks the budgets on its own, and doesn't set them aside for running queries. Queries that run at the same time, or on different query servers, can together use more than a user had left. * The [memory limits](/querying/limits#memory) still apply. :::warning The query server doesn't verify either header. The gateway must authenticate users and set both headers itself, replacing any a client sends. ::: For example, on your own Amp, with a `network_out` budget of 100,000 bytes for a user named `docs-reader`. Since the query server doesn't verify tokens, an unsigned one, whose payload is `{"sub":"docs-reader"}`, will do: ```bash TOKEN=eyJhbGciOiJub25lIn0.eyJzdWIiOiJkb2NzLXJlYWRlciJ9. LIMITS='{"capabilities": [{"name": "resource.limit", "scope": {"memory": 0, "query_duration": 60000, "network_in": 1048576, "network_out": 100000, "disk_in": 1073741824, "disk_out": 1073741824}}]}' ``` A small query works: :::terminal ```bash python flight_query.py 'SELECT block_num, gas_used FROM "acme/eth_mainnet".blocks ORDER BY block_num LIMIT 3' \ "authorization: Bearer $TOKEN" "amp-capabilities: $LIMITS" ``` ```text [mainnet 24000000-24000099] 3 rows block_num gas_used 24000000 57860235 24000001 27660983 24000002 17602635 [mainnet 24000000-24000099] complete ``` ::: `SELECT tx_hash, input FROM "acme/eth_mainnet".transactions` sends more than 100,000 bytes, so it fails with gRPC status `RESOURCE_EXHAUSTED`: ```text resource limit exceeded | Caused by: resource (network out) limit exceeded ``` From then on, each of the user's queries fails the same way as soon as it starts, even the small one. ## Query size The query server refuses a request larger than 4 MiB (4,194,304 bytes), from any client. A query's request is its SQL plus about 100 bytes, so SQL of about 4 MiB or more fails with gRPC status `OUT_OF_RANGE`, before it's planned. You can't change this limit. :::terminal ```bash python3 -c "print('SELECT 1 AS x -- ' + 'a' * 5000000)" > long_query.sql ampsql -f long_query.sql ``` ```text ✗ query failed → Arrow Flight error → Tonic error: code: 'Operation was attempted past the valid range', message: "Error, decoded message length too large: found 5000104 bytes, the limit is: 4194304 bytes" → code: 'Operation was attempted past the valid range', message: "Error, decoded message length too large: found 5000104 bytes, the limit is: 4194304 bytes" ``` ::: SQL this long usually comes from a script that writes out a long list of values, like an `IN` list of addresses. Split the list across several queries. ## Errors | Message contains | Status | Cause | Fix | | -------------------------------------------------------------------------------- | --------------------- | ---------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------- | | `query execution deadline exceeded` | `DEADLINE_EXCEEDED` | A batch query ran longer than `query_timeout_secs`. | [Read fewer blocks](/querying/limits#execution-deadline), or stream the query. | | `Resources exhausted`, with `memory limit of … exceeded` or `Failed to allocate` | `RESOURCE_EXHAUSTED` | A query needed more memory than the server, the query, or its user had left. | [Use less memory](/querying/limits#memory). If other queries were using the server's memory, retry later. | | `resource (memory) limit exceeded` | `RESOURCE_EXHAUSTED` | The user's `memory` budget is spent. | Raise the user's limits. | | `resource (query duration) limit exceeded` | `RESOURCE_EXHAUSTED` | The user's `query_duration` budget is spent. | Raise the user's limits. | | `resource (network in) limit exceeded`, `resource (network out) limit exceeded` | `RESOURCE_EXHAUSTED` | The user's `network_in` or `network_out` budget is spent. | Raise the user's limits. | | `resource (disk in) limit exceeded`, `resource (disk out) limit exceeded` | `RESOURCE_EXHAUSTED` | The user's `disk_in` or `disk_out` budget is spent. | Raise the user's limits. | | ``missing `resource.limit` capability in `amp-capabilities` header`` | `FAILED_PRECONDITION` | A request with a user has an `amp-capabilities` header without `resource.limit`. | Have the gateway add it. | | ``malformed `amp-capabilities` header`` | `FAILED_PRECONDITION` | The header isn't valid JSON, or a `resource.limit` misses a field or has an unknown one. | Check it against the [example above](/querying/limits#per-user-limits). | | ``malformed `authorization` header`` | `FAILED_PRECONDITION` | The token isn't a JWT with a `sub` claim. | Send a JWT with the user's ID in `sub`. | | `decoded message length too large` | `OUT_OF_RANGE` | The query's SQL is about 4 MiB or more. | [Shorten it](/querying/limits#query-size). |