> **Can't find what you're looking for?** Use `search_docs` on the docs MCP server at `https://docs.ampdb.io/api/mcp` to find what you need.

# Raw Datasets

A raw dataset holds one chain's data, as Amp extracts it: its blocks, transactions, logs, and whatever else the chain
records. It's where all data in Amp starts. You query raw datasets directly, and build
[derived datasets](/datasets/derived-datasets) on top of them.

Each raw dataset extracts one network of one chain, through a [provider](/providers): a connection to an RPC endpoint
or another source of that chain's data. The dataset says what to extract, and the provider says where from, so the
same dataset works with any endpoint you register.

## Kinds

A raw dataset's `kind` says which chain data it holds. It decides the dataset's tables, and which providers it can
extract through:

| `kind`        | Chains                                        | Provider                                 | Tables                                                |
| ------------- | --------------------------------------------- | ---------------------------------------- | ----------------------------------------------------- |
| `evm-rpc`     | Ethereum and EVM-compatible chains, like Base | [EVM RPC](/providers/evm-rpc)            | [EVM RPC Tables](/datasets/tables/evm-rpc)            |
| `solana`      | Solana                                        | [Solana](/providers/solana)              | [Solana Tables](/datasets/tables/solana)              |
| `tempo`       | Tempo                                         | [Tempo](/providers/tempo)                | [Tempo Tables](/datasets/tables/tempo)                |
| `bittensor`   | Bittensor                                     | [Bittensor](/providers/bittensor)        | [Bittensor Tables](/datasets/tables/bittensor)        |
| `bitcoin-rpc` | Bitcoin                                       | [Bitcoin RPC](/providers/bitcoin-rpc)    | [Bitcoin RPC Tables](/datasets/tables/bitcoin-rpc)    |
| `eth-beacon`  | Ethereum's consensus layer                    | [Ethereum Beacon](/providers/eth-beacon) | [Ethereum Beacon Tables](/datasets/tables/eth-beacon) |
| `stellar`     | Stellar                                       | [Stellar](/providers/stellar)            | [Stellar Tables](/datasets/tables/stellar)            |
| `phaser`      | Chains a Phaser bridge serves                 | [Phaser](/providers/phaser)              | Discovered from the bridge                            |

## Prerequisites

* A running Amp you can manage, like one in [solo mode](/running-amp/solo-mode).
* [`ampctl`](/clis/ampctl) and [`ampsql`](/clis/ampsql), which install with Amp.
* An RPC endpoint for the chain, in the environment `ampd` runs in. The examples extract Ethereum mainnet from block
  24,000,000, so the endpoint must serve that block and its receipts:

  ```bash
  export RPC_URL=…
  ```

## Create a raw dataset

::::steps
### Register a provider

Write the provider's configuration, then register it. Its kind and network must match the dataset's.

```toml [mainnet.toml]
kind = "evm-rpc"
network = "mainnet"
url = "${RPC_URL}"
```

```bash
ampctl provider register mainnet_rpc ./mainnet.toml
```

Amp reads `${RPC_URL}` when it extracts, from its own environment (see
[Environment variables](/providers#environment-variables)).

### Generate the manifest

`ampctl manifest generate` writes the dataset's [manifest](/datasets/manifests), with every table and column of its
kind. This one starts at block 24,000,000, so Amp skips the chain's earlier history:

```bash
ampctl manifest generate --kind evm-rpc --network mainnet --start-block 24000000 -o eth_mainnet.json
```

| Flag                          | Description                                                                                                                                    |
| ----------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- |
| `--kind <KIND>`               | The dataset's kind: `bittensor`, `bitcoin-rpc`, `eth-beacon`, `evm-rpc`, `solana`, `stellar`, or `tempo`. Required.                            |
| `--network <NETWORK>`         | The network to extract, like `mainnet`. Required.                                                                                              |
| `--start-block <START_BLOCK>` | The first block the dataset covers. Defaults to `0` (see [Start block](/datasets/raw-datasets#start-block)).                                                         |
| `--finalized-blocks-only`     | Extract only final blocks (see [Finalized blocks](/datasets/raw-datasets#finalized-blocks)).                                                                         |
| `-o`, `--out <OUT>`           | Where to write the manifest. A directory gets a file named after the kind, like `evm-rpc.json`. Without it, the manifest goes to the terminal. |

### Register the dataset

Register the manifest under a name, here `acme/eth_mainnet`, and tag it with a [version](/datasets/versioning):

```bash
ampctl dataset register acme/eth_mainnet ./eth_mainnet.json --tag 1.0.0
```

### Deploy it

Deploying starts a job that extracts the chain into the dataset's tables. This one stops after 100 blocks, at block
24,000,099; without `--end-block`, it keeps up with the chain.

```bash
ampctl dataset deploy acme/eth_mainnet@1.0.0 --end-block 24000099
```

[Managing Datasets](/managing-amp/datasets) covers deploying, following, and stopping jobs.

### Query it

Once the job has extracted some blocks, query them:

:::terminal
```bash
ampsql 'SELECT block_num, count(*) AS transactions FROM "acme/eth_mainnet".transactions GROUP BY block_num ORDER BY block_num LIMIT 3'
```

```
+-----------+--------------+
| block_num | transactions |
+-----------+--------------+
| 24000000  | 232          |
| 24000001  | 141          |
| 24000002  | 213          |
+-----------+--------------+
```
:::
::::

## The manifest

This is the manifest the steps generate, complete. Here, each column that fits takes one line; `ampctl` writes one
value per line.

```json [eth_mainnet.json]
{
  "kind": "evm-rpc",
  "network": "mainnet",
  "start_block": 24000000,
  "finalized_blocks_only": false,
  "tables": {
    "blocks": {
      "schema": {
        "arrow": {
          "fields": [
            { "name": "_block_num", "type": "UInt64", "nullable": false },
            { "name": "_ts", "type": { "Timestamp": ["Nanosecond", "+00:00"] }, "nullable": false },
            { "name": "block_num", "type": "UInt64", "nullable": false },
            { "name": "timestamp", "type": { "Timestamp": ["Nanosecond", "+00:00"] }, "nullable": false },
            { "name": "hash", "type": { "FixedSizeBinary": 32 }, "nullable": false },
            { "name": "parent_hash", "type": { "FixedSizeBinary": 32 }, "nullable": false },
            { "name": "ommers_hash", "type": { "FixedSizeBinary": 32 }, "nullable": false },
            { "name": "miner", "type": { "FixedSizeBinary": 20 }, "nullable": false },
            { "name": "state_root", "type": { "FixedSizeBinary": 32 }, "nullable": false },
            { "name": "transactions_root", "type": { "FixedSizeBinary": 32 }, "nullable": false },
            { "name": "receipt_root", "type": { "FixedSizeBinary": 32 }, "nullable": false },
            { "name": "logs_bloom", "type": "Binary", "nullable": false },
            { "name": "difficulty", "type": { "Decimal128": [38, 0] }, "nullable": true },
            { "name": "gas_limit", "type": "UInt64", "nullable": false },
            { "name": "gas_used", "type": "UInt64", "nullable": false },
            { "name": "extra_data", "type": "Binary", "nullable": false },
            { "name": "mix_hash", "type": { "FixedSizeBinary": 32 }, "nullable": false },
            { "name": "nonce", "type": "UInt64", "nullable": false },
            { "name": "base_fee_per_gas", "type": { "Decimal128": [38, 0] }, "nullable": true },
            { "name": "withdrawals_root", "type": { "FixedSizeBinary": 32 }, "nullable": true },
            { "name": "blob_gas_used", "type": "UInt64", "nullable": true },
            { "name": "excess_blob_gas", "type": "UInt64", "nullable": true },
            { "name": "parent_beacon_root", "type": { "FixedSizeBinary": 32 }, "nullable": true },
            { "name": "requests_hash", "type": { "FixedSizeBinary": 32 }, "nullable": true },
            { "name": "block_access_list_hash", "type": { "FixedSizeBinary": 32 }, "nullable": true },
            { "name": "slot_number", "type": "UInt64", "nullable": true }
          ]
        }
      },
      "network": "mainnet",
      "sorted_by": ["block_num", "timestamp"],
      "bloom_filter_columns": [],
      "segment_selection_policy": "hash-linked"
    },
    "logs": {
      "schema": {
        "arrow": {
          "fields": [
            { "name": "_block_num", "type": "UInt64", "nullable": false },
            { "name": "_ts", "type": { "Timestamp": ["Nanosecond", "+00:00"] }, "nullable": false },
            { "name": "block_hash", "type": { "FixedSizeBinary": 32 }, "nullable": false },
            { "name": "block_num", "type": "UInt64", "nullable": false },
            { "name": "timestamp", "type": { "Timestamp": ["Nanosecond", "+00:00"] }, "nullable": false },
            { "name": "tx_hash", "type": { "FixedSizeBinary": 32 }, "nullable": false },
            { "name": "tx_index", "type": "UInt32", "nullable": false },
            { "name": "log_index", "type": "UInt32", "nullable": false },
            { "name": "address", "type": { "FixedSizeBinary": 20 }, "nullable": false },
            { "name": "topic0", "type": { "FixedSizeBinary": 32 }, "nullable": true },
            { "name": "topic1", "type": { "FixedSizeBinary": 32 }, "nullable": true },
            { "name": "topic2", "type": { "FixedSizeBinary": 32 }, "nullable": true },
            { "name": "topic3", "type": { "FixedSizeBinary": 32 }, "nullable": true },
            { "name": "data", "type": "Binary", "nullable": false }
          ]
        }
      },
      "network": "mainnet",
      "sorted_by": ["block_num", "timestamp"],
      "bloom_filter_columns": [],
      "segment_selection_policy": "hash-linked"
    },
    "transactions": {
      "schema": {
        "arrow": {
          "fields": [
            { "name": "_block_num", "type": "UInt64", "nullable": false },
            { "name": "_ts", "type": { "Timestamp": ["Nanosecond", "+00:00"] }, "nullable": false },
            { "name": "block_hash", "type": { "FixedSizeBinary": 32 }, "nullable": false },
            { "name": "block_num", "type": "UInt64", "nullable": false },
            { "name": "timestamp", "type": { "Timestamp": ["Nanosecond", "+00:00"] }, "nullable": false },
            { "name": "tx_index", "type": "UInt32", "nullable": false },
            { "name": "tx_hash", "type": { "FixedSizeBinary": 32 }, "nullable": false },
            { "name": "to", "type": { "FixedSizeBinary": 20 }, "nullable": true },
            { "name": "nonce", "type": "UInt64", "nullable": false },
            { "name": "gas_price", "type": { "Decimal128": [38, 0] }, "nullable": true },
            { "name": "gas_limit", "type": "UInt64", "nullable": false },
            { "name": "value", "type": "Utf8", "nullable": false },
            { "name": "input", "type": "Binary", "nullable": false },
            { "name": "r", "type": { "FixedSizeBinary": 32 }, "nullable": false },
            { "name": "s", "type": { "FixedSizeBinary": 32 }, "nullable": false },
            { "name": "v_parity", "type": "Boolean", "nullable": false },
            { "name": "chain_id", "type": "UInt64", "nullable": true },
            { "name": "gas_used", "type": "UInt64", "nullable": false },
            { "name": "type", "type": "Int32", "nullable": false },
            { "name": "max_fee_per_gas", "type": { "Decimal128": [38, 0] }, "nullable": true },
            { "name": "max_priority_fee_per_gas", "type": { "Decimal128": [38, 0] }, "nullable": true },
            { "name": "max_fee_per_blob_gas", "type": { "Decimal128": [38, 0] }, "nullable": true },
            { "name": "from", "type": { "FixedSizeBinary": 20 }, "nullable": false },
            { "name": "status", "type": "Boolean", "nullable": false },
            { "name": "state_root", "type": { "FixedSizeBinary": 32 }, "nullable": true },
            {
              "name": "access_list",
              "type": {
                "List": {
                  "name": "element",
                  "data_type": {
                    "Struct": [
                      {
                        "name": "address",
                        "data_type": { "FixedSizeBinary": 20 },
                        "nullable": false,
                        "dict_id": 0,
                        "dict_is_ordered": false,
                        "metadata": {}
                      },
                      {
                        "name": "storage_keys",
                        "data_type": {
                          "List": {
                            "name": "element",
                            "data_type": { "FixedSizeBinary": 32 },
                            "nullable": false,
                            "dict_id": 0,
                            "dict_is_ordered": false,
                            "metadata": {}
                          }
                        },
                        "nullable": false,
                        "dict_id": 0,
                        "dict_is_ordered": false,
                        "metadata": {}
                      }
                    ]
                  },
                  "nullable": false,
                  "dict_id": 0,
                  "dict_is_ordered": false,
                  "metadata": {}
                }
              },
              "nullable": true
            },
            {
              "name": "blob_versioned_hashes",
              "type": {
                "List": {
                  "name": "element",
                  "data_type": { "FixedSizeBinary": 32 },
                  "nullable": false,
                  "dict_id": 0,
                  "dict_is_ordered": false,
                  "metadata": {}
                }
              },
              "nullable": true
            },
            {
              "name": "authorization_list",
              "type": {
                "List": {
                  "name": "element",
                  "data_type": {
                    "Struct": [
                      {
                        "name": "chain_id",
                        "data_type": "UInt64",
                        "nullable": false,
                        "dict_id": 0,
                        "dict_is_ordered": false,
                        "metadata": {}
                      },
                      {
                        "name": "address",
                        "data_type": { "FixedSizeBinary": 20 },
                        "nullable": false,
                        "dict_id": 0,
                        "dict_is_ordered": false,
                        "metadata": {}
                      },
                      {
                        "name": "nonce",
                        "data_type": "UInt64",
                        "nullable": false,
                        "dict_id": 0,
                        "dict_is_ordered": false,
                        "metadata": {}
                      },
                      {
                        "name": "y_parity",
                        "data_type": "Boolean",
                        "nullable": false,
                        "dict_id": 0,
                        "dict_is_ordered": false,
                        "metadata": {}
                      },
                      {
                        "name": "r",
                        "data_type": { "FixedSizeBinary": 32 },
                        "nullable": false,
                        "dict_id": 0,
                        "dict_is_ordered": false,
                        "metadata": {}
                      },
                      {
                        "name": "s",
                        "data_type": { "FixedSizeBinary": 32 },
                        "nullable": false,
                        "dict_id": 0,
                        "dict_is_ordered": false,
                        "metadata": {}
                      }
                    ]
                  },
                  "nullable": false,
                  "dict_id": 0,
                  "dict_is_ordered": false,
                  "metadata": {}
                }
              },
              "nullable": true
            }
          ]
        }
      },
      "network": "mainnet",
      "sorted_by": ["block_num", "timestamp"],
      "bloom_filter_columns": [],
      "segment_selection_policy": "hash-linked"
    }
  }
}
```

The [Manifest Reference](/datasets/manifest-reference#raw-datasets) describes each field, and
[EVM RPC Tables](/datasets/tables/evm-rpc) each column. Generate the manifest rather than editing its tables: a raw
dataset's tables must be the ones its kind extracts. The fields you choose are the ones below.

## Start block

`start_block` is the first block the dataset covers. Amp never extracts the blocks before it, so they're not in the
dataset and you can't query them. It defaults to `0`, the chain's first block.

Set it when a chain's early history doesn't matter to you: it shortens the time the first extraction takes, costs
fewer requests to your provider, and takes less storage. For example:

* **From a contract's deployment**: the contract you're interested in was deployed at block 18,000,000, so nothing
  before it concerns you.
* **Recent data only**: a dashboard that shows the last months doesn't need years of history.

Leave it at `0` when you need the whole history, or aren't sure yet. A start block is part of the manifest, so changing
it means registering a new version. Lowering it later means extracting the blocks you skipped.

A start block is only a lower bound: it can't stop the dataset at a block, or skip a range in the middle. To stop at a
block, deploy with `--end-block`. And it applies to every table of the dataset.

:::warning
Don't set a start block on chains whose block numbers have gaps, like Solana, where a slot can have no block. Amp
considers the dataset synced only once it has the start block itself, which may never exist.
:::

## Finalized blocks

`finalized_blocks_only` makes the dataset extract only the blocks the chain considers final: blocks a
[reorg](/get-started/concepts/reorgs) can no longer replace. For EVM chains, Amp asks the endpoint for its `finalized`
block instead of its latest one.

Your data then never changes once it's extracted, at the cost of freshness: on Ethereum, final blocks lag the latest
by about 15 minutes. Without it, which is the default, Amp extracts blocks as soon as the chain produces them, and
replaces any a reorg orphans.

Ethereum Beacon, Solana, and Stellar datasets extract only final blocks either way: see
[Ethereum Beacon](/providers/eth-beacon), [Solana](/providers/solana#commitment), and
[Stellar](/providers/stellar#ledgers).

## Reorgs

A reorg, short for chain reorganization, replaces some of a chain's latest blocks with others. Amp records each
block's hash and its parent's hash, and checks that each block it extracts links to the one before. When a new block
doesn't, Amp detects the reorg, extracts the new blocks, and serves them in place of the orphaned ones. Queries never
return both. [Reorgs](/get-started/concepts/reorgs) explains how streaming queries tell clients which data to replace.

### Segment selection

Each table's `segment_selection_policy` says how Amp links its blocks:

* `hash-linked`, the default: by hash and parent hash, as above.
* `ordered`: by block number alone, ignoring hashes. Amp then can't detect a reorg, and never re-extracts data
  because its source changed. Use it only for sources whose data never changes once published, or that resolve reorgs
  before Amp reads them, like some [Phaser](/providers/phaser) bridges.

## How extraction works

These behaviors apply to every kind of raw dataset:

* **Blocks in order**: a job extracts the blocks it's missing in order, and writes a block to every table before it
  moves on to the next. A table never has a block the others are missing.
* **Several endpoints**: when you register several providers of the dataset's kind and network, a job spreads its
  requests over all of them, sending more to the faster ones and fewer to those that fail. Their names don't set a
  priority. Use `ampctl dataset deploy --provider` to use only one.
* **Retries**: when a request fails for a reason that may pass, like a timeout or a rate limit, Amp retries it on
  another endpoint, up to three times within a deadline. When that fails too, or the error is permanent, like invalid
  data, the job fails, and Amp retries the job as its [retry strategy](/managing-amp/jobs/retries) says.

## Network names

A raw dataset's `network` names the network it extracts, like `mainnet`, `base`, or `solana-mainnet`: lowercase
letters, digits, and hyphens. Use the names in
[The Graph's networks registry](https://github.com/graphprotocol/networks-registry/blob/main/docs/networks-table.md), so
datasets and providers from different people match. A dataset extracts through the providers whose `kind` and `network`
match its own, so the two must use the same name.

Give each chain its own name, across kinds. A query or derived table that reads two chains with the same name, like
Ethereum and its Beacon chain both named `mainnet`, treats them as one network, with one blocks table.

## Errors

### `No provider found for kind 'evm-rpc' and network 'mainnet'`

The job couldn't find a usable provider for the dataset's kind and network. Either none is registered (list them with
`ampctl provider list`), its `network` doesn't match the dataset's exactly, or its configuration refers to an
environment variable that isn't set where `ampd` runs. In that last case, `ampd` also logs
`environment variable substitution failed for provider, skipping`: set the variable and restart `ampd`.