Running a Reth Archive Node

Crab Node Goes Brrr

You all know my preference is to operate my own node. Ethereum mainnet is the most approachable blockchain for self-hosting, and it is where I spend most of my time.

I’ve written two guides for setting up a Geth node and setting up an Arbitrum node.

A recent offering is Reth, developed by Paradigm and revealed in December 2022.

Ethereum nodes come in two main “flavors”: full and archive. A full node generally maintains a full state record for the last 128 blocks, and selectively prunes historical data as new blocks are added to the canonical chain. An archive node maintains this state record for all blocks, never pruning old block data.

Based on node usage metrics by Ethernodes, The most popular full node client is Geth, followed by Nethermind, Erigon, and Besu.

Geth is generally optimized for operation as a full node, and requires a huge amount of storage for use as an archive node. Besu is similar, especially with their “Bonsai Tries” storage layout. This optimization offers fast performance and low storage use for pruned blocks, but degrades quickly when used as an archive.

Erigon is the typical choice for archive node use because they have largely solved the problem of storage. Where a Geth archive node would consume 12+ TB, Erigon consumes ~2 TB.

Reth takes the database and staged sync design from Erigon and extends it further by writing the client in Rust and making the components modular. The various modules (called crates) can be used independently of the client, and extensions can be written for Reth without the same difficulty as integrating changes into a monolithic client like Geth.

MEV hero libevm published a guide to extending Reth with a new JSON-RPC API endpoint recently.

I have been running a Reth node for a few weeks. I ran into many problems along the way and filed some bug reports, but I’ve generally sorted them out to the point that others can copy-paste the setup and mostly use Reth without issues.

Warnings and Disclaimers

Reth is labeled as alpha quality software, and the Reth Github disclaims that Reth is not ready for production use.

I agree! Reth is not a drop-in replacement for your existing node if you are doing production work, operating as a validator, or acting as your blockchain connection for critical work.

It has some lingering issues which I’ll outline below, but the team’s development pace is good and they are responsive on Github and Twitter. There are also rumblings on Twitter of the first Reth beta release at the end of this month.

Problems

There are a handful of items that blocked and frustrated me along the way.

Lighthouse Consensus Degradation

Reth plays poorly with popular consensus client Lighthouse, which I used in my previous Geth guide. After my Reth node reached sync, it would frequently stall without receiving any new blocks from Lighthouse. The Lighthouse logs revealed that it was attempting to contact the execution client (Reth) and timing out. Sometimes a node restart would “un-stick” the two clients, but eventually they would start timing out again. Eventually, Reth would settle into a steady 20-30 blocks gap behind the current height and never catch up.

I observed this behavior on Reth 0.1.0-alpha.8, alpha.9, and alpha.10 (the current release at time of writing).

Luckily, a comment on Issue 4460 (error: Api { error: HttpClient(url: http://127.0.0.1:8551/, kind: timeout (Prysm, Lighthouse)) on the Reth Github alerted me to the presence of another consensus client called Nimbus that apparently performed better with Reth.

I tried Nimbus and confirmed that it maintains sync with the live chain, so I have used it in this guide.

I do not know if the issue is caused by Reth or Lighthouse, but I trust it will be solved eventually. A consensus client is easy to switch, so you are free to try different options as you see fit.

Sync Time

A benchmark shared on Twitter shows a Reth node syncing in two days on a bare metal server.

My Reth node took decidedly longer, roughly 10 days. My initial setup was Reth and Lighthouse, so the slow sync could have been related to the degradation mentioned above. I will never know because there’s no way I’m going to do a full sync again!

The key bottleneck with sync time is storage performance. The system will hit the storage hard during the sync, and whenever large reads from the archived state are required. An SSD is de facto required for good performance.

I have a 4 TB NVMe WD_Black SN850X (not an affiliate link) in my Linux development workstation. It is whole-disk encrypted via LUKS, with a BTRFS filesystem on top. I use the default Fedora BTRFS mount option for zstd level 1 compression.

The unfortunate side effect of LUKS plus BTRFS compression is a high load on the CPU. Both the encryption and the compression are done in software, so one core of my CPU is often maxed while heavy disk I/O is occuring.

I consulted the fantastic Great and Less Great SSDs for Ethereum Nodes guide on Github to confirm that my hardware is robust, so I believe that the NVMe is not bottlenecking my sync time. As one more data point, Geth synced in a day despite writing a third of Reth’s data (~700 GB for Geth vs ~2 TB for Reth).

I suspect that the two day target is realistic if you are not doing disk encryption or compression. But again, I’m not going to test! Just be aware that your sync will take a long time if you’re using a similar setup.

Event Subscription Performance vs. Geth

While watching new blocks and events via websocket, Reth is delayed by 1-2s compared to Geth. I frequently observe new Geth blocks arriving within 1-2s of the block timestamp. Reth blocks typically arrive within 3-4s.

This is likely an item that will be corrected in a later release. However, if you intend to pair Reth with a low latency bot, be aware that you will pay a penalty.

Missing RPC Methods

If you work with raw transactions (many of my examples do), you will quickly discover that the eth_getRawTransactionByHash endpoint is not available in Reth. The RPC Tracker shows Reth’s progress at implementing the various methods, but eth_getRawTransactionByHash does not appear. The similar method debug_getRawTransaction does appear, but I found that it returned slightly different results from an eth_getRawTransactionByHash call to my Geth node. I will look more closely at this later, since it’s either a bug or a gap in my understanding of the RPC method.

Also, the very popular trace_filter method is not yet implement, but development is ongoing.

The Setup

With all that out of the way, if you’re still interested in running a Reth node, you can use my setup. I prefer to use Docker whenever possible, and we are in luck here. Both Reth and Nimbus have official container packages. The Nimbus client is available on Docker Hub, and the Reth client is available on Github Container Registry.

docker-compose.yml

version: "2"

services:
  reth:
    image: ghcr.io/paradigmxyz/reth:latest
    container_name: reth
    restart: unless-stopped
    #network_mode: host <----- might be necessary for Mac / Windows
    ports:
      - "8543:8543"
      - "8544:8544"
      - "30304:30304/tcp"
      - "30304:30304/udp"
    volumes:
      - reth_data:/data
      - reth_jwt:/jwt:ro
    command:
      - node
      - --chain=mainnet
      - --datadir=/data
      - --authrpc.jwtsecret=/jwt/jwt.hex
      - --authrpc.addr=0.0.0.0
      - --authrpc.port=8551
      - --http
      - --http.api=debug,eth,net,trace,txpool,web3,rpc,reth,ots
      - --http.addr=0.0.0.0
      - --http.port=8543
      - --http.corsdomain="*"
      - --ws
      - --ws.api=debug,eth,net,trace,txpool,web3,rpc,reth,ots
      - --ws.addr=0.0.0.0
      - --ws.port=8544
      - --ws.origins="*"

  nimbus:
    image: statusim/nimbus-eth2:multiarch-latest
    container_name: nimbus
    restart: unless-stopped
    ports:
      - "9001:9001/tcp"
      - "9001:9001/udp"
    volumes:
      - nimbus_data:/data
      - reth_jwt:/jwt/reth:ro
    command:
      - --network=mainnet
      - --el=http://reth:8551#jwt-secret-file=/jwt/reth/jwt.hex
      - --enr-auto-update:true

volumes:
  reth_data:
  reth_jwt:
  nimbus_data:

Notes on Networking Setup

My Reth node runs in parallel with Geth, so I have chosen port numbers here that do not conflict with those defaults (30303 for p2p, 8545 for HTTP, 8546 for WS). Adjust as required, and don’t forget to forward those ports in your firewall and router if you’re behind NAT.

Before Startup

First, you should generate a new JWT key for authentication between Reth and Nimbus. You can use the script provided in the Reth repo, or generate your own 32-byte hex key (64 characters, any mixture of 0-9, lowercase a-f). I encourage you to generate your own, but here is an example JWT key file for illustration:

jwt.hex

d38d658d68167fc066eb82dd3dc3a65b38264108302dabab74c761d10a4b59b0

Use the following comment to create the containers, volumes, and networks without starting them:

root@pop-os:~/reth# docker-compose up --no-start

Docker will add a prefix to each volume when you first create it, depending on the path of where the docker-compose.yml file is located. I had mine saved in ~/reth, so Docker built three volumes:

• reth_reth_jwt

• reth_reth_data

• reth_nimbus_data

If you use different paths, you should expect to find different volumes. Please adapt as required for your setup.

Now copy the JWT key file into the appropriately volume, which will be in /var/lib/docker/volumes/[prefix]_reth_jwt/_data:

root@pop-os:~/reth# mv jwt.hex /var/lib/docker/volumes/reth_reth_jwt/_data/

Once your key file is appropriately located, start up the stack and observe the container logs for any errors:

root@pop-os:~/reth# docker-compose up

You’ll observe output similar to this:

nimbus    | INF 2023-10-21 17:55:46.245+00:00 Launching beacon node                      topics="beacnde" version=v23.10.0-8b07f4-stateofus bls_backend=BLST cmdParams="@[\"--network=mainnet\", \"--el=http://reth:8551#jwt-secret-file=/jwt/reth/jwt.hex\", \"--enr-auto-update:true\"]" config="(configFile: none(InputFile), logLevel: \"INFO\", logStdout: auto, logFile: none(OutFile), eth2Network: some(\"mainnet\"), dataDir: /home/user/.cache/nimbus/BeaconNode, validatorsDirFlag: none(InputDir), verifyingWeb3Signers: @[], provenBlockProperties: @[], web3Signers: @[], web3signerUpdateInterval: 3600, secretsDirFlag: none(InputDir), walletsDirFlag: none(InputDir), eraDirFlag: none(InputDir), web3ForcePolling: none(bool), web3Urls: @[], elUrls: @[(url: \"http://reth:8551\", jwtSecret: none(string), jwtSecretFile: some(/jwt/reth/jwt.hex), roles: none(EngineApiRoles))], noEl: false, optimistic: none(bool), requireEngineAPI: none(bool), nonInteractive: false, netKeyFile: \"random\", netKeyInsecurePassword: false, agentString: \"nimbus\", subscribeAllSubnets: false, slashingDbKind: v2, numThreads: 0, jwtSecret: none(InputFile), cmd: noCommand, runAsServiceFlag: false, bootstrapNodes: @[], bootstrapNodesFile: , listenAddress: 0.0.0.0, tcpPort: 9000, udpPort: 9000, maxPeers: 160, hardMaxPeers: none(int), nat: (hasExtIp: false, nat: NatAny), enrAutoUpdate: true, enableYamux: false, weakSubjectivityCheckpoint: none(Checkpoint), syncLightClient: true, trustedBlockRoot: none(Hash256), finalizedCheckpointState: none(InputFile), genesisState: none(InputFile), genesisStateUrl: none(Uri), finalizedDepositTreeSnapshot: none(InputFile), finalizedCheckpointBlock: none(InputFile), nodeName: \"\", graffiti: none(GraffitiBytes), strictVerification: false, stopAtEpoch: 0, stopAtSyncedEpoch: 0, metricsEnabled: false, metricsAddress: 127.0.0.1, metricsPort: 8008, statusBarEnabled: true, statusBarContents: \"peers: $connected_peers;finalized: $finalized_root:$finalized_epoch;head: $head_root:$head_epoch:$head_epoch_slot;time: $epoch:$epoch_slot ($slot);sync: $sync_status|ETH: $attached_validators_balance\", rpcEnabled: none(bool), rpcPort: none(Port), rpcAddress: none(ValidIpAddress), restEnabled: false, restPort: 5052, restAddress: 127.0.0.1, restAllowedOrigin: none(string), restCacheSize: 3, restCacheTtl: 60, restRequestTimeout: 0, restMaxRequestBodySize: 16384, restMaxRequestHeadersSize: 128, keymanagerEnabled: false, keymanagerPort: 5052, keymanagerAddress: 127.0.0.1, keymanagerAllowedOrigin: none(string), keymanagerTokenFile: none(InputFile), lightClientDataServe: true, lightClientDataImportMode: only-new, lightClientDataMaxPeriods: none(uint64), inProcessValidators: true, discv5Enabled: true, dumpEnabled: false, directPeers: @[], doppelgangerDetection: true, syncHorizon: 50, terminalTotalDifficultyOverride: none(string), validatorMonitorAuto: true, validatorMonitorPubkeys: @[], validatorMonitorDetails: false, validatorMonitorTotals: none(bool), safeSlotsToImportOptimistically: none(uint16), suggestedFeeRecipient: none(Address), suggestedGasLimit: 30000000, payloadBuilderEnable: false, payloadBuilderUrl: \"\", localBlockValueBoost: 0, historyMode: prune, trustedSetupFile: none(string), bandwidthEstimate: none(Natural), forkChoiceVersion: none(ForkChoiceVersion))"
reth      | 2023-10-21T17:55:46.260587Z  INFO reth::cli: reth 0.1.0-alpha.10 (1b16d80) starting
nimbus    | INF 2023-10-21 17:55:46.247+00:00 Threadpool started                         topics="beacnde" numThreads=16
reth      | 2023-10-21T17:55:46.260766Z  INFO reth::cli: Configuration loaded path="/data/reth.toml"
reth      | 2023-10-21T17:55:46.260778Z  INFO reth::cli: Opening database path="/data/db"
reth      | 2023-10-21T17:55:46.270082Z  INFO reth::cli: Database opened
reth      | 2023-10-21T17:55:46.270158Z  INFO reth::cli: Pre-merge hard forks (block based):
reth      | - Frontier                         @0
reth      | - Homestead                        @1150000
reth      | - Dao                              @1920000
reth      | - Tangerine                        @2463000
reth      | - SpuriousDragon                   @2675000
reth      | - Byzantium                        @4370000
reth      | - Constantinople                   @7280000
reth      | - Petersburg                       @7280000
reth      | - Istanbul                         @9069000
reth      | - MuirGlacier                      @9200000
reth      | - Berlin                           @12244000
reth      | - London                           @12965000
reth      | - ArrowGlacier                     @13773000
reth      | - GrayGlacier                      @15050000
reth      | Merge hard forks:
reth      | - Paris                            @58750000000000000000000 (network is not known to be merged)
reth      | 
reth      | Post-merge hard forks (timestamp based):
reth      | - Shanghai                         @1681338455
reth      | 
nimbus    | INF 2023-10-21 17:55:46.318+00:00 Loading block DAG from database            topics="beacnde" path=/home/user/.cache/nimbus/BeaconNode/db
reth      | 2023-10-21T17:55:46.337550Z  INFO reth::cli: Transaction pool initialized
reth      | 2023-10-21T17:55:46.337587Z  INFO reth::cli: Connecting to P2P network
reth      | 2023-10-21T17:55:46.337729Z  INFO net::peers: Loading saved peers file=/data/known-peers.json
reth      | 2023-10-21T17:55:46.338332Z  INFO reth::cli: Connected to P2P network peer_id=0xa5ea…2b07 local_addr=0.0.0.0:30303 enode=enode://a5eac151242d605a0fed7fc8fb70a4c01efdb3bec362af50cc34f8518c5433cd6edd2997f9a0728e7a4cdc9ab321a84dd752702e41518adb830129601d2a2b07@127.0.0.1:30303
reth      | 2023-10-21T17:55:46.338514Z  INFO reth::cli: Consensus engine initialized
reth      | 2023-10-21T17:55:46.338536Z  INFO reth::cli: Engine API handler initialized
reth      | 2023-10-21T17:55:46.341248Z  INFO reth::cli: RPC auth server started url=0.0.0.0:8551
reth      | 2023-10-21T17:55:46.341444Z  INFO reth::cli: RPC IPC server started url=/tmp/reth.ipc
reth      | 2023-10-21T17:55:46.341453Z  INFO reth::cli: RPC HTTP server started url=0.0.0.0:8543
reth      | 2023-10-21T17:55:46.341455Z  INFO reth::cli: RPC WS server started url=0.0.0.0:8544
reth      | 2023-10-21T17:55:46.341464Z  INFO reth::cli: Starting consensus engine
nimbus    | INF 2023-10-21 17:55:46.342+00:00 Block DAG initialized                      head=4d611d5b:0 finalizedHead=4d611d5b:0 tail=4d611d5b:0 backfill="(0, \"00000000\")" loadDur=6ms613us700ns summariesDur=8ms192us687ns finalizedDur=8ms817us400ns frontfillDur=40ns keysDur=53us341ns
nimbus    | INF 2023-10-21 17:55:46.342+00:00 Generating new networking key              topics="networking" network_public_key=08021221033dd83c6d75b3947f9475dc035a872c20c4d1e53cc2a9e9969c8f74c81810c7af network_peer_id=16Uiu2HAmGpQTfCD8MUsPZJeAWJsNfabc5EYCz8piGb1kVWNaEgb8
reth      | 2023-10-21T17:55:49.340144Z  INFO reth::cli: Status connected_peers=0 latest_block=0
nimbus    | WRN 2023-10-21 17:55:50.363+00:00 UPnP/NAT-PMP not available                 topics="eth net nat"
nimbus    | INF 2023-10-21 17:55:50.364+00:00 Discovery ENR initialized                  topics="eth p2p discv5" enrAutoUpdate=true seqNum=1 ip=none(ValidIpAddress) tcpPort=some(9000) udpPort=some(9000) customEnrFields="@[eth2:0xBBA4DA9603000000FFFFFFFFFFFFFFFF, attnets:0x0000000000000000]" uri=enr:-Kq4QOXU9m0RYgV85w0RSqnFquFpVvzI_vHkHbkP_BrwRdcxHJQVlNo0IufRjKOyBoWn1HZ_nauNv4UaxvnZT6TT7YEBh2F0dG5ldHOIAAAAAAAAAACEZXRoMpC7pNqWAwAAAP__________gmlkgnY0iXNlY3AyNTZrMaEDPdg8bXWzlH-UddwDWocsIMTR5TzCqemWnI90yBgQx6-DdGNwgiMog3VkcIIjKA
nimbus    | NOT 2023-10-21 17:55:50.364+00:00 No external IP provided for the ENR, this node will not be discoverable until the ENR is updated with the discovered external IP address topics="eth p2p discv5"
nimbus    | INF 2023-10-21 17:55:50.364+00:00 Loading slashing protection database (v2)  topics="beacnde" path=/home/user/.cache/nimbus/BeaconNode/validators
nimbus    | INF 2023-10-21 17:55:50.377+00:00 Initializing fork choice                   topics="beacnde" unfinalized_blocks=0
nimbus    | INF 2023-10-21 17:55:50.377+00:00 Fork choice initialized                    topics="beacnde" justified=0:00000000 finalized=0:00000000
nimbus    | INF 2023-10-21 17:55:50.377+00:00 Loading validators                         topics="beacval" validatorsDir=/home/user/.cache/nimbus/BeaconNode/validators keystore_cache_available=true
nimbus    | NOT 2023-10-21 17:55:50.377+00:00 Starting beacon node                       topics="beacnde" version=v23.10.0-8b07f4-stateofus nimVersion=1.6.14 enr=enr:-Kq4QOXU9m0RYgV85w0RSqnFquFpVvzI_vHkHbkP_BrwRdcxHJQVlNo0IufRjKOyBoWn1HZ_nauNv4UaxvnZT6TT7YEBh2F0dG5ldHOIAAAAAAAAAACEZXRoMpC7pNqWAwAAAP__________gmlkgnY0iXNlY3AyNTZrMaEDPdg8bXWzlH-UddwDWocsIMTR5TzCqemWnI90yBgQx6-DdGNwgiMog3VkcIIjKA peerId=16Uiu2HAmGpQTfCD8MUsPZJeAWJsNfabc5EYCz8piGb1kVWNaEgb8 timeSinceFinalization=150w4d5h55m27s377ms308us281ns head=4d611d5b:0 justified=0:00000000 finalized=0:00000000 finalizedHead=4d611d5b:0 SLOTS_PER_EPOCH=32 SECONDS_PER_SLOT=12 SPEC_VERSION=1.4.0-beta.2-hotfix dataDir=/home/user/.cache/nimbus/BeaconNode validators=0
nimbus    | INF 2023-10-21 17:55:50.380+00:00 Listening to incoming network requests     topics="beacnde"
nimbus    | INF 2023-10-21 17:55:50.380+00:00 Starting discovery node                    topics="eth p2p discv5" node=22*5044dc:unaddressable bindAddress=0.0.0.0:9000
nimbus    | INF 2023-10-21 17:55:50.381+00:00 Starting execution layer deposit syncing   topics="elmon" contract=0x00000000219ab540356cbb839cbe05303d7705fa
nimbus    | INF 2023-10-21 17:55:50.382+00:00 Connection attempt started                 topics="elmon"
nimbus    | NOT 2023-10-21 17:55:50.382+00:00 Starting light client                      topics="lightcl" trusted_block_root=none(Hash256)

[...]

If everything looks correct, stop the stack with CTRL+C and restart it with the detach option:

root@pop-os:~/reth# docker-compose up -d

Running Updated Builds

The Reth team makes frequent fixes on Github, but they generally release updates to the Container Registry every few weeks. If you want to run a current version, you can compile the latest binary and provide a Docker volume mount that overrides the reth binary inside the container.

Pull the repo and follow the instructions in the Reth documentation’s Build From Source page.

I keep my remote code archives in a directory called code, so my reth binary ultimately lives in /home/btd/code/reth/target/release/reth

To override the binary in the container, add one line in the volumes: section of the Reth service, overlaying the compiled binary over the container’s provided binary:

version: "2"

services:
  reth:
    [ ... ]    
    volumes:
      - reth_data:/data
      - reth_jwt:/jwt:ro
      - /home/btd/code/reth/target/release/reth:/usr/local/bin/reth:ro
    [ ... ]

The newer version will be visible in the container logs when you start, and you can verify it against the newest commit when you ran the git pull (3a8fe5a7 at the time of writing). The commit will be shown after the version string, as higlighted below:

root@pop-os:~/reth# docker-compose logs -f reth
2023-10-21T18:10:57.552028Z  INFO reth::cli: reth 0.1.0-alpha.10 (3a8fe5a7) starting
2023-10-21T18:10:57.552271Z  INFO reth::cli: Configuration loaded path="/data/reth.toml"
2023-10-21T18:10:57.552284Z  INFO reth::cli: Opening database path="/data/db"
2023-10-21T18:10:57.555982Z  INFO reth::cli: Database opened
2023-10-21T18:10:57.557220Z  INFO reth::cli: Pre-merge hard forks (block based):
- Frontier                         @0
- Homestead                        @1150000
- Dao                              @1920000
- Tangerine                        @2463000

[ ... ]