This blog post reveals a threat to the Ethereum network that was present from the merge until the Dencun hard fork.
Background
Prior to the merge, different message size limits for RPC communication were defined to protect clients from denial-of-service (DOS) attacks. These limits, applied to messages received via HTTP endpoints, were carried over to the engine API, which plays a crucial role in connecting execution and consensus layer clients during block production. Due to the involvement of the engine API in block production, it became possible to produce blocks that exceeded the RPC size limits of some clients, but remained within the acceptable range for others.
If an attacker creates a message that exceeds the client’s size limit at the lowest setting, while still meeting gas limit requirements, and then waits for a block to be produced, this could result in a situation where some clients consider the block valid, while others reject it, issuing an HTTP error code “413: Content Too Large”.
Impact
An attacker able to create these messages could force the majority of nodes (=geth) to reject blocks that a minority would accept. These blocks would then be duplicated and the proposer would lose his rewards.
At first, we thought that it was only possible to create these blocks using builders or a modified version of a client. Geth has a built-in transaction limit of 128KB, meaning that a large transaction like the one we are talking about would not end up in a geth node’s transaction pools. It was possible, however, to trigger the limit by having a client with a higher limit propose the block and the CL request validation of this proposed larger block.
We proposed a solution to temporarily lower the RPC limit on all clients to the lowest value (5MB). This would render the block invalid and an attacker would be very limited in the chaos he could cause in the network since the majority of nodes would reject his blocks.
However, on February 7th, we discovered that it was possible to create a block that would reach the 5MB limit with a set of transactions below the 128KB limit and not exceeding 30 million gas.
This is a bigger problem because we realized that an attacker could create a bunch of highly rewarding transactions and send them to the network. Since he pays more than everyone else in the mempool, every node (even geth nodes) would include the attack transactions in their block, creating a block that would not be accepted by the majority of the network, leading to many forks (all deemed valid by the minority nodes) and the chain would continue to reorganize itself over and over again.
Later, on February 7, we came to the conclusion that increasing RPC limits by everyone would be the safest alternative.
Chronology
- 2024-02-06 13:00: Toni (EF), Pari (EF), and Justin (Besu) attempt to submit a specifically shredded transaction to the network. The transaction contributes blocks up to 2.7 MB when compressed quickly.
- 2024-02-06 13:25: Pari receives errors from his local Geth node although the transaction should be valid.
- 2024-02-06 15:14: Justin successfully put the transaction into a block and submitted it via the Besu client.
- 2024-02-06 20:46: Sam (EF) alerts Pari (special thanks to mysticryuujin on X), Toni and Alex about some troubled Sepolia nodes.
- 2024-02-06 21:05: The team checks with Marius from Geth and confirms the bug.
- 2024-02-06 21:10 : The gang gets together to debug it
- 2024-02-07 23:40 : We have decided for all customers to limit their RPC request limit to 5MB
- 2024-02-07 6:40: We discovered that there might be a bigger problem and that the attack can be executed with transactions smaller than 128KB.
- 2024-02-07 10:00: We have decided for all customers to increase the RPC request limit.
- 2024-02-07 21:00: Fix has been merged into geth.
- 02/09/2024: Geth has been released
While Geth is the only client affected by this bug, other clients have also updated their default settings to be safe from this attack even if gas limits are increased. Client teams have reported that the following updates have the safe RPC limits:
Geth: v1.13.12
Void: v1.25.4
Version: 24.1.2
Erigon: v2.58.0
Reth: v0.1.0-alpha.18
