Uniswap — Transaction Prediction (Part VI)

Pool Creation and Tracking

Abstractions are good — they increase repeatability and decrease maintenance effort, but they require a lot of front-loaded effort.

You’ll often find yourself playing abstraction whack-a-mole when you start going down this path. There is danger in over-abstraction, where doing anything useful requires interacting with several different API layers. And there is danger in under-abstraction, where you still find yourself picking little pieces and parts from various objects and connecting them directly.

The various projects I’ve written and shared generally follow a pattern of increasing abstraction. Early bots did a lot of stuff by hand and they were very cumbersome to update. Successive releases introduced many improvements, like generalized pool helpers, arbitrage path generators, standardized exceptions, and more robust use of structured data like JSON.

Most of my effort lately has been to develop transaction simulation abstractions, which naturally extend into a mempool-aware backrun bot. But since a transaction is very complex, it requires careful attention to detail when connecting the various components. To properly simulate a transaction that you observe in the mempool, at bare minimum, you need a way to model the liquidity pools that the transaction will go through. We have many tools to do this, but it can be very difficult to keep all of the various components in order.

Pool Collection

I covered the Pool Manager concept earlier. It allows us to directly (and indirectly) abstract away the process of creating pool helpers. This is work the execution bot had to do before, but now it can be decoupled.

However, it does not end here. The execution bot still needs to be aware of the set of pool helpers. It’s one thing to create a pool, but another to maintain that pool, keep it updated, and to make it available to other processes when needed.

This spurs the desire for a general-purpose pool collection helper. This collection will be used to maintain a record of all pools available to the bot, and to share these pools through a simple interface.

If we want to track this pool set, we need to answer some important questions:

What is Responsible for Tracking the Pools?

Answered simply by the AllPools class, described in the previous post.

What Adds New Pools to the Set?

This question is more nuanced.

The temptation is to make the specific pool managers responsible for adding the pools to the set. This was my first thought. It made a lot of sense because the pool manager is a highly “user-facing” class, and already tracks its known pools. However if I make the manager responsible for interfacing with the total pool set, that requires me to create a pool manager for everything. For something like Uniswap V2 or V3, that’s fine because the number of pools is huge and I already want to use the manager. But take something like Camelot DEX (a hybrid Solidly/V2-fork). Do I want to create a pool manager for this? Maybe! But also maybe not.

I have no interest in maintaining managers for dozens of random forks across various networks, so I scrapped the idea.

Instead, I’ve chosen to move a level down to the pool helper class itself. Right now the degenbot code base has two parent level pool helper classes: LiquidityPool and V3LiquidityPool. I can extend these classes so they register themselves with the total pool set in their constructor. Now I can guarantee that a new pool is properly added whenever it is built, regardless of who starts the process.

Python 101: A “method” is a function that is bound to a “class”. A class is the building block for an “object”. Whenever a new object is “instantiated”, the Python interpreter builds it by creating a new object with two special methods. First it runs __new__, which reserves a memory address and creates a reference to the object (self is the accepted label). Then it runs the __init__ method using that self reference. The __init__ method is commonly called the “constructor”, and does the work of setting values and attributes for the object. The __new__ method is typically omitted, because most objects are created in this way. You can override this if you need to.

For example, let’s extend the V3 helper by adding a few instructions to the end of the constructor. First we need to import the AllPools class into our class definition file:

v3_liquidity_pool.py

[...]
from degenbot.manager import (
    AllPools, 
    Erc20TokenHelperManager, # <--- ADDED
)
[...]

And at the end of the constructor, create an instance of AllPools and add the object (via the self shorthand), to the tracked set:

v3_liquidity_pool.py

[...]
AllPools(chain.id)[self.address] = self
[...]

Now let’s launch a console and test it using the familiar WBTC/WETH V3 pool:

(.venv) [devil@dev bots]$ brownie console --network mainnet-local
Brownie v1.19.3 - Python development framework for Ethereum

BotsProject is the active project.
Brownie environment is ready.

>>> import degenbot as bot
>>> lp = bot.V3LiquidityPool('0xCBCdF9626bC03E24f779434178A73a0B4bad62eD')
• WBTC (Wrapped BTC)
• WETH (Wrapped Ether)
WBTC-WETH (V3, 0.30%)
• Token 0: WBTC
• Token 1: WETH
• Liquidity: 1834555147501331913
• SqrtPrice: 31462565816156305628701414535432459
• Tick: 257852

Now create an AllPools instance and see if the pool is there:

>>> all_pools = bot.AllPools(chain.id)
>>> all_pools.pools
{
    '0xCBCdF9626bC03E24f779434178A73a0B4bad62eD': WBTC-WETH (V3, 0.30%)
}

Nice!

Adding similar functionality to the LiquidityPool class is left as an exercise to the reader.

How Should The Bot Use This?

You should take care whenever you’re dealing with a huge set of objects, especially when you’re dealing with abstractions and the many sources that can generate these objects.

For example, let’s say that you are running an old bot example that manages its own LP helpers (using the degenbot_lp_helpers dictionary). You have an event watcher that observes logs from various pools. You use the pool address from the event to find the associated LP helper, then update it with the values from the event.

All good! You get excited about the pool managers and you convert the arb loading portion of the bot to use V2LiquidityPoolManager and V3LiquidityPoolManager. The LiquidityPool and V3LiquidityPool classes automatically add themselves to the AllPools dictionary, and you’re maintaining a reference to that dictionary in your execution bot.

Now whenever you need to look up a pool from the event watcher, you check the dictionary managed by AllPools. Everything you’re tracking matches up, so it works fine and you’re happy.

However, what if you didn’t really understand the value in all this abstraction and wanted to keep managing the pool helpers yourself? This is where the danger appears! It is possible that if you choose to build pool helpers directly, you could construct a pool helper for an address that was already built by the pool manager. If this happened, the new pool helper that you’ve created would overwrite the object already tracked by AllPools. Now, it’s possible that this is what you want, but unlikely.

So we have to make a decision. Should AllPools allow this? Should it reject it outright? Should it warn you in either condition? Should it throw a special exception?

There is a limit to the number of foot guns the code can protect us from. I’ve chosen to raise a PoolAlreadyExistsError exception if a pool helper is added to an existing key. Any users that need to overwrite pools for any reason can work around this by catching the exception, deleting the value at that key, and re-adding.

Here’s how it is implemented:

pool_manager.py

def __setitem__(
    self,
    pool_address: Union[ChecksumAddress, str],
    pool_helper: PoolHelper,
):
    _pool_address = to_checksum_address(pool_address)

    if self.pools.get(_pool_address):
        raise PoolAlreadyExistsError(
            f"Address {_pool_address} already known! Tracking {self.pools[_pool_address]}"
        )

    self.pools[_pool_address] = pool_helper

If you use the various abstractions and utilize the AllPools class to do your pool lookups, you should not expect to see this exception and all of the pools that you work with will be in a consistent state.

Example: Mempool Watcher With Updater

The previous mempool watcher script was extremely simplistic. It watched the Sushiswap V2, Uniswap V2, and Uniswap V3 routers for transactions, ran them through the fledgling UniswapTransaction helper, and printed the output.

The script had no mechanism to keep the various pools updated. For a proof-of-concept it was fine, since the point of the script was to demonstrate the transaction helper. But many readers thought ahead and started comparing simulated results against the actual transactions recorded on-chain. I got several questions about results that didn’t match, and I quickly realized that the cause was outdated pool states.

Now that the transaction helper is mostly finished and the pool tracker is built, we can improve the script. This is a key building block for the mempool-aware bot.

The script is presented below without much additional commentary, but I encourage you to check out a few things:

• Observe how the AllPools class is instantiated before the websocket watchers come online. The AllPools constructor uses the existing state information for that particular chain ID when it runs, so you can create or destroy your particular instance of AllPools as you like. You are free and encouraged to recreate an instance whenever you like without worry that it will overwrite info. Maintaining state data as a singleton, but allowing multiple instances of objects that reference it is called the “Borg” singleton pattern.

• I have included some logic to support nodes that support sending full transaction data with a pending transaction subscription, instead of just the hash. Set the SUBSCRIBE_TO_FULL_TRANSACTIONS boolean depending on what your node supports.

• The four event processing functions in watch_events will attempt to create a pool helper whenever a new address is discovered. These helpers will register themselves with AllPools so there is no need to perform additional tracking.

• A simple status coroutine (count_pools) prints the total number of pools tracked every 10 seconds. You should expect to see this increase over time as new pools are observed by the pending transaction and event watchers.

• The watcher now supports both Universal Router contracts.

• The script emits a lot of debugging output. To reduce the visual noise, set the bot logging level to INFO instead of DEBUG.

• It is set up by default to use a local Ethereum mainnet node. Adjust as needed if you are using a remote machine or a 3rd party RPC.

ethereum_pending_tx_watcher.py