Pickling For Faster Startup

Why So Salty Brah?

After exploring Redis, I continued my testing and spent some time in the workshop. I like Redis and I’m glad I learned how to do some stuff with it, but there are some admittedly clunky aspects. The first issue is that Redis is another external dependency that I cannot easily control. If I forget to run the server prior to starting up the bot, it will either fail complete (bad) or just not hotload my V3 LP data (OK but horribly unoptimized).

There are other issues, like the inability to nest data inside the hash. And the requirement that all keys are strings. Not horrible in practice, but it just added more complexity than I was looking for.

So I kept looking and discovered Python’s pickle module. pickle is a module that serializes and deserializes data, which is a techy way of saying “generate bytecode to deconstruct and later rebuild some arbitrary object”.

It turns out that pickle is the perfect solution to my needs. One advantage of using pickle is that is it very simple, and is built into the standard Python library. No need to run a server and communicate with it, I just read and write to a file as needed. Obviously things can go wrong, but fewer moving parts are better.

Also since we’re on the subject don’t forget to check out my friend BowTiedPickle, who is a top tier smart contract dev and can help you with Solidity and getting web3 developer work.

Console Demonstration

Let’s do a quick console demo of pickle against some data that we care about:

(.venv) [devil@dev bots]$ brownie console --network mainnet-local
Brownie v1.19.3 - Python development framework for Ethereum

BotsProject is the active project.
Brownie environment is ready.
>>> import degenbot as bot
>>> lp = bot.V3LiquidityPool('0xCBCdF9626bC03E24f779434178
A73a0B4bad62eD')
WBTC-WETH (V3, 0.30%)
• Token 0: WBTC
• Token 1: WETH
• Liquidity: 1668429917445348335
• SqrtPrice: 31470896437140864437925877869807545
• Tick: 257857

Now I have a V3 helper for the WBTC-WETH pool (which should be familiar from previous lessons). It will have liquidity information for the current word only, since the constructor does not attempt to over-fetch this data.

>>> lp.tick
257894
>>> lp._get_tick_bitmap_position(lp.tick)
(16, 201)
>>> lp.tick_bitmap[16]
{
    'bitmap': 115792089237316195423570985008687907853268655437644779123584680198628393876616,
    'block': 16916211
}

Now let’s import pickle and play with the dumps and loads functions, which can deserialize and serialize this data.

>>> import pickle
>>> pickle.dumps(lp.tick)
b'\x80\x04\x95.\x00\x00\x00\x00\x00\x00\x00\x8c\x19brownie.convert.datatypes\x94\x8c\x03Wei\x94\x93\x94JA\xef\x03\x00\x85\x94\x81\x94.'

The dumps method returns a string of bytes, which should look familiar enough given our earlier explorations of the eth_abi module. You’ll also notice that 'brownie.convert.datatypes' appears in there, which is somewhat odd. Why? Behind the scenes, pickle will attempt to discover the class associated with a particular object. In this case, the lp.tick value is a data type that comes from Brownie:

>>> type(lp.tick)
<class 'brownie.convert.datatypes.Wei'>

The Wei class from Brownie is a sub-classed int. This means it inherits all the functionality of the int, plus other features that the author needs. It is returned by calls to smart contracts, which is why lp.tick is that data type (refer to the constructor, which sets this value in this way:

slot0 = self._brownie_contract.slot0()
self.sqrt_price_x96 = slot0[0]
self.tick = slot0[1]

Anyway, back to pickle. When we call dumps, pickle will encode the type into the bytestring so that the exact object can be recreated. Let’s demonstrate!

>>> pickled_value = pickle.dumps(lp.tick)
>>> restored_value = pickle.loads(pickled_value)

Now compare the restored value to the original:

>>> restored_value
257857
>>> type(restored_value)
<class 'brownie.convert.datatypes.Wei'>
>>> restored_value == lp.tick
True
>>> restored_value is lp.tick
False

We’ve discovered that the restored value is functionally equal to the original. It has the same nominal value, same type, but is a different object. Not a problem if we’re using pickle to save and reload values, but note that a pickled object is not exactly equal in all respects. The restored object will exist at a different address in memory, so comparisons of the form 'restored_value is original_value' will evaluate to False.

Now let’s take this pool helper and fill it up with some more liquidity data:

>>> lp._update_tick_data_at_word(word_position=15,single_word=True)
>>> lp._update_tick_data_at_word(word_position=14,single_word=True)
>>> lp._update_tick_data_at_word(word_position=13,single_word=True)

And check the contents of the tick_bitmap dictionary:

>>> lp.tick_bitmap
{
    13: {
        'bitmap': 28948022309329048855892746252171976963317620781534745845727480733890183692288,
        'block': 16916248
    },
    14: {
        'bitmap': 54277541829991966604798899222822457154669449040327651580219569950521011732480,
        'block': 16916248
    },
    15: {
        'bitmap': 65136584752943729027328156206251506231096675066511205915769163927270678585344,
        'block': 16916248
    },
    16: {
        'bitmap': 115792089237316195423570985008687907853268655437644779123584680198628393876616,
        'block': 16916211
    }
}

Simple enough to understand what’s going on here, I think.

Now let’s say that we have several hundred of these pools. Instead of storing liquidity information by fetching a bunch of words after each helper is built, we’d like to provide that up front and save time.

I’ve already built the interface for this into the V3LiquidityPool helper, so let’s save the tick_data and tick_bitmap dictionaries, delete the helper, then recreate it with that interface:

>>> _tick_bitmap = lp.tick_bitmap
>>> _tick_data = lp.tick_data

Now delete the LP helper and recreate it using the optional tick_data= and tick_bitmap= arguments:

>>> del lp
>>> lp = bot.V3LiquidityPool('0xCBCdF9626bC03E24f779434178A73a0B4bad
62eD', tick_data=_tick_data, tick_bitmap=_tick_bitmap)
WBTC-WETH (V3, 0.30%)
• Token 0: WBTC
• Token 1: WETH
• Liquidity: 1668429917445348335
• SqrtPrice: 31470896437140864437925877869807545
• Tick: 257857

Now inspect the new helper and confirm that the dictionary matches:

>>> lp.tick_bitmap
{
    13: {
        'bitmap': 28948022309329048855892746252171976963317620781534745845727480733890183692288,
        'block': 16916248
    },
    14: {
        'bitmap': 54277541829991966604798899222822457154669449040327651580219569950521011732480,
        'block': 16916248
    },
    15: {
        'bitmap': 65136584752943729027328156206251506231096675066511205915769163927270678585344,
        'block': 16916248
    },
    16: {
        'bitmap': 115792089237316195423570985008687907853268655437644779123584680198628393876616,
        'block': 16916211
    }
}

Pickled For Later

It’s simple to pass this data back and forth like this inside an in-memory console session. But how do we persist this data across sessions? Memory is volatile, so we need to explore the dump and load methods.

The difference between dump and dumps (and similarly load and loads) is that the non-s versions read and write to files instead of strings.

For example, let’s pickle the tick_bitmap dictionary to a file, close our session, start another one, then restore that data via unpickling.

First we need a file object, so we will open a new reference to some file name with the open function using 'wb' as the mode (which stands for ‘write’ and ‘binary’:

>>> file = open('test.pickle','wb')
>>> pickle.dump(lp.tick_bitmap,file)
>>> file.close()

Now close the session completely launch a new one:

>>> exit()

(.venv) [devil@dev bots]$ brownie console --network mainnet-local
Brownie v1.19.3 - Python development framework for Ethereum

BotsProject is the active project.
Brownie environment is ready.
>>>

Now create a file handle using 'rb' as the mode (which stands for ‘read’ and ‘binary’ and store the data:

>>> import pickle
>>> file = open('test.pickle','rb')
>>> _tick_bitmap = pickle.load(file)

And inspect the data:

>>> _tick_bitmap
{
    13: {
        'bitmap': 28948022309329048855892746252171976963317620781534745845727480733890183692288,
        'block': 16916248
    },
    14: {
        'bitmap': 54277541829991966604798899222822457154669449040327651580219569950521011732480,
        'block': 16916248
    },
    15: {
        'bitmap': 65136584752943729027328156206251506231096675066511205915769163927270678585344,
        'block': 16916248
    },
    16: {
        'bitmap': 115792089237316195423570985008687907853268655437644779123584680198628393876616,
        'block': 16916211
    }
}

Voila!

Full Liquidity Snapshot

Now let’s really expand the approach by extending our liquidity state reference from the very first block the LP was available until the current block.

Brownie (through the magic of web3.py) makes this simple by exposing methods that fetch logs for certain events.

For example, say that we want to discover when the pool was first deployed. We know from the contract source code that an Initialize event is emitted on deployment.

Rebuild our LP helper with completely empty tick data so we start from a “blank” state:

>>> import degenbot as bot
>>> lp = bot.V3LiquidityPool('0xCBCdF9626bC03E24f779434178A73a0B4bad
62eD', tick_data={}, tick_bitmap={})
WBTC-WETH (V3, 0.30%)
• Token 0: WBTC
• Token 1: WETH
• Liquidity: 1668429917445348335
• SqrtPrice: 31470944680879643993738163239962828
• Tick: 257857
>>> lp.tick_bitmap
{}
>>> lp.tick_data
{}

Now let’s find the block where the pool was deployed: