Vyper Contract Testing and Profiling With Titanoboa

Yo Dawg I Heard You Liked Snake Code So We Made Snake Code For Your Snake Code

The smart contract writing process is pretty straightforward once you’ve done it a few times, but there are some parts that can (and do) quickly become a bottleneck.

In the case of Vyper development, the process usually slows considerably when you’ve gotten your contract mostly written but want to do some testing. You can quickly determine if your syntax is correct by using an IDE plugin, or running a quick vyper contract_name.vy on the console. But just because a contract compiles does not mean it is correct, or whether it operates the way you think it should.

The pain begins when you step away from the comfy Python tooling and enter the lower level of the Ethereum Virtual Machine (EVM).

For example, say that I’m trying to implement some hashing function in a Vyper contract. I write the function, and I want to test it by passing some inputs in and checking that the values are correct. Traditionally I would do this by writing a series of tests that will start a fork, deploy the contract, send inputs to it, and check the outputs. It’s manageable if you already have a set of scripts built to automate the process, but it’s painful to build this all from scratch, and outsourcing execution to another tool introduces a dependency out of your control.

For example — what if your installed version of Brownie is incompatible with a new Ganache release? This has happened several times, and with Brownie going end-of-life it will likely occur again.

Titanoboa

This tooling gap became apparent last summer when the legends bantg and big_tech_sux started discussing the possibility of a native Python interpreter for Vyper contracts. After a few days of devs doing something, Titanoboa was pushed to the Vyperlang github repo.

The goal of Titanoboa is to lower the barrier to development by removing the need to stand up a full EVM stack for Vyper contract testing.

It does this in a very elegant way: it loads Vyper (which is written in Python) as a module, compiles a contract identically to the standalone program, then acts as a middleman to py-evm, interpreting inputs and outputs and formatting the low-level EVM into native Python.

It can integrate with pytest, do gas profiling, and it runs in a Jupyter notebook if you’re into that kind of thing.

Something as simple as being able to print() from within a contract is just so so so nice. No more messing around with logging ad-hoc events just to inspect a variable.

It also has the ability to interact with a live blockchain, or work from actual chain state by forking from some particular block via JSON-RPC.

In short, it allows you to quickly write, test, and deploy Vyper without having to leave your Python environment!

The recently-launched Curve stablecoin contracts include a complete Titanoboa test suite, which you can scrape for state-of-the-art gigabrain testing techniques.

Extremely based individual Gerritt Hall, who authors

crv.mktcap.eth

, published a thorough writeup on the Curve test suite last year.

November 29, 2022: Titanoboa Testing 🐍🕵️

This post is free for everyone, since I want to encourage everyone to try Vyper and Titanoboa.

I’ve been using Titanoboa to profile my executor contract and reduce gas usage. I will detail those improvements in an upcoming subscriber-only post, and this will serve as a standalone introduction.

Installing

I recommend creating a dedicated virtual environment for use with Titanoboa. Simple to do:

[btd@main code]$ mkdir boa_sandbox/
[btd@main code]$ cd boa_sandbox/
[btd@main boa_sandbox]$ python3 -m venv .venv
[btd@main boa_sandbox]$ . .venv/bin/activate
(.venv) [btd@main boa_sandbox]$ pip install titanoboa

[...]

Successfully installed asttokens-2.4.0 attrs-23.1.0 cached-property-1.5.2 certifi-2023.7.22 charset-normalizer-3.2.0 cytoolz-0.12.2 eth-abi-4.2.0 eth-bloom-2.0.0 eth-hash-0.5.2 eth-keys-0.4.0 eth-stdlib-0.2.6 eth-typing-3.4.0 eth-utils-2.2.0 exceptiongroup-1.1.3 hexbytes-0.3.1 hypothesis-6.84.2 idna-3.4 importlib-metadata-6.8.0 iniconfig-2.0.0 lru-dict-1.2.0 markdown-it-py-3.0.0 mdurl-0.1.2 mypy-extensions-1.0.0 packaging-23.1 parsimonious-0.9.0 pluggy-1.3.0 py-ecc-6.0.0 py-evm-0.7.0a4 pycryptodome-3.18.0 pyethash-0.1.27 pygments-2.16.1 pytest-7.4.2 regex-2023.8.8 requests-2.31.0 rich-13.5.2 rlp-3.0.0 safe-pysha3-1.0.4 semantic-version-2.10.0 six-1.16.0 sortedcontainers-2.4.0 titanoboa-0.1.7 tomli-2.0.1 toolz-0.12.0 trie-2.1.1 urllib3-2.0.4 vyper-0.3.9 wheel-0.41.2 zipp-3.16.2

With that done, let’s take a tour through a few Titanoboa testing snippets that you may find useful.

Testing A Separate Contract

Let’s write a very simple contract that stores a 24 character string, initially set to “Hello, world!”, and allows you to change the message if you deposit 1 Ether or more.

hello_world.vy

# @version ^0.3

message: String[24]

@external
def __init__():
    self.message = 'Hello, world!'

@external
@payable
def update_message(message: String[24]):
    assert len(message) <= 24, "Message too long"
    assert msg.value >= 1*10**18, "Insufficient deposit, send 1 or more Ether"
    self.message = message

Now open a Python shell and poke at the contract:

>>> import boa
>>> contract = boa.load('hello_world.vy')
>>> contract

<hello_world.vy at 0x0000000000000000000000000000000000000066, compiled with vyper-0.3.9+66b9670>
<storage: message=b'Hello, world!\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'>

Right away, Titanoboa gives us a highly useful representation of the contract including the address where it is deployed (0x0000…0066), its compiler version (0.3.9), and the storage keys with their values. Note that the message variable is right-padded with zeroes to its maximum size of 24 characters.

Now let’s try to change the message:

>>> contract.update_message("Degen Code was here")

Traceback (most recent call last):

[...]

boa.vyper.contract.BoaError: Insufficient deposit, send 1 or more Ether

<hello_world.vy at 0x0000000000000000000000000000000000000066, compiled with vyper-0.3.9+66b9670>
<storage: message=b'Hello, world!\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'>
 <compiler: user revert with reason>
  contract "hello_world.vy:13", function "update_message", line 13:4 
       12     assert len(message) <= 24, "Message too long"
  ---> 13     assert msg.value >= 1*10**18, "Insufficient deposit, send 1 or more Ether"
  ------------^
       14     self.message = message
 <locals: message=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00 '>
 <storage: message=b'Hello, world!\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'>

The traceback is a bit messy, but the key info you get is where the contract reverted (line 13), and a BoaError exception with the revert message thrown by the contract (“Insufficient deposit, send 1 or more Ether”).

OK so let’s try to satisfy it by sending some Ether along with our call:

>>> contract.update_message("Degen Code was here",value=1*10**18)

Traceback (most recent call last):

[...]

eth.exceptions.InsufficientFunds: Insufficient funds: 0 < 1000000000000000000

Hey, that’s no fair! How can I test without Ether? Titanoboa has a simple method to handle this. It exposes a low-level object called env that will let you override addresses, balances, contract bytecode, block number, timestamp, etc.

First let’s see who Titanoboa thinks we are:

>>> boa.env.eoa
'0x0000000000000000000000000000000000000065'

And now we can give ourselves 1000 Ether to play with, then try the contract call again:

>>> boa.env.set_balance(boa.env.eoa, 1000*10**18)
>>> contract.update_message("Degen Code was here",value=1*10**18)

It worked! No output though, so let’s check the storage state:

>>> contract
<hello_world.vy at 0x0000000000000000000000000000000000000066, compiled with vyper-0.3.9+66b9670>
<storage: message=b'Degen Code was here\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'>

You can get access to the storage more directly by inspecting the elements within the _storage attribute:

>>> contract._storage.message.get()
b'Degen Code was here\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'

The elements also expose their storage slot and EVM storage key:

>>> contract._storage.message.slot
0
>>> contract._storage.message.addr.hex()
'0000000000000000000000000000000000000066'

Testing an Inline Contract

You don’t need to write a separate contract to use Titanoboa. You can copy-paste Vyper source code instead, for rapid hot-testing without needing to store anything on disk.

Here’s a once-through script that performs the same steps (skipping the parts that revert) without needing to write a Vyper contract first:

hello_world.py

import boa

contract = boa.loads(
    """
# @version ^0.3

message: String[24]

@external
def __init__():
    self.message = 'Hello, world!'

@external
@payable
def update_message(message: String[24]):
    assert len(message) <= 24, "Message too long"
    assert msg.value >= 1*10**18, "Insufficient deposit, send 1 or more Ether"
    self.message = message    
"""
)
print(contract)
print("Setting balance...")
boa.env.set_balance(boa.env.eoa, 1000 * 10**18)
print("Updating message...")
contract.update_message("Degen Code was here", value=1 * 10**18)
print(f"Message = {contract._storage.message.get()}")
print(f"Slot = {contract._storage.message.slot}")
print(f"Addr = {contract._storage.message.addr.hex()}")

Running this gives identical results:

(.venv) [btd@main boa_sandbox]$ /home/btd/code/boa_sandbox/.venv/bin/python /home/btd/code/boa_sandbox/hello_world.py

<VyperContract at 0x0000000000000000000000000000000000000066, compiled with vyper-0.3.9+66b9670>
<storage: message=b'Hello, world!\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'>
Setting balance...
Updating message...
Message = b'Degen Code was here\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
Slot = 0
Addr = 0000000000000000000000000000000000000066

Testing a Vyper Statement

Titanoboa can also get the value of a single statement using eval, provided that the statement does not need to load or store contract state.

This example script gets the keccak256 hash of several input strings:

single_function_test.py

import boa

for input in (
    "Everyday I check price",
    "Bad price",
    "Can devs do something?",
):
    output = boa.eval(f'keccak256("{input}")')
    print(
        f"keccak256('{input}') = ",
        boa.eval(f'keccak256("{input}")').hex(),
    )

Output:

(.venv) [btd@main boa_sandbox]$ /home/btd/code/boa_sandbox/.venv/bin/python /home/btd/code/boa_sandbox/single_function_test.py

Evaluating keccak256...
keccak256('Everyday I check price') =  7eb70f9f5196f04f55da6fd8d2c3c65858ccb6737da00357ad9bb4e59559fb3b
keccak256('Bad price') =  d206708be28d03196f819df4b2f0537150d892d70597868f3cde63fa4663f6aa
keccak256('Can devs do something?') =  e3f6c1653d62cff08af45df0803c7fdba23f533a0a43ece8af6776b84795ea6f

You can also use the eval method from a contract instance, allowing you to reference a particular storage value. Now we extend our previous hello_world.py with a one-liner that gets the keccak256 hash of the string stored in message:

hello_world.py

[...]

print(
    f"keccak256(message) =", 
    f"{contract.eval('keccak256(self.message)').hex()}",
)

Output:

[...]

keccak256(message) = a7746fd96eccdbe5223f4e30ae79d27064a5264fe87fdd0cb4eac8f8484c1132

Access Internal Functions

Internal functions are good, but they are very hard to test in isolation because a contract must be activated externally. And by definition, an internal function cannot be accessed externally! The typical solution is to provide a shim, which is an external function that passes inputs to the internal function, captures its output, and passes it back to the caller. Shims are dangerous because you might forget to remove them prior to deployment, and flaky because changes to the internal function’s interface must be updated in the shim.

It also makes gas testing tough, because the overhead of the shim has to be offset later.

Titanoboa allows you to test an internal function directly via the internal attribute of the contract object. Here’s a simple example using the inline loading method:

import boa

contract = boa.loads(
    """
# @version ^0.3

@internal
def test(val: uint256) -> uint256:
    return val
"""
)

print(f"{contract.internal.test(420)}")

Output:

(.venv) [btd@main boa_sandbox]$ /home/btd/code/boa_sandbox/.venv/bin/python /home/btd/code/boa_sandbox/internal_function_test.py
420

No shim needed, just clean access to internal functions.

And for completeness’ sake, we can verify that attempting to access the function externally throws an exception.

[...]

print(f"{contract.test(420)}")

Output:

(.venv) [btd@main boa_sandbox]$ /home/btd/code/boa_sandbox/.venv/bin/python /home/btd/code/boa_sandbox/internal_function_test.py

Traceback (most recent call last):

[...]

AttributeError: 'VyperContract' object has no attribute 'test'

Gas Profiling

This is my favorite feature. If you decorate a function with @pytest.mark.profile and run it with pytest, you get a nicely formatted table at the end showing the total gas usage of your function calls.

Here we wrap the entire hello_world.py file into a pytest, decorated with the profiling mark:

hello_world_test.py

import boa
import pytest


@pytest.mark.profile
def test_hello_world():
    contract = boa.loads(
        """
    # @version ^0.3

    message: String[24]

    @external
    def __init__():
        self.message = 'Hello, world!'

    @external
    @payable
    def update_message(message: String[24]):
        assert len(message) <= 24, "Message too long"
        assert msg.value >= 1*10**18, "Insufficient deposit, send 1 or more Ether"
        self.message = message    
    """
    )

    print("Setting balance...")
    boa.env.set_balance(boa.env.eoa, 1000 * 10**18)
    print("Updating message...")
    contract.update_message("Degen Code was here", value=1 * 10**18)
    print(f"Message = {contract._storage.message.get()}")
    print(f"Slot = {contract._storage.message.slot}")
    print(f"Addr = {contract._storage.message.addr.hex()}")

Run this with pytest hello_world_test.py and enjoy the nicely formatted line-by-line gas use report.

You’ll notice that I’ve stripped the eval statement out, because it imposes a gas load that pytest cannot profile correctly. If you leave it in, a second Computation row called “unnamed” will show up with an associated gas cost. Make sure to clean up your source code before doing real gas profiling!

EVM State Patching

You can modify the state of the EVM environment on demand. Say that you want to run a test on block height 1000, then repeat it on block 2000. If you were running a fork, you’d have to drive it externally to the new height by sleeping, mining some number of blocks, or both.

But with Titanoboa you can just set the new values by reading/writing to the boa.env.vm object’s patch attribute:

>>> import boa
>>> boa.env.vm.patch.block_number = 1000
>>> boa.env.vm.patch.timestamp = 1700000000
>>> boa.eval('block.number')
69
>>> boa.eval('block.timestamp')
1700000000

Literally Just Print Stuff

You can throw a print() statement into your Vyper code to inspect arbitrary local variables during execution. Not as good as a step debugger, but still really nice.

Here let’s modify the Vyper contract within hello_world.py to include a print before and after the message update:

hello_world_print.py

import boa

contract = boa.loads(
    """
# @version ^0.3

message: String[24]

@external
def __init__():
    self.message = 'Hello, world!'

@external
@payable
def update_message(message: String[24]):
    assert len(message) <= 24, "Message too long"
    assert msg.value >= 1*10**18, "Insufficient deposit, send 1 or more Ether"
    print('Old Message =', self.message)
    self.message = message    
    print('New Message =', self.message)
"""
)
print(contract)
print("Setting balance...")
boa.env.set_balance(boa.env.eoa, 1000 * 10**18)
print("Updating message...")
contract.update_message("Degen Code was here", value=1 * 10**18)
print(f"Message = {contract._storage.message.get()}")
print(f"Slot = {contract._storage.message.slot}")
print(f"Addr = {contract._storage.message.addr.hex()}")
print(
    f"Keccak(message) =",
    f"{contract.eval('keccak256(self.message)').hex()}",
)

Output:

[...]

Updating message...
Old Message = Hello, world!
New Message = Degen Code was here

[...]

Fork It, We’ll Do It Live

Titanoboa can also set its EVM state based on a live chain over JSON-RPC. Here we fork from the live chain via Ankr, then check some block attributes and balances:

>>> import boa

>>> boa.eval('block.number')
18096785
>>> boa.eval('block.timestamp')
1694237675

# vitalik.eth
>>> boa.eval('0xd8dA6BF26964aF9D7eEd9e03E53415D37aA96045.balance')
933812942372671549329
# jaredfromsubway.eth (he can't keep getting away with it!)
>>> boa.eval('0xae2Fc483527B8EF99EB5D9B44875F005ba1FaE13.balance')
122410420368990577633

The forking implementation is limited, but gives you enough to create good scripted tests against known chain states.

Impersonating An EOA

Titanoboa provides a method of impersonating an externally owned account (EOA) with the prank context manager. This is helpful if you want to test your contract against intrusion by randomized addresses, or verify that your access controls work correctly.

Let’s set up a contract with some roles, then test for the permissions listed in the Roles enum by pranking the EOAs we set, plus a random address that should return zero (no permission):

import boa

contract = boa.loads(
    """
# @version ^0.3

OWNER: immutable(address)

enum Roles:
    USER # value = 1
    MOD # value = 2
    ADMIN # value = 4

roles: HashMap[address, Roles]

@external
def __init__():
    OWNER = msg.sender


@external
def add_user(addr:address):
    assert msg.sender == OWNER    
    self.roles[addr] |= Roles.USER


@external
def add_mod(addr:address):
    assert msg.sender == OWNER    
    self.roles[addr] |= Roles.MOD


@external
def add_admin(addr:address):
    assert msg.sender == OWNER    
    self.roles[addr] |= Roles.ADMIN

@external
@view
def check_role() -> Roles:
    return self.roles[msg.sender]
"""
)

unpermissioned = (boa.env.generate_address(), "Unpermissioned User")
user_address = (boa.env.generate_address(), "Authorized User")
mod_address = (boa.env.generate_address(), "Moderator")
admin_address = (boa.env.generate_address(), "Admin")

contract.add_user(user_address[0])
contract.add_mod(mod_address[0])
contract.add_admin(admin_address[0])

for address, description in (
    unpermissioned,
    user_address,
    mod_address,
    admin_address,
):
    with boa.env.prank(address):
        print(f"Role for {description}: {contract.check_role()}")

Output:

Role for Unpermissioned User: 0
Role for Authorized User: 1
Role for Moderator: 2
Role for Admin: 4