I started out cowboy coding: write some new code, run it, see what happens. Nothing immediately failed? Assume it was good, write the next section of code.
Repeat until something breaks.
Then I’d start cowboy debugging: scratch my head, add a few prints to see the contents of some variables, run and see if my assumptions held near the crash.
Repeat until fixed.
This is a terrible process but it’s the only one I knew when starting out. Being self-taught, I generally made progress by identifying an “acceptable” next step and iterating, but wasted time because of ignorance of the “best” next step.
Saved By The Testing Goat
Eventually enough subtle bugs caught me that I realized I needed to get better at writing tests. And by “get better” I mean actually learn how to write one and then write one.
I read an excellent book “Architecture Patterns with Python” by Harry Percival, a followup to his earlier book “Test-Driven Development with Python: Obey the Testing Goat: Using Django, Selenium, and JavaScript” which got into the guts of a test-driven development process. I recommend the book — which you can read for free on the author’s website.
The book focuses, among other things, on correct use of pytest and developing a test suite. With some considerable effort, I’ve added a test suite for degenbot that achieves roughly 85% coverage at time of writing.
I add new tests all the time, and the frequency of bugs has decreased considerably. It was instrumental in developing the Curve V1 pool helper; a complex class like that could not be developed cowboy-style.
Smart Contract and Chain Testing
Smart contracts are long-lived, and they handle unpredictable inputs from a hostile network running 24/7. If you don’t properly test a contract before deploying, you’re simply NGMI and begging to get rekt.
Well, look no further than Ape Framework. I’ve written about Ape before, so please review the introductory article below if you are unfamiliar.
Ape is Python-based, so it integrates cleanly with pytest
.
A Gentle Introduction to Pytest
The pytest
package is complex and sophisticated, but it relies on concepts that are quite simple.
In a typical setup, pytest
will look for files with function starting with test_
, run them, then print a summary report after the run has concluded.
Any function that executes and return normally (i.e. without throwing an unexpected exception) counts as a passing test. Some functions can be marked as being expected to fail in certain ways, so you are not limited to the “happy path” only.
Tests serve two primary functions:
They verify that the programmer’s assumptions hold true.
They verify that the model’s constraints hold true.
A basic test will verify that the code you’ve written executes as expected with some input data. Perhaps you’ve written a very simple function that calculates the hypotenuse of a triangle using the Pythagorean Theorem to three decimal places:
hypotenuse.py
import math
def hypotenuse(a: float, b:float) -> float:
return round(math.sqrt(a**2 + b**2),3)
Now you want to test it.
Write a simple test that loads the function, and asserts some values:
test_hypotenuse.py
from hypotenuse import hypotenuse
def test_hypotenuse():
assert hypotenuse(0,1) == 1.0
assert hypotenuse(1,2) == 2.236
assert hypotenuse(2,3) == 3.606
Then run pytest
:
(test) btd@main:~/tmp$ pytest
========================= test session starts =========================
platform linux -- Python 3.12.1, pytest-8.0.2, pluggy-1.4.0
rootdir: /home/btd/tmp
collected 1 item
test_hypotenuse.py . [100%]
========================== 1 passed in 0.00s ==========================
If any assert
statement fails, the test will fail and the reason will be given:
test_hypotenuse.py
from hypotenuse import hypotenuse
def test_hypotenuse():
assert hypotenuse(0,1) == 1.0
assert hypotenuse(1,2) == 2.236
assert hypotenuse(2,3) == 3.606
assert hypotenuse(3,4) == 4.0
(test) btd@main:~/tmp$ pytest
========================= test session starts =========================
platform linux -- Python 3.12.1, pytest-8.0.2, pluggy-1.4.0
rootdir: /home/btd/tmp
collected 1 item
test_hypotenuse.py F [100%]
============================== FAILURES ===============================
___________________________ test_hypotenuse ___________________________
def test_hypotenuse():
assert hypotenuse(0,1) == 1.0
assert hypotenuse(1,2) == 2.236
assert hypotenuse(2,3) == 3.606
> assert hypotenuse(3,4) == 4.0
E assert 5.0 == 4.0
E + where 5.0 = hypotenuse(3, 4)
test_hypotenuse.py:7: AssertionError
======================= short test summary info =======================
FAILED test_hypotenuse.py::test_hypotenuse - assert 5.0 == 4.0
========================== 1 failed in 0.02s ==========================
Helpfully, pytest
gives you both values in the order they were asserted. So I see clearly that the return value of hypotenuse(3,4)
is 5.0, while I am asserting it should equal 4.0.
Whenever a test fails, first confirm that you are asserting the correct value (check your assumptions, then the constraints of the problem), then confirm the code is operating correctly. I know that the assertion is wrong above, since the correct value is simple to verify by hand, so I just fix the test to assert equalith with 5.0, and it will pass as expected.
Fixtures
A key benefit of using pytest
(and other suites) is access to testing fixtures. A fixture is a function that returns an object of a guaranteed type and contents.
This is particularly useful when doing smart contract or blockchain testing, because state changes with each new block. You must have easy access to predictable chain state to run your tests. Further, you must take care that running one test does not affect another test. Fixtures help because when built properly, you can guarantee that the starting condition for a given test is uniform, regardless of the ordering and timing of that test.
If you wanted to operate on a given copy of some data across several tests, you should define a fixture that returns a known “clean” copy of that data to each test. This is highly preferable to the alternative, which is hand-resetting the data before or after each test.
A standard pytest
fixture is annotated with a decorator @pytest.fixture
, with optional arguments to control how often the fixture is “reset”. By default, a fixture returns a new copy of its value on each new run of the test function. For heavier fixtures that take a long time to generate, you might choose to set this to “module” or “session”. Read more about this in the Pytest Fixture Scope documentation.
Ape Fixtures
Ape provides several out-of-the-box fixtures that expose useful functionality.
accounts
chain
networks
project
contract
Read about them in more detail in the Ape Testing Fixtures documentation.
Whenever a fixture is provided as an argument to a particular test, that fixture will be accessible inside the test.
The fixtures above are pre-built instances of high level Ape manager classes. If you want to discover the methods and attributes of these managers, review them at the Ape Managers documentation.
Ape Test Structure
By default, a set of tests in an Ape project can be run by executing ape test
from the base project directory.
The tests are assumed to be stored in the tests/
subdirectory of the main project. Files and functions must be named per the pytest
standard, as shown above.
Example Ape Testing
The example below is performed inside a dedicated virtual environment running Python 3.11.8, using Ape version 0.7.10 installed via pip install eth-ape[recommended-plugins]
. All project files are stored in ~/code/ape_testing
.
First, activate the virtual environment and prepare the project directory:
btd@main:~/code/ape_testing$ pyenv local ape
(ape) btd@main:~/code/ape_testing$ ape init
Please enter project name: Ape Testing
SUCCESS: Ape Testing is written in ape-config.yaml
(ape) btd@main:~/code/ape_testing$ ls -l
total 4
-rw-r--r--. 1 btd btd 18 Feb 26 21:48 ape-config.yaml
drwxr-xr-x. 1 btd btd 0 Feb 26 21:48 contracts
drwxr-xr-x. 1 btd btd 0 Feb 26 21:48 scripts
drwxr-xr-x. 1 btd btd 0 Feb 26 21:48 tests
Notice that it creates a tests/
directory.
We will create some simple tests there that use the chain and networks fixtures, connect to a local node, verify that the network is up, and the chain height has moved beyond the genesis block:
tests/test_mainnet.py
import ape
def test_mainnet_online(chain, networks):
with networks.parse_network_choice('http://localhost:8545'):
assert chain.blocks.head.number > 0
I can use the parse_network_choice
method to connect to an arbitrary node endpoint. A similar test verifies that my Arbitrum Nitro node is operating properly:
test_arbitrum.py
import ape
def test_arbitrum_online(chain, networks):
with networks.parse_network_choice('http://localhost:8547'):
assert chain.blocks.head.number > 0
Both tests run as expected:
(ape) btd@main:~/code/ape_testing$ ape test
========================= test session starts ==========================
platform linux -- Python 3.11.8, pytest-7.4.4, pluggy-1.4.0
rootdir: /home/btd/code/ape_testing
plugins: eth-ape-0.7.10, web3-6.15.1
collected 2 items
INFO: Connecting to existing Geth node at http://localhost:8545/[hidden].
tests/test_arbitrum.py . [ 50%]
tests/test_mainnet.py . [100%]
========================== 2 passed in 0.08s ===========================
Network and Provider Defaults
As we progress, we will often be running tests against a local fork. I prefer to use Anvil (part of the Foundry suite), which will automatically connect to a local Geth node at port 8545 if available. If one is not available, you should specify it in your ape-config.yaml
file, like below:
ape-config.yaml
name: Ape Testing
plugins:
- name: vyper
- name: foundry
foundry:
host: auto
fork:
ethereum:
mainnet:
upstream_provider: geth
geth:
ethereum:
mainnet:
uri: http://localhost:8545
In particular, pay attention to the uri
key under the geth
section. Adjust it as needed for your node!
Testing Deployments
To see a more sophisticated example of ape test
, let’s write a simple Vyper contract with a deployment function that sets an owner address, and a view function that calculate a hypotenuse.
The ownership portion isn’t particularly relevant to this contract, but it does demonstrate that you can test and verify values exposed by public getter functions.
Then we will write a series of tests to confirm it works as expected.
hypotenuse.vy
#pragma version >=0.3.10
owner: public(address)
@external
def __init__():
self.owner = msg.sender
@external
@pure
def hypotenuse(a: uint256, b: uint256) -> decimal:
_a: decimal = convert(a, decimal)
_b: decimal = convert(b, decimal)
return sqrt(_a*_a + _b*_b)
Save the contract in the contracts/
directory.
Now write a test that deploys the contract using the first testing account, accessible at accounts[0]
.
test_hypotenuse.py
def test_deployment(project, accounts):
hypotenuse_contract = project.hypotenuse.deploy(
sender=accounts[0]
)
assert hypotenuse_contract.owner() == accounts[0].address
Run ape test
with the ecosystem:network:provider
network format associated with the Foundry plugin to confirm that deploying the contract on an Anvil fork works as expected:
(ape) btd@dev:~/code/ape_testing$ ape test --network ethereum:local:foundry
========================= test session starts =========================
platform linux -- Python 3.11.8, pytest-7.4.4, pluggy-1.4.0
rootdir: /home/btd/code/ape_testing
plugins: eth-ape-0.7.10, web3-6.15.1
collected 1 item
INFO: Connecting to existing Geth node at http://localhost:8545/[hidden].
tests/test_hypotenuse.py . [100%]
========================== 1 passed in 0.30s ==========================
INFO: Stopping 'anvil' process.
Now let’s transform the deployment into a fixture, which gives us simple access to a fresh deployment for subsequent tests:
import ape
import pytest
@pytest.fixture
def hypotenuse_contract(project, accounts):
hypotenuse_contract = project.hypotenuse.deploy(sender=accounts[0])
return hypotenuse_contract
def test_ownership(hypotenuse_contract, accounts):
assert hypotenuse_contract.owner() == accounts[0].address
Running the tests again confirms it works as expected:
(ape) btd@dev:~/code/ape_testing$ ape test --network ethereum:local:foundry
========================= test session starts =========================
platform linux -- Python 3.11.8, pytest-7.4.4, pluggy-1.4.0
rootdir: /home/btd/code/ape_testing
plugins: eth-ape-0.7.10, web3-6.15.1
collected 1 item
INFO: Starting 'anvil' process.
tests/test_hypotenuse.py . [100%]
========================== 1 passed in 0.29s ==========================
INFO: Stopping 'anvil' process.
Now add another test to assert that the calculation works correctly:
def test_calc(hypotenuse_contract):
assert hypotenuse_contract.hypotenuse(1, 2) == Decimal("2.2360679774")
NOTE: the Vyper decimal
type uses 10 decimal places, so the test asserts equality with a Decimal
value of that same precision.
ANOTHER NOTE: I am using uint256 for the inputs, since Ape throws a ConversionError exception when I attempt to set a function argument typed as a decimal. Ape currently does not have a converter to handle Vyper’s decimal type, so I will investigate and submit a PR if I can resolve this.
FOLLOW-UP: my pull request to fix this was accepted, so decimal
inputs will work as expected after the next release.
Testing Bad Inputs
The hypotenuse
function above is very optimistic. It assumes that you will provide reasonable values that will not result in overflows, underflows, underruns, overruns and all sorts of other nasty things.