Testing With Ape Framework
Ape Tests Together Strong
I started out cowboy coding: write some new code, run it, see what happens. Nothing immediately failed? Assume it was good, write the next section of code.
Repeat until something breaks.
Then I’d start cowboy debugging: scratch my head, add a few prints to see the contents of some variables, run and see if my assumptions held near the crash.
Repeat until fixed.
This is a terrible process but it’s the only one I knew when starting out. Being self-taught, I generally made progress by identifying an “acceptable” next step and iterating, but wasted time because of ignorance of the “best” next step.
Saved By The Testing Goat
Eventually enough subtle bugs caught me that I realized I needed to get better at writing tests. And by “get better” I mean actually learn how to write one and then write one.
I read an excellent book “Architecture Patterns with Python” by Harry Percival, a followup to his earlier book “Test-Driven Development with Python: Obey the Testing Goat: Using Django, Selenium, and JavaScript” which got into the guts of a test-driven development process. I recommend the book — which you can read for free on the author’s website.
The book focuses, among other things, on correct use of pytest and developing a test suite. With some considerable effort, I’ve added a test suite for degenbot that achieves roughly 85% coverage at time of writing.
I add new tests all the time, and the frequency of bugs has decreased considerably. It was instrumental in developing the Curve V1 pool helper; a complex class like that could not be developed cowboy-style.
Smart Contract and Chain Testing
Smart contracts are long-lived, and they handle unpredictable inputs from a hostile network running 24/7. If you don’t properly test a contract before deploying, you’re simply NGMI and begging to get rekt.
Well, look no further than Ape Framework. I’ve written about Ape before, so please review the introductory article below if you are unfamiliar.
Ape is Python-based, so it integrates cleanly with pytest.
A Gentle Introduction to Pytest
The pytest package is complex and sophisticated, but it relies on concepts that are quite simple.
In a typical setup, pytest will look for files with function starting with test_, run them, then print a summary report after the run has concluded.
Any function that executes and return normally (i.e. without throwing an unexpected exception) counts as a passing test. Some functions can be marked as being expected to fail in certain ways, so you are not limited to the “happy path” only.
Tests serve two primary functions:
They verify that the programmer’s assumptions hold true.
They verify that the model’s constraints hold true.
A basic test will verify that the code you’ve written executes as expected with some input data. Perhaps you’ve written a very simple function that calculates the hypotenuse of a triangle using the Pythagorean Theorem to three decimal places:
hypotenuse.py
import math
def hypotenuse(a: float, b:float) -> float:
return round(math.sqrt(a**2 + b**2),3)Now you want to test it.
Write a simple test that loads the function, and asserts some values:
test_hypotenuse.py
from hypotenuse import hypotenuse
def test_hypotenuse():
assert hypotenuse(0,1) == 1.0
assert hypotenuse(1,2) == 2.236
assert hypotenuse(2,3) == 3.606Then run pytest:
(test) btd@main:~/tmp$ pytest
========================= test session starts =========================
platform linux -- Python 3.12.1, pytest-8.0.2, pluggy-1.4.0
rootdir: /home/btd/tmp
collected 1 item
test_hypotenuse.py . [100%]
========================== 1 passed in 0.00s ==========================If any assert statement fails, the test will fail and the reason will be given:
test_hypotenuse.py
from hypotenuse import hypotenuse
def test_hypotenuse():
assert hypotenuse(0,1) == 1.0
assert hypotenuse(1,2) == 2.236
assert hypotenuse(2,3) == 3.606
assert hypotenuse(3,4) == 4.0(test) btd@main:~/tmp$ pytest
========================= test session starts =========================
platform linux -- Python 3.12.1, pytest-8.0.2, pluggy-1.4.0
rootdir: /home/btd/tmp
collected 1 item
test_hypotenuse.py F [100%]
============================== FAILURES ===============================
___________________________ test_hypotenuse ___________________________
def test_hypotenuse():
assert hypotenuse(0,1) == 1.0
assert hypotenuse(1,2) == 2.236
assert hypotenuse(2,3) == 3.606
> assert hypotenuse(3,4) == 4.0
E assert 5.0 == 4.0
E + where 5.0 = hypotenuse(3, 4)
test_hypotenuse.py:7: AssertionError
======================= short test summary info =======================
FAILED test_hypotenuse.py::test_hypotenuse - assert 5.0 == 4.0
========================== 1 failed in 0.02s ==========================Helpfully, pytest gives you both values in the order they were asserted. So I see clearly that the return value of hypotenuse(3,4) is 5.0, while I am asserting it should equal 4.0.
Whenever a test fails, first confirm that you are asserting the correct value (check your assumptions, then the constraints of the problem), then confirm the code is operating correctly. I know that the assertion is wrong above, since the correct value is simple to verify by hand, so I just fix the test to assert equalith with 5.0, and it will pass as expected.
Fixtures
A key benefit of using pytest (and other suites) is access to testing fixtures. A fixture is a function that returns an object of a guaranteed type and contents.
This is particularly useful when doing smart contract or blockchain testing, because state changes with each new block. You must have easy access to predictable chain state to run your tests. Further, you must take care that running one test does not affect another test. Fixtures help because when built properly, you can guarantee that the starting condition for a given test is uniform, regardless of the ordering and timing of that test.
If you wanted to operate on a given copy of some data across several tests, you should define a fixture that returns a known “clean” copy of that data to each test. This is highly preferable to the alternative, which is hand-resetting the data before or after each test.
A standard pytest fixture is annotated with a decorator @pytest.fixture, with optional arguments to control how often the fixture is “reset”. By default, a fixture returns a new copy of its value on each new run of the test function. For heavier fixtures that take a long time to generate, you might choose to set this to “module” or “session”. Read more about this in the Pytest Fixture Scope documentation.
Ape Fixtures
Ape provides several out-of-the-box fixtures that expose useful functionality.
accountschainnetworksprojectcontract
Read about them in more detail in the Ape Testing Fixtures documentation.
Whenever a fixture is provided as an argument to a particular test, that fixture will be accessible inside the test.
The fixtures above are pre-built instances of high level Ape manager classes. If you want to discover the methods and attributes of these managers, review them at the Ape Managers documentation.
Ape Test Structure
By default, a set of tests in an Ape project can be run by executing ape test from the base project directory.
The tests are assumed to be stored in the tests/ subdirectory of the main project. Files and functions must be named per the pytest standard, as shown above.
Example Ape Testing
The example below is performed inside a dedicated virtual environment running Python 3.11.8, using Ape version 0.7.10 installed via pip install eth-ape[recommended-plugins]. All project files are stored in ~/code/ape_testing.
First, activate the virtual environment and prepare the project directory:
btd@main:~/code/ape_testing$ pyenv local ape
(ape) btd@main:~/code/ape_testing$ ape init
Please enter project name: Ape Testing
SUCCESS: Ape Testing is written in ape-config.yaml
(ape) btd@main:~/code/ape_testing$ ls -l
total 4
-rw-r--r--. 1 btd btd 18 Feb 26 21:48 ape-config.yaml
drwxr-xr-x. 1 btd btd 0 Feb 26 21:48 contracts
drwxr-xr-x. 1 btd btd 0 Feb 26 21:48 scripts
drwxr-xr-x. 1 btd btd 0 Feb 26 21:48 testsNotice that it creates a tests/ directory.
We will create some simple tests there that use the chain and networks fixtures, connect to a local node, verify that the network is up, and the chain height has moved beyond the genesis block:
tests/test_mainnet.py
import ape
def test_mainnet_online(chain, networks):
with networks.parse_network_choice('http://localhost:8545'):
assert chain.blocks.head.number > 0I can use the parse_network_choice method to connect to an arbitrary node endpoint. A similar test verifies that my Arbitrum Nitro node is operating properly:
test_arbitrum.py
import ape
def test_arbitrum_online(chain, networks):
with networks.parse_network_choice('http://localhost:8547'):
assert chain.blocks.head.number > 0Both tests run as expected:
(ape) btd@main:~/code/ape_testing$ ape test
========================= test session starts ==========================
platform linux -- Python 3.11.8, pytest-7.4.4, pluggy-1.4.0
rootdir: /home/btd/code/ape_testing
plugins: eth-ape-0.7.10, web3-6.15.1
collected 2 items
INFO: Connecting to existing Geth node at http://localhost:8545/[hidden].
tests/test_arbitrum.py . [ 50%]
tests/test_mainnet.py . [100%]
========================== 2 passed in 0.08s ===========================Network and Provider Defaults
As we progress, we will often be running tests against a local fork. I prefer to use Anvil (part of the Foundry suite), which will automatically connect to a local Geth node at port 8545 if available. If one is not available, you should specify it in your ape-config.yaml file, like below:
ape-config.yaml
name: Ape Testing
plugins:
- name: vyper
- name: foundry
foundry:
host: auto
fork:
ethereum:
mainnet:
upstream_provider: geth
geth:
ethereum:
mainnet:
uri: http://localhost:8545In particular, pay attention to the uri key under the geth section. Adjust it as needed for your node!
Testing Deployments
To see a more sophisticated example of ape test, let’s write a simple Vyper contract with a deployment function that sets an owner address, and a view function that calculate a hypotenuse.
The ownership portion isn’t particularly relevant to this contract, but it does demonstrate that you can test and verify values exposed by public getter functions.
Then we will write a series of tests to confirm it works as expected.
hypotenuse.vy
#pragma version >=0.3.10
owner: public(address)
@external
def __init__():
self.owner = msg.sender
@external
@pure
def hypotenuse(a: uint256, b: uint256) -> decimal:
_a: decimal = convert(a, decimal)
_b: decimal = convert(b, decimal)
return sqrt(_a*_a + _b*_b)Save the contract in the contracts/ directory.
Now write a test that deploys the contract using the first testing account, accessible at accounts[0].
test_hypotenuse.py
def test_deployment(project, accounts):
hypotenuse_contract = project.hypotenuse.deploy(
sender=accounts[0]
)
assert hypotenuse_contract.owner() == accounts[0].addressRun ape test with the ecosystem:network:provider network format associated with the Foundry plugin to confirm that deploying the contract on an Anvil fork works as expected:
(ape) btd@dev:~/code/ape_testing$ ape test --network ethereum:local:foundry
========================= test session starts =========================
platform linux -- Python 3.11.8, pytest-7.4.4, pluggy-1.4.0
rootdir: /home/btd/code/ape_testing
plugins: eth-ape-0.7.10, web3-6.15.1
collected 1 item
INFO: Connecting to existing Geth node at http://localhost:8545/[hidden].
tests/test_hypotenuse.py . [100%]
========================== 1 passed in 0.30s ==========================
INFO: Stopping 'anvil' process.Now let’s transform the deployment into a fixture, which gives us simple access to a fresh deployment for subsequent tests:
import ape
import pytest
@pytest.fixture
def hypotenuse_contract(project, accounts):
hypotenuse_contract = project.hypotenuse.deploy(sender=accounts[0])
return hypotenuse_contract
def test_ownership(hypotenuse_contract, accounts):
assert hypotenuse_contract.owner() == accounts[0].addressRunning the tests again confirms it works as expected:
(ape) btd@dev:~/code/ape_testing$ ape test --network ethereum:local:foundry
========================= test session starts =========================
platform linux -- Python 3.11.8, pytest-7.4.4, pluggy-1.4.0
rootdir: /home/btd/code/ape_testing
plugins: eth-ape-0.7.10, web3-6.15.1
collected 1 item
INFO: Starting 'anvil' process.
tests/test_hypotenuse.py . [100%]
========================== 1 passed in 0.29s ==========================
INFO: Stopping 'anvil' process.Now add another test to assert that the calculation works correctly:
def test_calc(hypotenuse_contract):
assert hypotenuse_contract.hypotenuse(1, 2) == Decimal("2.2360679774")NOTE: the Vyper decimal type uses 10 decimal places, so the test asserts equality with a Decimal value of that same precision.
ANOTHER NOTE: I am using uint256 for the inputs, since Ape throws a ConversionError exception when I attempt to set a function argument typed as a decimal. Ape currently does not have a converter to handle Vyper’s decimal type, so I will investigate and submit a PR if I can resolve this.
FOLLOW-UP: my pull request to fix this was accepted, so decimal inputs will work as expected after the next release.
Testing Bad Inputs
The hypotenuse function above is very optimistic. It assumes that you will provide reasonable values that will not result in overflows, underflows, underruns, overruns and all sorts of other nasty things.
We want to guard against that in production, so let’s write a test that we’d expect to behave strangely:
def test_calc_with_maximum_inputs(hypotenuse_contract):
hypotenuse_contract.hypotenuse(2**256 - 1, 2**256 - 1)Running it results in a failure:
(ape) btd@dev:~/code/ape_testing$ ape test --network ethereum:mainnet-fork:foundry
========================= test session starts ==========================
platform linux -- Python 3.11.8, pytest-7.4.4, pluggy-1.4.0
rootdir: /home/btd/code/ape_testing
plugins: eth-ape-0.7.10, web3-6.15.1
collected 3 items
INFO: Starting 'anvil' process.
INFO: Connecting to existing Geth node at http://localhost:8545/[hidden].
tests/test_hypotenuse.py ..F [100%]
=============================== FAILURES ===============================
____________________ test_calc_with_maximum_inputs _____________________
/home/btd/code/ape_testing/tests/test_hypotenuse.py:20: in test_calc_with_maximum_inputs
hypotenuse_contract.hypotenuse(2**256 - 1, 2**256 - 1)
E ape_vyper.exceptions.IntegerUnderflowError: Integer underflow
hypotenuse_contract = <hypotenuse 0x31403b1e52051883f2Ce1B1b4C89f36034e1221D>
------------------------ Captured stdout setup -------------------------
INFO: Confirmed 0xb0d8533b622261c3f90914567895f848222e95a39ca34bb139c2fd3a421443fd (total fees paid = 185085000000000)
SUCCESS: Contract 'hypotenuse' deployed to: 0x31403b1e52051883f2Ce1B1b4C89f36034e1221D
-------------------------- Captured log setup --------------------------
INFO ape:provider.py:567 Confirmed 0xb0d8533b622261c3f90914567895f848222e95a39ca34bb139c2fd3a421443fd (total fees paid = 185085000000000)
SUCCESS ape:logging.py:34 Contract 'hypotenuse' deployed to: 0x31403b1e52051883f2Ce1B1b4C89f36034e1221D
======================= short test summary info ========================
FAILED tests/test_hypotenuse.py::test_calc_with_maximum_inputs - ape_vyper.exceptions.IntegerUnderflowError: Integer underflow
===================== 1 failed, 2 passed in 1.33s ======================
INFO: Stopping 'anvil' process.It fails with an IntegerUnderflowError, which is thrown by the conversion from the uint256 to a decimal. Vyper’s decimal type has a maximum value of (2167 - 1) / 1010, so it’s clear that the maximum uint256 value cannot be converted to the constrained type.
We can test for this condition explicitly with the ape.reverts() decorator:
def test_calc_with_maximum_inputs(hypotenuse_contract):
with ape.reverts():
hypotenuse_contract.hypotenuse(2**256 - 1, 2**256 - 1)(ape) btd@dev:~/code/ape_testing$ ape test --network ethereum:mainnet-fork:foundry
========================= test session starts ==========================
platform linux -- Python 3.11.8, pytest-7.4.4, pluggy-1.4.0
rootdir: /home/btd/code/ape_testing
plugins: eth-ape-0.7.10, web3-6.15.1
collected 3 items
INFO: Starting 'anvil' process.
INFO: Connecting to existing Geth node at http://localhost:8545/[hidden].
tests/test_hypotenuse.py ... [100%]
========================== 3 passed in 0.45s ===========================
INFO: Stopping 'anvil' process.The test expects the call to revert, which it does, so the test passes.
Now ask yourself: do you prefer to rely on the built-in type checks from Vyper to guard against this, or add additional checks?
For demonstration, let’s add a guard inside the contract function that reverts with a custom message whenever an input of greater than (2167 - 1) * 1010 is provided:
@external
@pure
def hypotenuse(a: uint256, b: uint256) -> decimal:
if a > 18707220957835557353007165858768422651595:
raise "a too large"
if b > 18707220957835557353007165858768422651595:
raise "b too large"
_a: decimal = convert(a, decimal)
_b: decimal = convert(b, decimal)
return sqrt(_a*_a + _b*_b)And modify the test so the revert messages are explicit:
def test_calc_with_maximum_inputs(hypotenuse_contract):
with ape.reverts(expected_message="revert: a too large"):
hypotenuse_contract.hypotenuse(2**256 - 1, 0)
with ape.reverts(expected_message="revert: b too large"):
hypotenuse_contract.hypotenuse(0, 2**256 - 1)Run it again and observe that both tests pass with the expected revert reason:
(ape) btd@dev:~/code/ape_testing$ ape test --network ethereum:mainnet-fork:foundry
========================= test session starts ==========================
platform linux -- Python 3.11.8, pytest-7.4.4, pluggy-1.4.0
rootdir: /home/btd/code/ape_testing
plugins: eth-ape-0.7.10, web3-6.15.1
collected 3 items
INFO: Starting 'anvil' process.
INFO: Connecting to existing Geth node at http://localhost:8545/[hidden].
tests/test_hypotenuse.py ... [100%]
========================== 3 passed in 0.50s ===========================
INFO: Stopping 'anvil' process.And finally, write a test that would pass both of those checks, but overflow the final calculation:
def test_calc_with_valid_inputs_that_overflow_final_calc(
hypotenuse_contract
):
hypotenuse_contract.hypotenuse(
18707220957835557353007165858768422651595,
18707220957835557353007165858768422651595,
)Running the test and observe the revert on overflow:
(ape) btd@dev:~/code/ape_testing$ ape test --network ethereum:mainnet-fork:foundry
========================= test session starts ==========================
platform linux -- Python 3.11.8, pytest-7.4.4, pluggy-1.4.0
rootdir: /home/btd/code/ape_testing
plugins: eth-ape-0.7.10, web3-6.15.1
collected 3 items
INFO: Starting 'anvil' process.
INFO: Connecting to existing Geth node at http://localhost:8545/[hidden].
tests/test_hypotenuse.py ..F [100%]
=============================== FAILURES ===============================
____________________ test_calc_with_maximum_inputs _____________________
/home/btd/code/ape_testing/tests/test_hypotenuse.py:27: in test_calc_with_maximum_inputs
hypotenuse_contract.hypotenuse(
E ape_vyper.exceptions.IntegerOverflowError: Integer overflow
hypotenuse_contract = <hypotenuse 0x31403b1e52051883f2Ce1B1b4C89f36034e1221D>
------------------------ Captured stdout setup -------------------------
INFO: Confirmed 0x34594b24ea3066f5b11495d8c3e08a1fbc0b7ca0e047dc262ded72a383cba3e2 (total fees paid = 235176000000000)
SUCCESS: Contract 'hypotenuse' deployed to: 0x31403b1e52051883f2Ce1B1b4C89f36034e1221D
-------------------------- Captured log setup --------------------------
INFO ape:provider.py:567 Confirmed 0x34594b24ea3066f5b11495d8c3e08a1fbc0b7ca0e047dc262ded72a383cba3e2 (total fees paid = 235176000000000)
SUCCESS ape:logging.py:34 Contract 'hypotenuse' deployed to: 0x31403b1e52051883f2Ce1B1b4C89f36034e1221D
======================= short test summary info ========================
FAILED tests/test_hypotenuse.py::test_calc_with_maximum_inputs - ape_vyper.exceptions.IntegerOverflowError: Integer overflow
===================== 1 failed, 2 passed in 0.82s ======================
INFO: Stopping 'anvil' process.This can be converted to an expected-to-revert case in the same way as before:
def test_calc_with_valid_inputs_that_overflow_final_calc(
hypotenuse_contract
):
with ape.reverts():
hypotenuse_contract.hypotenuse(
18707220957835557353007165858768422651595,
18707220957835557353007165858768422651595,
)Which passes as expected:
(ape) btd@dev:~/code/ape_testing$ ape test --network ethereum:mainnet-fork:foundry
========================= test session starts ==========================
platform linux -- Python 3.11.8, pytest-7.4.4, pluggy-1.4.0
rootdir: /home/btd/code/ape_testing
plugins: eth-ape-0.7.10, web3-6.15.1
collected 4 items
INFO: Starting 'anvil' process.
INFO: Connecting to existing Geth node at http://localhost:8545/[hidden].
tests/test_hypotenuse.py .... [100%]
========================== 4 passed in 0.69s ===========================
INFO: Stopping 'anvil' process.You can also test for conditions that would throw an exception within Ape instead of relying on the contract to revert:
Notice above that the overflow exception was of type ape_vyper.exceptions.IntegerOverflowError. This can be caught using the pytest.raises() decorator:
def test_calc_with_valid_inputs_that_overflow_final_calc(
hypotenuse_contract
):
with pytest.raises(
ape_vyper.exceptions.IntegerOverflowError,
match="Integer overflow",
):
hypotenuse_contract.hypotenuse(
18707220957835557353007165858768422651595,
18707220957835557353007165858768422651595,
)Which catches the expected exception, and the test passes:
(ape) btd@dev:~/code/ape_testing$ ape test --network ethereum:mainnet-fork:foundry
========================= test session starts ==========================
platform linux -- Python 3.11.8, pytest-7.4.4, pluggy-1.4.0
rootdir: /home/btd/code/ape_testing
plugins: eth-ape-0.7.10, web3-6.15.1
collected 4 items
INFO: Starting 'anvil' process.
INFO: Connecting to existing Geth node at http://localhost:8545/[hidden].
tests/test_hypotenuse.py .... [100%]
========================== 4 passed in 0.63s ===========================
INFO: Stopping 'anvil' process.In Summary
Test your code!
Using Ape’s built-in testing functions will really help when you’re deep in the weeds of contract development. If you can get into a rhythm of add feature → test basic functionality → test corner cases → repeat, you will make quick progress and save a lot of frustration later.
I will cover testing again when I do another round of Vyper contract development, so look out for that soon.