Python is a wonderful language. It’s easy to learn, flexible, works well for most tasks, and allows developers to access a huge selection of useful modules and build applications very quickly. There is a Python module for almost anything you want to do.
But we can’t avoid the fact that Python performs some tasks slowly. It’s a natural consequence of its flexibility as an interpreted language. You can write dynamic Python code in 30 seconds and then run it immediately — this has a significant runtime cost since the interpreter has to interrupt “useful” work to perform type checks, look up references, and manage the stack & heap. This is no problem for I/O bound tasks or short scripts, but the burden becomes apparent when trying to do computationally expensive work.
A common way to address Python’s performance gap is to outsource tasks to code written in another language. The bridge between the Python interpreter and non-Python code is an interface known as a “binding”. Using this approach, Python will deliver a formatted set of inputs to a foreign runtime, wait for the result, and then resume operating.
Here are some popular Python packages that use this approach:
SciPy — Fortran, C
Numpy — Fortran, C
Pandas — C
TensorFlow — C++
PyTorch — C++
There are other tools which are even more specialized:
Numba — enables JIT compilation of a subset of Python code, focusing on Numpy
Cython — enables pre-compilation of Python code blocks using a C-like language
We should be comfortable with the idea that Python is a good tool that becomes great when it outsources certain tasks to address weaknesses.
Why Rust?
If you’ve spent some time around coders on YouTube, X, or Reddit, you will see strong opinions about Rust.
Rust is used to build several tools that I’ve already featured here:
The in-joke is that anything written in Rust becomes “blazingly fast” and the solution for any performance problem is to simply “rewrite it in Rust”.
I won’t take that position here, since I still love Python and intend to use it. But I assert that Rust is a good language and an attractive option for Python programmers.
These are the key reasons I enjoy writing Rust: strong typing, memory management, simple concurrency, and simple integration with Python.
Strong Typing
Python is a dynamic, strongly typed language. Strong typing means that an object has a fixed type and cannot change. Dynamic typing refers to the variables, and means that a variable can be re-assigned to a different object with no restriction on type.
This dynamic & strong pair confuses a lot of people, who expect that strong typing implies static typing. Code like this scrambles their brain:
>>> x = 5
>>> type(x)
<class 'int'>
>>> x = 'a'
>>> type(x)
<class 'str'>But no matter how much they argue, 5 is always an integer, and 'a' is always a string:
>>> type(5)
<class 'int'>
>>> type('a')
<class 'str'>Rust takes a similar approach. A Rust variable can be re-declared with a different value of a different type. This concept is called “shadowing”, and should be comfortable to Python developers.
Python
Python has types, but they are not enforced at runtime. They likely never will! However adding type hints to your code is allowed and encouraged.
If you use a type-checking tool regularly, you will eliminate entire categories of bugs. You declare the types for your variables, as well as input and output types for functions and methods. Then you run the type-checking tool on the code, which verifies that you have not violated any of the type expectations.
I routinely run mypy using strict mode on degenbot, which keeps all the code that I write working as expected. I can’t force users to be as strict, but I can guarantee that when my own code calls itself, it does so correctly.
Writing fully typed strictly-verified Python code is challenging, but I find it both rewarding and useful.
Rust
The Rust typing paradigm is simple — you must do it, and the compiler is very strict about it.
Rust variables, methods, and functions must have a declared type. The compiler will refuse to compile your code if you’ve specified incompatible types or missed an annotation.
Python developers who use type hinting will find Rust typing quite easy and familiar.
Memory Management
The first serious language I learned was C. I enjoyed it but often ran into trouble when handling memory.
Free free to comment “skill issue” below — you’re right! Perhaps I’d be better at it with more study, but I just don’t like it and would rather not have to deal with it.
Python
Python handles memory automatically — allocating when needed and deallocating it later according to its own rules and timing. The process of deallocating memory for unused objects is known as “garbage collection”. Unfortunately, the discovery process for these unused objects carries a cost. Objects have to be tracked and regularly inspected, then cleaned up when it is safe.
This is largely invisible to us for most simple Python programs. But when working with large data sets or lots of objects, you will begin feeling the impact of the garbage collector. Garbage collection is single-threaded, so progress will halt as the interpreter does this sweep.
Rust
Rust applies an abstraction to the whole idea of memory management. It follows a paradigm introduced in C++ called Resource Acquisition Is Initialization (RAII). This means that resources are allocated when a particular object is initialized and deallocated when that object is destroyed.
Thus, the resources associated with an object are tied to the life of that object.
At a high level, Rust enforces this concept through its ownership model.
Here are the concepts that you must know:
An object can have one owner at a time
An object can be passed from one owner to another, usually as an input to a function, method, or macro
A mutable reference can be created, which allows a non-owner to manipulate the object
An immutable reference can be created, which allows a non-owner to read the object, but not manipulate it
Only one mutable reference can exist at any time, which allows a non-owner to manipulate the data
Any number of immutable references to an object can exist, which allows a non-owner to read the data.
An immutable reference may only be created if there are no mutable references
When these invariants are maintained, the programmer is guaranteed two things:
Memory corruption and out-of-bounds access is prevented — the compiler tracks the lifetime of an object, and can ensure that the data is not accessed through a dropped variable
Data races are prevented — the reference rules enforce that one non-owner (“borrower”) can modify the data within its scope, and that the data cannot be modified while any immutable references exist
When Rust programmers complain about the “borrow checker”, they refer to the compiler checking and enforcing the lifetime and reference invariants.
So instead of considering memory directly, Rust programmers consider both the ownership of objects and their lifetimes. The compiler enforces the invariants, which means that resource management is performed automatically as part of the normal control flow, and no intermittent garbage collection is necessary.
By performing resource management in this way, the impact of individual cleanup operations is minimized instead of being concentrated into unpredictable garbage collector runs.
Simple Concurrency
A given task requires many sequential operations. Modern processors are fast, but are still limited to performing one operation at a time. Most machines have multiple processors, which means they can perform multiple tasks simultaneously.
Multiprocessing comes at the cost of complexity.
Python
The Python interpreter is single-threaded by default. It runs on a single processor and executes code sequentially. Attempting to distribute work to multiple processors will involve calling external code (C, C++, Rust, etc), creating threads, or using a native process pool executor that starts external interpreters and communicate with them to offload work.
Python threads are great for I/O intensive work, but offer zero advantage for computationally-bound work since the threads still run on a single processor. Spreading two numeric calculations to two threads will offer no speedup, and will often perform worse than just running the calculations sequentially in a single thread.
I use process pool executors heavily, but there is still a natural scaling bottleneck. The main interpreter must serialize the work and send it to the external process. Then when it is done, it must receive it and deserialize it.
Further, the workers are just copies of the same Python interpreter running on another processor, so they are capped at Python-native speeds.
Rust
Rust uses a threading model. This might scare off Python developers who are used to the GIL spoiling all your fun. But Rust doesn’t implement global locking like Python, so it can used threads for concurrency even when parallelism is required.
Rust spawns operating system threads directly, and defers to it for their execution. Most operating systems will spread these threads among multiple processors, which is a simple way to unlock parallel performance.
Async / Await
Both languages have high performance async / await functionality. The most common are Python’s asyncio and Rust’s tokio.
Covering the two is beyond the scope of this introduction, but we will review them later.
Simple Integration
The fantastic PyO3 toolkit allows you to run Python within Rust or Rust within Python.
Using PyO3 and Maturin, it’s simple to write Rust functions, build them, and wrap them as Python functions that you can import.
This allows Python devs to implement Rust versions of portions of their code without having to rewrite the whole thing all at once.
Code Example — Fast Checksum
Consider the checksumming method employed by EIP-55, which encode a checksum into a hex-formatted address by capitalizing certain values that correspond to positions from its keccak hash.
It does this by upper-casing any alphabetic character where the corresponding value from the hash has a value of at least 8.
The to_checksum_address function built into the eth-utils library is simple:
def to_checksum_address(
value: Union[AnyAddress, str, bytes]
) -> ChecksumAddress:
"""
Makes a checksum address given a supported format.
"""
norm_address = to_normalized_address(value)
address_hash = encode_hex(
keccak(text=remove_0x_prefix(HexStr(norm_address)))
)
checksum_address = add_0x_prefix(
HexStr(
"".join(
(
norm_address[i].upper()
if int(address_hash[i], 16) > 7
else norm_address[i]
)
for i in range(2, 42)
)
)
)
return ChecksumAddress(HexAddress(checksum_address))Let’s rewrite this simple algorithm in Rust using the keccak256 function from the Alloy crate.