Rewrite It in Rust: Architecture

Part I: Take The Polars Pill

Last year I wrote a short Rust series (Parts I, II, III, IV) surveying the basics of the language.

I was particularly excited about the ability to integrate Rust directly into degenbot, which was pure Python at its core and relied on external libraries like Pydantic, SciPy, and CVXPY for the heavier tasks.

The “magic” of degenbot is in the focused abstraction over commonly used DeFi smart contracts that unlock integer-perfect off-chain simulation. This all lives in Python which is quite friendly for users and developers, but the performance trade-off is very awkward.

Consider this nightmare scenario: You load up a Uniswap V3 pool with a bunch of liquidity positions from a verified snapshot. All of the state for this pool lives in Python, so it can easily be shuffled around, printed, inspected, exported to Pandas/Polars, etc.

But if you want to do parallel operations on it, you’re stuck in an awkward place. You can distribute the work to a process pool, but that requires the main interpreter to serialize the necessary input data for each worker, then deserialize the result. So you end up with a lot of bottlenecked sequential work which kills the responsiveness of your main thread.

Now you might consider converting your state to a format more performance-friendly. NumPy might be your first choice, but then you discover that the underlying C library is limited to 64 bits. So you can have integer-perfect results for values up to 2**64-1, but you have to convert to a floating point for anything larger. That might be a good option for rough simulations, but then you lose the integer-perfect guarantee. Given the EVM’s 256-bit word size, you will frequently pay the accuracy cost when crossing the Python-NumPy boundary.

Then you read my Rust series and discover Alloy and its high-performance primitive types like arbitrary precision integers. Then you think “I’ll just rewrite it in Rust!" and get to work. It all goes great for a while, but then you discover that holding state in Python and passing it to Rust for calculation involves that same sequential conversion penalty from the process pool option. Essentially the calculation is fast in Rust, but it can’t start until Python completes the hand-off which is slow.

Argh! What’s a performance-minded Python dev to do?

Polars

Data science operators know that Polars is (often) the superior option to Pandas. But they probably don’t appreciate the architecture that makes it so good.

Polars uses the PyO3 crate to create Python bindings. Bindings enable Python users to execute function calls through the foreign function interface (FFI) and move data across the Rust boundary.

Though everyone thinks of Polars as a Python package, all of its critical components are written in Rust. In fact, the polars crate can be used directly without any Python involvement!

Python users interact with the wrapper layer via PyO3.

The three layer architecture is shown in depth on DeepWiki. But you can understand it pretty well by inspecting the layers involved with the DataFrame class.

Within the Rust core, DataFrame is defined in frame/dataframe.rs:

#[derive(Clone)]
pub struct DataFrame {
    height: usize,
    /// All columns must have length equal to `self.height`.
    columns: Vec<Column>,
    /// Cached schema. Must be cleared if column names / dtypes in `self.columns` change.
    cached_schema: OnceLock<SchemaRef>,
}

The Rust core also defines the PyDataFrame in dataframe/mod.rs:

#[pyclass(frozen, from_py_object)]
#[repr(transparent)]
pub struct PyDataFrame {
    pub df: RwLock<DataFrame>,
}

PyDataFrame has the #[pyclass(frozen, from_py_object)] attribute applied to it, which PyO3 uses to turn the Rust struct into a Python class of the same name. This middle layer is the “glue” that holds the two halves together.

PyDataFrame holds a RwLock that wraps the Rust DataFrame core struct. RwLock is a synchronization primitive that allows concurrent readers while still providing Mutex-style single writes when needed.

The Python package is located in the py-polars directory. The DataFrame class in dataframe/frame.py is a simple wrapper class that allows runtime construction of the PyDataFrame class shown above:

class DataFrame:
    ...
    _df: PyDataFrame
    ...

Python callers interact with this DataFrame type through the polars namespace, which keeps a reference to the PyO3-wrapped PyDataFrame instance at self._df.

The Python interaction therefore flows in this direction: polars.DataFrame (Python)→ PyDataFrame (PyO3)→ polars_core::DataFrame (Rust).

Polars users aren’t aware of the PyO3 layer, but it’s important to understand how it’s use.

Next-Gen Bot Architecture

I have been working on applying this Polars-style layered architecture to degenbot in recent weeks.

I started with a simple question “can I build a Rust core to manage pool states and calculations?” Answering a question like this is easy and fast thanks to agentic coding harnesses.

Agentic Coding For Degens