JSON Data Exchange

Friendship Ended With CSV, Now JSON Is My Best Friend

My first attempts at developing an automated input pipeline relied on a simple approach: fetch pool data from the network, generate arbitrage paths with NetworkX, then automatically load them into the bot during startup.

The approach is still a good one, but I chose a very simple data format to store the data: comma-separated values (CSV). CSV is essentially a spreadsheet, stored in an unformatted text file. Each row is a line, and each cell is separate by a comma (or some other delimiter).

For very simple use cases, the CSV was fine. However there’s a pretty significant flaw: the format is not flexible about storing multiple values inside a cell. A two-pool arbitrage requires a borrow pool, a swap pool, and a borrow token. A multi-pool arbitrage (triangle) requires a borrow pool, two or more swap pools, and a borrow token. To make the automated path loading work in a generic way, the CSV must have a predictable format, where the elements are always in the same position. This quickly breaks down into an irritating exercise in string manipulation, so I began looking for alternatives.

JSON Saves the Day!

JavaScript Object Notation (JSON) is recognized as a robust data format for sharing data between isolated programs or systems. Thus it works well as a means to store and load pre-generated arbitrage paths (and some other useful data).

This lesson will cover some basic introduction to loading and storing JSON data, then build an example of automated path building using JSON instead of CSV. Future versions of the bot will use JSON instead of CSV, and I will include JSON-compatible examples for common tasks (LP fetching, path building, path loading) in the #files channel on Discord.

Simple JSON Example

Start up a Python console and import the json module.

>>> import json

Now create a list with some arbitrary data, and create a file object to interact with the file system:

>>> data = [1, 2, 'devil', True, 3.14159]
>>> file = open('test.json', 'w')

To transform Python data into JSON format, we use the dumps() function:

>>> json.dumps(data)
'[1, 2, "devil", true, 3.14159]'

Notice that the JSON boolean true has differing capitalization compared to the Python boolean True. They are otherwise mapped one-to-one, but you must take care to pass JSON data through their equivalent translation functions (dumps and loads).

Now record that data to our file using the dump() function (notice the 's' is omitted). Then close the file object and terminate the console:

>>> json.dump(data,file)
>>> file.close()
>>> exit()

Now back on the terminal, inspect the contents of test.json:

(.venv) devil@hades:~/bots$ cat test.json 
[1, 2, "devil", true, 3.14159]

The data is now safely stored, so let’s start the Python console again and load that data via JSON instead of entering it by hand:

>>> import json
>>> file = open('test.json')
>>> data = json.load(file)
>>> data
[1, 2, 'devil', True, 3.14159]
>>> file.close()
>>> exit()

This is a very simple demonstration, but it shows clearly how JSON can be used to save and load data into Python between separate runs.

Now let’s move to more complex uses.

JSON LP Fetcher

This script will replace the CSV LP fetcher first introduced HERE. I will write it for Ethereum (using UniswapV2 and Sushiswap) instead of Avalanche, but the basic structure is the same.

Instead of translating lists to CSV, we will build a list of dictionaries and store that to JSON. The advantage of a dictionary is that keys can be labeled, so differing arbitrage paths can have different data depending on their needs, and the dictionary can be expanded later without affecting any load-time logic. Each dictionary will be specific to an individual LP, and the dictionaries will be aggregated together inside a list container.