JSON Data Exchange
Friendship Ended With CSV, Now JSON Is My Best Friend
My first attempts at developing an automated input pipeline relied on a simple approach: fetch pool data from the network, generate arbitrage paths with NetworkX, then automatically load them into the bot during startup.
The approach is still a good one, but I chose a very simple data format to store the data: comma-separated values (CSV). CSV is essentially a spreadsheet, stored in an unformatted text file. Each row is a line, and each cell is separate by a comma (or some other delimiter).
For very simple use cases, the CSV was fine. However there’s a pretty significant flaw: the format is not flexible about storing multiple values inside a cell. A two-pool arbitrage requires a borrow pool, a swap pool, and a borrow token. A multi-pool arbitrage (triangle) requires a borrow pool, two or more swap pools, and a borrow token. To make the automated path loading work in a generic way, the CSV must have a predictable format, where the elements are always in the same position. This quickly breaks down into an irritating exercise in string manipulation, so I began looking for alternatives.
JSON Saves the Day!
This lesson will cover some basic introduction to loading and storing JSON data, then build an example of automated path building using JSON instead of CSV. Future versions of the bot will use JSON instead of CSV, and I will include JSON-compatible examples for common tasks (LP fetching, path building, path loading) in the #files channel on Discord.
Simple JSON Example
Start up a Python console and import the
>>> import json
Now create a list with some arbitrary data, and create a file object to interact with the file system:
>>> data = [1, 2, 'devil', True, 3.14159] >>> file = open('test.json', 'w')
To transform Python data into JSON format, we use the
>>> json.dumps(data) '[1, 2, "devil", true, 3.14159]'
Notice that the JSON boolean
true has differing capitalization compared to the Python boolean
True. They are otherwise mapped one-to-one, but you must take care to pass JSON data through their equivalent translation functions (
Now record that data to our file using the
dump() function (notice the 's' is omitted). Then close the file object and terminate the console:
>>> json.dump(data,file) >>> file.close() >>> exit()
Now back on the terminal, inspect the contents of
(.venv) devil@hades:~/bots$ cat test.json [1, 2, "devil", true, 3.14159]
The data is now safely stored, so let’s start the Python console again and load that data via JSON instead of entering it by hand:
>>> import json >>> file = open('test.json') >>> data = json.load(file) >>> data [1, 2, 'devil', True, 3.14159] >>> file.close() >>> exit()
This is a very simple demonstration, but it shows clearly how JSON can be used to save and load data into Python between separate runs.
Now let’s move to more complex uses.
JSON LP Fetcher
This script will replace the CSV LP fetcher first introduced HERE. I will write it for Ethereum (using UniswapV2 and Sushiswap) instead of Avalanche, but the basic structure is the same.
Instead of translating lists to CSV, we will build a list of dictionaries and store that to JSON. The advantage of a dictionary is that keys can be labeled, so differing arbitrage paths can have different data depending on their needs, and the dictionary can be expanded later without affecting any load-time logic. Each dictionary will be specific to an individual LP, and the dictionaries will be aggregated together inside a list container.