Skip to main content

Writing A Data Connector

info

This data connector is based on AssemblyScript.

Writing a Data connector

If you wish to use any sort of data in a job or app, you will need a data connector.

The Goal

In this example, we will set up a data connector that returns the average interest rate on the US Treasury Securities for the past 'n' number of months. See the US Treasury's Fiscal Data site for more information.

info

Data Connectors need to be deterministic for proper consensus to occur. There is no specified date on this API call, but the data is updated with each meeting of the FOMC which is infrequent (over a month). Updates in close proximity to execution could cause problems.

Getting Started

To get started we will use the Assembly script template repository which is available on Github. This repository contains a template for the data connector which can be used to fetch swaps from the Uniswapv3 subgraph, and convert the swaps into candle data. This connector could be edited for most types of GraphQL calls or used for making candles from another dataset.

tip

The AssemblyScript template can be found here: https://github.com/SteerProtocol/app-template-assemblyscript

Project Structure

Data Connectors have 3 functions that will be directly called by the Keeper nodes during runtime. These functions are necessary along with configuration form function, other helper functions and classes will likely be helpful. For more information please see the Data Connector Interface. However, this design means that as a developer you only need to implement the methods which are required for the data connector to work.

Below are the significant files and folders which you will want to get familiar with:

├── assembly      // Source code for the data connector
├── build // Output of the build process aka `yarn asbuild`
├── coverage // Coverage report for testing
├── tests // Test files with a built in test runner
├── asconfig.json // Assemblyscript config
├── index.js // Javascript entrypoint for the data connector when running tests
├── package.json // Dependencies for the data connector
tip

The template comes with assemblyscript-json which helps with parsing and encoding.

Project Setup

Once the template has been cloned, you will need to install the project dependencies. This can be done via the following command:

  yarn install
info

You will notice that there is a post-install script which will compile the ./assembly source folder and populate the ./build folder. This is done to make it easier to run the tests. We will cover this later.

Once you have set up your project, you can begin defining your data connector.

Setup Data Connector Params

info

If you are not familiar with JSON Schema the following guide is very helpful: JSON Schema Guide

For our goals, we will only require the number of months in the past to pull data for. The object limit for a single page of data is 100, and since we are targeting treasury bills, notes, and bonds, we will impose a max of 33 months. We could paginate to get even more data, but this should suffice for our goal. We add a numMonths parameter to our input config.

tip

You can find an online JSON Schema builder with examples here: Online JSON Schema Builder

At the bottom of the template, you will find the configForm which we will update with our own configuration form.

// Renders the config object in JSON Schema format, which is used
// by the frontend to display input value options and validate user input.
export function configForm(): string {
return `{
"title": "Avg Interest Rates on US Treasury Securities",
"description": "returns arrays of interest rates for treasury bills, notes, and bonds",
"type": "object",
"required": [
"numMonths"
],
"properties": {
"numMonths": {
"type": "integer",
"title": "Number of Months",
"description": "Number of months back from the present to pull data from (<=33)"
}
}
}`;
}

Once you have defined your config, any parameters must be initialized with the initialize function. The predetermined timestamp will also be passed to be used in any calls requiring time-specific information.

import { JSON } from "assemblyscript-json";

// Local Variables
var numMonths: i32;
var data: Array<JSON.Value> = []; // collector array of interest objects
const first: string = "first"; // saved to memory for comparison


// Initializes variables from the config file
export function initialize(config: string, _timestamp: i32): void {
// parse through the config and assing locals
const configObj = <JSON.Obj>JSON.parse(config);
const _numMonths = configObj.getInteger("numMonths");
if (_numMonths == null) {
throw new Error("Flawed config");
}
numMonths = i32(_numMonths._num)
}

Request & Response Logic Implementation

Now that we have all instance specific parameters we need, the node will call our main function. The main function takes in a string as input, which on the first call will always be an empty string: "". This signals the bundle that the first request payload should be returned. The Keeper Node runs off of Axios' request framework. The bundle should return request config objects as strings to the node. The Keeper node makes an Axios request with said config, and will return the response data back the the bundle by calling the main function and passing the data as a parameter. To signal to the Keeper node that the callback loop should terminate, the main function should return the string: "true". In this example we will only require one call to fetch all the data needed, but more complex logic can be written to parse through the data returned to see if conditions are met.

  // To be called back until "true" is returned,
// using the all other return payloads as axios request config objects
// Ref: https://axios-http.com/docs/req_config/
export function main(response: string): string {

if (response == '') { // Presumably the first call
return `{
"method": "get",
"url": "https://api.fiscaldata.treasury.gov/services/api/fiscal_service/v2/accounting/od/avg_interest_rates?sort=-record_date&filter=security_desc:in:(Treasury Bills,Treasury Notes,Treasury Bonds)",
"headers": {}
}`
}
else{
// We have a response, parse it and test condition, update iteration logic
const new_response = <JSON.Obj>JSON.parse(response);
const interest_array = new_response.getArr("data");
if (interest_array == null) {throw new Error("No data in response");}
// write to wasm memory
data = interest_array._arr;
// end loop
return "true";

}
}
note

There are dozens of ways to implement the callback logic. The exact design will depend on the data you are fetching, make sure to test your bundle.

To walk through the process again, the node will call main where the response parameter will initially be "first". We catch this in our 'if statement,' and return the Axios request config object for the node to call. Main is called again, this time with the response parameter set to the response from our Axios call. We then parse the response and set our array.

Transform

With the conclusion of the main callback loop, the Keeper node will call transform which returns the data in its final shape for the execution bundle. In our case, all calls will return 100 objects of our selected securities. Since we have a desired number of months, we simply splice our ordered data (already sorted by date from the api) and make our final payload.

We also make a helper function to extract the interest rate from the object.

  export function transform(): string {
const bills: Array<f32> = [];
const notes: Array<f32> = [];
const bonds: Array<f32> = [];
// desired number of objects
const d_length = numMonths * 3;
for (let i = d_length - 1; i > 0; i-=3){
// Fill our arrays
bonds.push(getInterestRate(data[i]))
notes.push(getInterestRate(data[i-1]))
bills.push(getInterestRate(data[i-2]))
}
// The JSONencoder might be preferred here
return `{"data":{
"bills": [`+ bills.toString()+`],
"notes": [`+ notes.toString()+`],
"bonds": [`+ bonds.toString()+`]
}}`;
}

function getInterestRate(record: JSON.Value): f32 {
const _obj = <JSON.Obj>record
const _interest = _obj.getString("avg_interest_rate_amt")
if (_interest == null) continue;
return f32(parseFloat(_interest._str))
}

Success!

Testing

The template comes with a number of tests simulating the various calls from the node and the front end fetching the config form. There are tests for validating the config, the first call of main, the nth call of main, the last call of main, and finally the transformation. You may need less or more tests depending on your data connector.

Let's make sure the test can run by using the following command:

  yarn test

You should see the following output:

Successful tests

Final Result

As you can see, it is very quick to create a simple data connector for the Steer Protocol! By having custom parameters and calling through axios, we are able to have a flexible platform for fetching data. Transforming the data can shape things as needed to be consumed by an app or execution bundle. In the end, the decentralized and deterministic fetching of data provides a fast and secure way of bringing information on-chain.

Next Steps

tip

Use the Steer Protocol Backtesting Platform to test your app!

Steer provides a state-of-the-art backtesting platform for all concentrated liquidity apps! Run your execution bundle on historical data and see how it would have performed! Examine risks and debug your app to improve its performance. You can find more information about this tool in the Steer Backtesting Documentation.