Author: @Craig Tutterow, Edge & Node Data Science, craig@edgeandnode.com
Implementation
A team of core developers has produced an oracle that:
- posts aggregated quality and cost of service data from available gateways to IPFS every 5 minutes
- posts the IPFS file metadata to Gnosis chain via a DataEdge contract
- and indexes the IPFS files in a subgraph for consumption by protocol stakeholders
The community can use this data to aggregate quality and cost of service data in different ways and build a variety of analytics and monitoring dashboards related to quality and cost of service at the Indexer and subgraph deployment levels.
Below are details on the published IPFS data
- Chain:
gnosis
- DataEdge Contract:
0x5b4293b4c0f36cb5d4448950830bc777759b6c4f (gnosisscan)
- Schema for oracle data edge events (published using a virtual
submitQoSPayload function):
- Current production QoS data topic names:
- Example subgraph indexing Oracle IPFS hash payloads (does not index or process underlying data)
Architecture
.png)
The subgraph oracle is a data pipeline based on the following components:
- Log messages from gateways are streamed to a Redpanda cluster. Redpanda is a high performance, Kafka compatible streaming platform that allows us to process a high volume of log data in a cost efficient manner using internal hardware. At peak, our Redpanda and Materialize instances are handling 300mb per seconds of data throughput.
- Log messages are consumed and aggregated into analytical tables in a Materialize database instance. Materialize is a streaming database that supports Postgres SQL and calculates OLAP results incrementally based on streaming inputs.