Summary

This document will propose the creation of a new service, Mercury, backed with Lily data.

Mercury’s objectives are twofold:

• to provide a centralized service with a user-friendly and language-agnostic API containing the different datasets (historical and monitoring) that Sentinel offers
• to help analysts and research scientists get their hands on historical Filecoin chain analytics data using tools they’re most familiar with: Python, Pandas, and Jupyter.

Background

Currently, we have Filecoin chain data in two different places:

• TimescaleDB for real-time monitoring
• BigQuery for analytics and historical analyses

However, neither of these tools have a pleasant user experience. While both can be queried with SQL for basic analytic workloads, it is neither expressive nor easy enough to perform complex data models quickly.

Moreover, while the Sentinel team currently provides the BigQuery service at no cost to the rest of the network, it comes with limitations that can hinder the quality and sustainability in the long term.

Goals

We need a service that fulfills that:

• is a Python package that can fit seamlessly with the analysts’ and scientists’ toolbelt: Python, Pandas, scikit-learn, Jupyter
• has an API with excellent user experience for analysts and research scientists
• uses historical data hosted on BigQuery and monitoring data on TimescaleDB
• provides an API key and bills each key according to compute (i.e. query) usage

Non-Goals

• Mercury does not seek to provide a solution for visualization on tools that rely primarily on SQL queries such as Grafana and Preset/Superset