Data Exporters

There are numerous open source data exporters to expose data to prometheus that run on the target nodes themselves including:

• Node exporter
  • System metrics - cpu, ram, disk
  • Useful in alerting node operators if their hardrives fill up
• Nginx exporter (any rate limiting reverse proxy for the matter ie HAProxy / Envoy)
  • Network metrics - Number of connections, requests, throughput
  • Useful for detecting various types of DoS attacks and network failures
• Cadvisor
  • Container metrics
  • Useful for understanding application performance
• Blackbox (http)
  • Endpoint metrics - query api/v1 and export any data
  • Useful for doing block statistics data aggregations on prometheus (ie blocks behind leader)
  • Will be partially replaced by custom exporter
• Custom exporter
  • Block metrics - time to finality, seconds / block, latency to leader
  • The most useful metrics need to be built from a custom exporter as doing so with the blackbox exporter would require too high of a sample rate. This way we can customize any kind of data we need to serve, logs or otherwise.

Each one of these exporters can be run as a side car container and either exposed directly on their 9090+ ports or over a reverse proxy like nginx with htpasswd and firewall. Due to the added complexity, this will require some automation to evangelize across the community. As such, we are also proposing an [Ansible P-Rep Roles project] that makes it easy to configure this through a single high level configuration file. Numerous nodes on the network are already using early forms of these roles which have made it easier to deploy more secure / complicated configurations that the operator would otherwise be unable to implement.

Management Services

The prometheus ecosystem have a number of standard components but can be put together in a variety of ways. Bootstrapping the clusters can be fairly involved if you deal exclusively with static configuration files as the network is dynamic and will continually need updating. Therefor effort needs to be put into automating parts of mapping the network that will change with agents that update the configuration automatically. These include,

1. Service discovery - update off api/v3 on cron
2. Alarms routes - phone numbers and contact details for node operators and status subscribers
3. Silencing of alarms - when alarms go off, we need a fast way of being able to resolve them so that we don't inadvertently raise alarm over a minor warning

With these three customized agents built in the context of managing the prometheus configuration settings will create a perfect starting point from which additional features or improvements can be implemented. Each of these agents will be built as a microservice and deployed on docker. Initial focus will be a simple docker-compose deployment but over time will be migrated to Kubernetes. A simple CI pipeline will be built for each microservice with tests performed on live infrastructure.