Providing Data to a Network Indexer
New data providers are discovered when an announcement about the availability of a new advertisement arrives on the pub-sub channel. The indexer fetches the advertisement and learns about a new provider.
Data providers can also be discovered by independent agents, which can send announcements on behalf of advertisement publishers. This allows any type of agent to be built independently of the indexer and to inform any indexers about discovered advertisement publishers.
Advertisements can be filtered using allow and block lists. These lists are defined in the indexer config. Each list contains zero or more provider IDs.
Allow-block policy configuration
An indexer node may poll known providers for index updates. This is useful if an indexer or a provider’s publisher was offline and may have missed updates, or if a provider chooses not to advertise updates. The indexer requests the latest advertisement CID from the advertisement publisher. The publisher responds with the latest advertisement CID, which the indexer handles the same as if that CID had been received over gossip pubsub.
Polling a provider, after not receiving updates for the provider for a configured time threshold, is also used to detect unresponsive providers. This can eventually be used to delist inactive providers.
The poll schedule is specified is configurable to specify how long a provider can have no updates before it is polled, how long unsuccessful polling should continue before stopping the polling, and when to delist providers when their publishers remain unresponsive.
Polling Configuration Configuration
The polling configuration can be overridden for individual providers. This is useful in the case where polling is necessary to keep the indexer up-to-date with a provider, or where a provider’s publisher is known to be offline for long periods of time between updated.