Overview

Data providers, such as miners, will have local indices of their content. They want to advertise the availability of this content so that any consumers can easily find it. Data providers will also want to revoke advertisements of content that they no longer provide.

Data providers may advertise different kinds of indices, out of an enumerated set. e.g. semantic or full indices.

Indexer nodes want to discover data providers and to track updates provided by the data providers. They also want to handle incoming content requests and resolve them to providers who have that content. Indexers may reroute client requests to other indexers if they do not handle that content.

Indexer clients want to issue 'findProvider' style requests for content to an indexer node and receive a set of providers that have that content. The response should include any other information about how to choose between providers as well as information that the provider may have requested to present to them to authenticate the request? (e.g. deal ID)

Indexer nodes discover new data providers and receive notification of content updates, by receiving notifications published on a known gossip pub-sub topic. Indexers can also discover this by looking at on-chain miner activity. The notifications let indexers know that new data is available and that index entries can be fetched from an existing or a new provider. The indexer decides if and when to pull the actual index data, and which kind of index, from a provider, based on its own policies pertaining to the data provider.

To handle any amount of index data from any number of providers, the indexer nodes are capable of scaling horizontally to distribute workload over a pool of indexers. The indexer pool size can change while the pool remains online.

Key Design Decisions

• Indexer uses multihashes, not entire CIDs, to refer to indexed content The codec in a CID is independent of the content that is indexed or how the content is retrieved from a provider.
• Indexer keeps an in-memory cache of multihashes that have been asked for by clients, unless configured for in-memory only operation Most multihashes in an index will not be requested, since clients will generally only ask for the top-level multihash for any object. Therefore only the set that is actually requested by clients is kept in memory. The remaining will be kept in space-efficient secondary storage. A generation cache eviction mechanism for the in-memory cache prevents unbounded growth.
• storethehash will serve as persistent storage Designed to efficiently store hash data, requires only two disk reads to get any data stored for a multihash. This is being evaluated against more mature systems because of a high confidence assumption that it will save significant space. However, that savings needs to be very significant to outweigh the risk and development cost of a new storage facility. The indexer will be constructed to allow different implementations to be chosen as appropriate for different deployments. Having an embedded storage implementation may reduce admin work and allow the indexer to be used more easily as a library.
• Indexer will track multihash distribution and usage The indexer will keep statistics on multihashes that can be used to predict the growth rate of data stored and cached as well as the distribution of multihashes over different providers. The usage of multihashes will also be tracked and will show the demand (by clients) for various multihahses.
• Indexer will not index IDENTITY Multihashes The Indexer will not index IDENTITY multihashes and filters out any such multihash from the received advertisements. See: Appendix C, Handling of IDENTITY Multihashes

Data Provider Interface

Miners and other data providers will have local indices of their content. Abstractly, we can think of the resources exposed by a data provider as a tree of the following:

Catalog (List of Indexes)
Catalog/<id> (Individual Index)
Catalog/<id>/multihashes (list of multihashes in the Index)

Catalog/<id>/TBD (semantic for selection)

A provider has a Catalog of one or more indices, where the current index is the set of all multihashes known to the provider, and the other indices are past versions of it, each representing the multihashes at some previous time. Semantically, changes to the catalog of indexes by the provider are seen as happening in a globally ordered log of additions and revocations, each referencing the previous action. lndexers are able to track these changes to keep their view of a provider's Index current.