Miners, and other data providers will have local indices of their content. Index nodes will want to track (and discover providers) and help route incoming 'findProvider' style requests for content to provides with that content. Abstractly, we can think of the resources exposed by a data provider as a tree of

Catalog (Paginated list of indices)
Catalog/<id> (Individual Index manifest)
Catalog/<id>/CIDs (list of CIDs in the index)

The semantics of index addition and revocation should be accompanied with the same considerations as other authenticated resources, and performed such that a certificate-transparency-equivalent structure can be used to incentivize a single global view of a data provider's advertised indexes.

Semantically, we think of changes to the catalog of indexes by the provider as happening in a globally ordered log of additions and revocations, each referencing the previous action.

Advertisements

Data providers will publish advertisements on new indices on the ContentIndex gossipsub pub-sub channel, which index nodes can subscribe to. The format of these advertisement messages of catalog entries would look like

{Index: <cid of index manifest>, Previous: <cid of previous advertisement> Provider: <libp2p.AddrInfo>, Signature: <signature over index, previous>}

The latest head would also be made available as an IPNS Record.

Note: this should follow the IPFS Collections best practices.

Catalog

CIDs of individual resources are made available as IPFS-accessible content. the top-level catalog can also be accessed as a bulk list of log records for efficiency. This is done through a new multiprotocol

TODO: validate that this is the right approach

Over this protocol, an indexer requests

{
"start": n, // optional
"end": n, //optional
}

And receives a subset of the log in response

{
"totalEntries": n, // size of provider's catalog
"error": "", // optional. indicate the request was too big or could not be served
"entries": [
  "<catalog entry advertisement>",
  ...
]
}

Manifest

The manifest for an individual index entry would have the following structure