Motivation

A large majority of users onboard data onto the Filecoin network via an aggregator. Today the work done by aggregators is unverifiable and unprovable. The user relies on the aggregator to perform the work correctly and at the same time, it is impossible to prove to a third party that a given piece of data was included in a deal.

Requirements

Provable inclusion of Client’s data within Aggregator’s on-chain data deal
Storage Provider is able to trivially find user’s data within the larger data segment
Malicious behaviour of the aggregator or another user, whose data was aggregated, does not cause irrecoverable problems with retrieval.

Specification

The proposal is split into three parts each of them addressing a part of the requirements.

Data Segment format for sectors which facilitates inclusion proofs and indexing on the side of Storage Provider
The Proof of Data Segment Inclusion within a sector or deal
A format for Batched Merkle Inclusion Proofs
Optionally a scheme for proving and verification of non-power-of-two aligned or length data segments

Data Segments without out-of-band data

Today a sector carries a set of deals, with information about where which deal is situated stored out-of-band within Filecoin’s consensus layer. The Data Segment Format aims to allow for this data to be stored in-band, within the deal or sector data itself. The mechanism is independent of whether, in the future, Filecoin enables fully off-chain deals, where the deal maker and storage provider come to an agreement without involving Filecoin’s consensus, or the on-chain deal-making continues; it can be applied either on the sector or a deal level.

The primary goal is to inform the Storage Provider what data is stored where within the deal or sector in a verifiable way, without having to utilize the limited chain space.

Format

A data segment is a sequence of data which is guaranteed the following properties: it is possible to verify its inclusion within the container and it can be trivially found in the container by the Storage Provider to facilitate data retrieval.

It is important to highlight that on-chain deals, as realised today, fulfil the definition of a data segment but use the chain to facilitate the execution of the second property.