Project Lighting: High Level Update

Done:

• Tracing throughout the stack
• Many bugs fixed
• Cool usability features
• Some speed issues identified and resolved
• Some zero bytes issues solved
• Big refactors in go-graphsync for code clarity

WIP:

• Protocol refactor
• Using tracing data to make optimizations to the stack and solve bugs

Learnings:

• Bugs/reliability/complexity live in the “multi-repo/multi-protocol” stack — go-data-transfer & go-fil-markets retrieval
  • Results in lots of extraneous code, extra go-routines, duplicated state, unclear seperation of concerns → difficult to work on, unreliable, etc
• Speed issues may go down the stack at this point to libp2p and underlying transports

TO DO:

• Auto-retrieve retrievals bug fix/reliability sprint (~1-2 months)
• “The great simplification”:
  • State tracked together between layers of the stack
  • 1 go-routine per transfer, all layers
  • Persistence is a side process
  • All network comms down to graphsync

Bigger Picture

Graphsync + Bitswap mixing

• Graphsync stengths
  • Streaming data (HDD optimization)
  • Discovering DAG semantics
  • Already supports auth/payments
• Bitswap is good at other things
  • Splitting requests
  • Discovery (WANT-HAVE)
  • go-bitswap does query planning + content-routing