Filecoin Lotus (libp2p/Gossipsub)

This report is the result of ProbeLab’s audit on the Gossipsub configuration for the Filecoin network. The recommended changes are being taken into account and tracked in this issue: https://github.com/filecoin-project/core-devs/issues/218

1. PeerExchange Logic

1.1 PeerExchange flag only for bootnodes

The current implementation only enables PeerExchange for bootstrapper nodes. This means that only bootnodes share information about other peers subscribed to the topic over PRUNE messages. From our continuous network crawls, we’ve observed that there are two types of success rates related to the PeerExchange feature:

• The success rate of nodes helping with peer discovery over GossipSub: we saw that only 5 to 10 nodes from the Filecoin network share peers over PRUNE messages.
• The success rate after attempting connections to peers shared after a PRUNE message: the vast majority of shared SignedPeerRecords include non-reachable peering-information.
  • https://probelab.io/filecoin/topology#chart-dial-error-ts using the

The reviewed GossipSub configuration for Lotus and Venus nodes explains both peer discovery low success rates observed during the Nebula network crawling. This could be a limiting factor for the overall peer discovery in FIlecoin, as nodes have limited interactions with bootstrapper nodes (basically, when they join the network) and therefore receiving PRUNE messages with PX happens rarely.

• Reference: https://github.com/filecoin-project/lotus/blob/master/node/modules/lp2p/pubsub.go#L329

Recommendation:

• Enabling PeerExchange globally for all nodes participating in the Gossipsub network is recommended. However, doing so means that nodes will start providing info around peers that the node is currently connected to (PX peers are taken from the fanout peer-list). That said, enabling PeerExchange shouldn’t have a privacy implication, given that Lotus has a high threshold for connections (150 to 180 peers).
• Update the GossipSub configuration to enable the feature added in this PR:
  • WithCustomPXRecordReducer + OnlyPublicAddrsOnPeerExchange
  • https://github.com/libp2p/go-libp2p-pubsub/pull/640
  • This change will ensure that only public IPs are shared over PX, however it still depends on the reachability of the remote node.

1.2 PeerExchange is currently not supported for rust-libp2p

After a review of the existing implementations (Lotus, Venus, and Forest), we’ve identified that GossipSub’s PeerExchange feature only applies for Lotus and Venus, as they are the ones using the Go implementation. The rust implementation (Forest) does not fully support the PeerExchange feature. The current code can successfully parse and read the full PRUNE message with the shared SignedPeerRecords, but since there is no support for SignedPeerRecords in rust, the feature is effectively unsupported.

• Reference: https://github.com/libp2p/rust-libp2p/issues/2398

Recommendation:

• Extend rust-libp2p to support SignedPeerRecords and finish the PeerExchange logic in rust-libp2p.

2. Pubsub Configuration Issues

2.1 Global Flood Publishing