Rust-Rust libp2p: rust-libp2p P2P — The Rust Implementation of libp2p networking stack.

Central repository for work on libp2p

dependency status

This repository is the central place for Rust development of the libp2p spec.

Warning: While we are trying our best to be compatible with other libp2p implementations, we cannot guarantee that this is the case considering the lack of a precise libp2p specifications.

Documentation

How to use the library?

Where to ask questions?

  • In the Rust section of https://discuss.libp2p.io.
  • In the #libp2p IRC channel on freenode.
  • By opening an issue in this repository.

Notable users

(open a pull request if you want your project to be added here)

Comments

  • Our implementation of ping might not be conformant
    Our implementation of ping might not be conformant

    Jun 2, 2020

    I'm opening this issue because someone has been reporting this to me a long time ago, but I haven't verified the claim.

    In our libp2p-ping implementation we open a new substream every time we want to ping the remote, while for js-libp2p and go-libp2p the substream seems to be kept open.

    This needs be checked before actually making any change.

    difficulty:easy priority:important 
    Reply
  • [libp2p-swarm] Opt-in connection-oriented backpressure for NetworkBehaviours.
    [libp2p-swarm] Opt-in connection-oriented backpressure for NetworkBehaviours.

    Jun 2, 2020

    This is a simplified variant of https://github.com/libp2p/rust-libp2p/pull/1586 since I figured that the introduction of the configurable event_lead in https://github.com/libp2p/rust-libp2p/pull/1586 may have obscured the primary intention of the PR: To permit NetworkBehaviours to opt-in to more fine-grained connection-oriented backpressure, thereby improving upon the pending_event bottleneck. So here is a variant that removes the notion of allowing the network to "run ahead" for a configurable limit when there is a pending_event. Below is a redacted description of the PR description of #1586 that accommodates for the simplification and elaborates on the motivation.

    Motivation

    Currently, if a single background task of a connection is slow to process its events for whatever reason, the effect is that the Swarm stalls all progress until this background task is ready to receive the event (into its queue). The increase of the underlying buffers in libp2p-core are a means to delay this happening but it does not change the fact that the NetworkBehaviour is unaware of any such issues of particular connections / connection handlers until the entire Swarm stalls. A NetworkBehaviour should be allowed to opt-in to handle such backpressure from the Network more gracefully, which is what this proposal of NetworkBehaviour::inject_connections_busy provides. In such an implementation, a behaviour can e.g. react by prioritising events emitted for other peers, delay (re-)issuing events for peers of the affected connections, initiate the opening of new connections with these peers etc. The behaviour can still, the moment it deems itself unable to handle more backpressure, just start emitting Err from inject_connections_busy which will result in the swarm stalling progress via the pending_event as usual.

    Network to NetworkBehaviour

    When network connections become slow to process events, which can currently either happen because ConnectionHandler::inject_event is slow or because the task scheduler is busy or overwhelmed, the Swarm starts being unable to deliver new events emitted by the NetworkBehaviour to the respective connection(s). When that happens, it first tries to propagate backpressure to the NetworkBehaviour by calling NetworkBehaviour::inject_connections_busy, i.e. giving the behaviour the event back to be emitted again at a later time. If the behaviour rejects that event (i.e. does not implement backpressure), the Swarm buffers the pending_event as usual.

    The default implementation of NetworkBehaviour::inject_connections_busy is Err(event), meaning these changes are fully backward-compatible and without implementing NetworkBehaviour::inject_connections_busy, the existing behaviour is preserved, allowing a gradual transition and experimentation.

    Missing: What I think is really missing to make this backpressure pipeline complete is a backward-incompatible change to ConnectionHander::inject_event by allowing it to outright return (i.e. reject) the given event and thus exert backpressure on the background task which in turn would propagate backpressure back through the Network to the Swarm by not consuming new events from its inbound channel until the inject_event accepted the event.

    NetworkBehaviour to Network

    The NetworkBehaviour has no direct means for backpressure on NetworkBehaviour::inject_event as it is expected from the contract of the trait that whenever the behaviour returns Poll::Pending, it is ready to receive more events, and the single pending_event prevents further calls to inject_event.

    Polling Priority

    I changed the polling behaviour of the swarm such that it always polls the NetworkBehaviour first, until it returns Poll::Pending, which signals that the behaviour is waiting for progress and input from the Network (or the network connections signal backpressure, in which case the behaviour is also no longer polled). This has the effect that events buffered in the behaviour as a result of network activity resulting in calls to inject_event, inject_connected etc. are drained before further such calls happen, thus preventing potentially unbounded buffer growth in the behaviour if there is a lot of network activity. I clarified the documentation on the matter that a behaviour that from some point onwards never again emits Poll::Pending usually is misbehaved, as this is a behaviour that, from some point onwards, no longer needs any input or progress from the Network. I also think that it makes for easier to understand semantics of how the Swarm does the polling, as it makes clear that the NetworkBehaviour takes the lead and the underlying Network is only an input/output facility driven on demand by the behaviour.

    Reply
  • Expose a C interface
    Expose a C interface

    Jun 8, 2020

    Not sure if this has been asked about before, finding anything with just "C" in the query is a mess and "interface" "ABI" and so on return irrelevant stuff for me.

    Rust supports exporting a C-ABI interface for dynamic libraries and building C-ABI libraries (DLL/SO). Would it be in scope for this project to provide the C-interface by itself, or would the maintainers prefer that being developed as a separate crate?

    With a C interface we could use this crate inside other languages that support C-ABI, for instance C (duh), Python, PHP, Ruby, and many more, often via regular FFI tooling.

    Reply
  • Remove temporary peer ID compatibility mode.
    Remove temporary peer ID compatibility mode.

    Jun 15, 2020

    This removes the temporary compatibility mode between peer IDs using identity hashing and sha256, thus being the last step in the context of https://github.com/libp2p/rust-libp2p/issues/555. This has low priority, I just wanted to get this out as a PR as a reminder to do this.

    priority:oneday 
    Reply
  • Avoid recursive dependencies due to tests.
    Avoid recursive dependencies due to tests.

    Jun 15, 2020

    Integration tests from core and core-derived are moved into a top-level tests directory.

    Reply
  • protocols/kad: Node with incompatible protocol name can be added to routing table
    protocols/kad: Node with incompatible protocol name can be added to routing table

    Jun 16, 2020

    A Kademlia NetworkBehaviour adds a node to its routing table via the inject_connection_established hook:

    https://github.com/libp2p/rust-libp2p/blob/44236b5ba39cbd324b029cb9fb5ed77a75806706/protocols/kad/src/behaviour.rs#L1280-L1291

    This hook is triggered before the protocol negotiation on substreams. Thereby a remote node with a protocol name other than the set protocol name could be added to the local node's routing table.

    See https://github.com/libp2p/rust-libp2p/pull/1473#discussion_r440218337 for details.

    bug 
    Reply
  • Problem with use
    Problem with use

    Nov 16, 2018

    After reading the libp2p api and substrace network-libp2p source code, I tried to refer to substrace write a repo , using only the ping protocol for testing.

    However, from the operation of this repo, the inject_substream method of NodeHandler is not called, which means that the substream is not upgraded.

    I don't understand now that there is a problem with my code or a problem with the library call. After all, there is almost no documentation for this library right now.

    repo:https://github.com/driftluo/p2p-test

    output:

    success /ip4/127.0.0.1/tcp/1337
    dial true
    /ip4/127.0.0.1/tcp/40240, /ip4/127.0.0.1/tcp/1337
    sever connected PeerId(QmXh9VaHR2Fbh1QKBECbp45sh46onc4AuRn6T6L4eQA1p3)
    accept one link
    client connected PeerId(QmVdFTSJcUcys1zSM8Jx8qGf4CiGge5V4hdq4Qo4bW5A2i)
    
    discussion 
    Reply
  • Wasm: add a transport that binds to a JS libp2p transport
    Wasm: add a transport that binds to a JS libp2p transport

    Feb 26, 2019

    Using wasm-bindgen, we can write an implementation of Transport that gets passed a JsValue which is expected to conform to https://github.com/libp2p/interface-transport This way, we can use the existing js-libp2p transports if we compile Rust in the browser, instead of having to rewrite everything in Rust.

    difficulty:easy difficulty:moderate getting-started priority:oneday 
    Reply
  • FloodSub is not working as of b8a312f7d512c25f86bc16989cb588900483946d
    FloodSub is not working as of b8a312f7d512c25f86bc16989cb588900483946d

    Jan 10, 2019

    At present master (commit b8a312f7d512c25f86bc16989cb588900483946d) FloodSub doesn't seem to work properly.

    When I run provided chat example, I don't see messages being delivered between nodes. Running example with debug enabled, I see, that dialed TCP connection is dropped immediately after successful negotiation.

    Changing FloodsubHandler::connection_keep_alive() to always return true resolves this issue.

    My guess is that this happens because:

    1. libp2p closes idle (useless?) TCP connections
    2. floodsub closes the substream after successful send.

    So, we end up with no TCP connection between nodes.

    bug difficulty:moderate priority:important 
    Reply
  • Gossipsub Protocol
    Gossipsub Protocol

    Jan 29, 2019

    This is an implementation of the gossipsub protocol as in the libp2p-specs.

    This is feature-complete, except for the control-message piggy-backing and peer-tagging, which still need to be implemented.

    Currently this implementation is not backwards compatible with floodsub nodes (see #880).

    Further testing and debugging will be done over the next few weeks.

    changelog 
    Reply
  • [On ice] Gossipsub: an extensible baseline pubsub protocol,  based on randomized topic meshes and gossip
    [On ice] Gossipsub: an extensible baseline pubsub protocol, based on randomized topic meshes and gossip

    Dec 11, 2018

    This partially implements the spec, with some variations around error handling and using message hashes and topic hashes. Control-message piggy-backing is not implemented, however you can graft_peers_to_topics(), graft_peer_to_topics(), graft_peers_to_topic(), graft_peer_to_topic(), and the same for prune. i_have(), i_want(), and heartbeat() are not implemented, as well as GossipSubConfig, floodsub compatibility, testing and examples. I suggest that it is reviewed in addition to #898, and could possibly be merged wtih that.

    Gossipsub is a scalable, efficient, extensible baseline pubsub (publish-subscribe, messaging) protocol for p2p networks and decentralized applications on top of them.

    See the spec here.

    This is the specification for an extensible baseline pubsub protocol, based on randomized topic meshes and gossip. It is a general purpose pubsub protocol with moderate amplification factors and good scaling properties. The protocol is designed to be extensible by more specialized routers, which may add protocol messages and gossip in order to provide behaviour optimized for specific application profiles.

    Unlike floodsub, which broadcasts or publishes a message of a topic to all peers that are subscribed to a topic in the network, gossipsub publishes to a subset of peers, known as the target mesh degree, which is chosen as 6 here and in go-gossipsub. Peers request messages of a topic with an IWant message and state what they have recently seen in their cache of messages with an IHave message. This gossipsub implementation in Rust, once complete, will be scalable for decentralized, p2p networks, unlike floodsub, and is thus suited for networks that aim to be mainstream, such as Ethereum 2.0 (as developed in Rust with paritytech/shasper and sigp/lighthouse), parity-ethereum, Polkadot, as well as any other p2p network that uses Rust.

    Closes #521. Updated for the latest API with handlers like ProtocolsHandler and NodeHandler, NetworkBehaviour, Topology, etc.

    Because gossipsub extends on floodsub and needs to be backwards compatible with it, gossipsub reimplements and adapts much of floodsub. Therefore, please let me know if any changes are made to floodsub, and avoid making changes if feasible.

    Donations

    Donations to jamesray.eth are appreciated via https://gitcoin.co/grants/42/gossipsub. Thanks to Philippe Castonguay, Peter Kieltyka and Ric Burton from 0xHorizon Games for a 30 ETH one-off donation. Donations also help for previous contributions towards Eth 2 development; Eth docs such as the wiki, Yellow Paper; reviewing research on https://ethresear.ch and papers such as Casper FFG and CBC; plus potential future contributions towards Ethereum development e.g. with paritytech/shasper, which unlike sigp/lighthouse uses substrate, providing various benefits e.g. modularity, flexibility, integration of Wasm, libp2p, GRANDPA consensus, blockchain interoperability, automatic upgradability and governance.

    See these timesheets for details of previous contributions from June 2017 until now: https://tinyurl.com/DoDspsht and https://tinyurl.com/Ethtimesht. I think it seems reasonable to work on this full-time after I get ~$36000 USD or DAI / $50000 AUD per year, or $3000 DAI per month. That's not much, and is probably undervalued, particularly given ~1.25 years FTE has been mostly unfunded, but it's enough to work on gossipsub and Eth 2, which will of course be very useful. I will work on this in proportion to how much funding I have. >3000 DAI/m, FT. 1500 DAI/m, at least 18 hours per week, etc. By linear interpolation and dimensional analysis, H/w = X DAI/m / 3000 DAI/m * 35 h/w = X * 0.016667 hr/wk, if total funding per average month is less than 3000 DAI, otherwise at least 35 hours per week.

    Grant applications

    • Protocol Labs. Applied around June. They initially said that we should fund this, then requested a plan/timeline. I said I am not a big fan of them (probably because at that time I was still studying rust-libp2p, tokio, futures, libp2p, gossipsub, etc.) and would keep working on it. I think this was probably a dealbreaker for them.
    • Ethereum Foundation (typeform): applied Jan 14 2019, also Jan 2018 for eth 2.0 development / Drops of Diamond, with which I have made several other grant applications for.
    • https://github.com/w3f/Web3-collaboration/pull/69: rejected since they are taking a different direction with messaging protocols, and referring me upon my querying to https://github.com/w3f/messaging.
    • ECF: Jan 30 2019 (typeform)
    • https://github.com/aragon/nest/issues/132
    • Spoke with Sigma Prime about getting support for gossipsub development, however they also have limited funding and have a deadline to meet the Serenity testnet by the end of March, and feel that their implementation of gossipsub (#898) is sufficient for their needs.
    • WeTrust

    See also https://twitter.com/JamesCRay01/status/1082138888037003264.

    Reply
  • rust-libp2p hashes pubsub topics which prevents Go/JS interoperability
    rust-libp2p hashes pubsub topics which prevents Go/JS interoperability

    Sep 12, 2018

    I'm trying to demonstrate floodsub interoperability between Go, JS and Rust. In all cases, the topic name is libp2p-demo-chat. Go and JS use that topic string verbatim, but rust-libp2p hashes it to RDEpsjSPrAZF9JCK5REt3tao.

    Thoughts? I'd really like to demo the 3 languages inter-operating at some conferences this fall. Guessing you did this for privacy reasons @tomaka?

    Example output below. This one is a Go peer connecting to my bootstrapper:

    <NOTICE> Got connection from /ip4/127.0.0.1/tcp/46023
    handleIncomingRPC: subopt.GetTopicid() = 'libp2p-demo-chat'
    

    Now a JS peer connecting:

    <NOTICE> Got connection from /ip4/127.0.0.1/tcp/56098
    handleIncomingRPC: subopt.GetTopicid() = 'libp2p-demo-chat'
    

    Now a rust peer connecting:

    <NOTICE> Got connection from /ip4/127.0.0.1/tcp/56076
    handleIncomingRPC: subopt.GetTopicid() = 'RDEpsjSPrAZF9JCK5REt3tao'
    
    bug difficulty:easy getting-started 
    Reply