Hypercore, originally known as the Dat protocol [Ogden, 2017], and apparently now known as HolePunch, is a p2p protocol designed for versioned transfer of large files.
Merkle Trees - The underlying data model is a tree!
Specifically an ordered tree
Version Controlled - including incremental versioning
Sparse Replication - Like bittorrent, it is possible to only download part of a given dataset.
Discovery - Multiple mechanisms
DNS name servers
Data structure that supports traversing dat graphs
Message container format:
<varint - length of rest of message> <varint - header> <message>
Header consists of
info- state changes, like changing from uploading to downloading
have- telling other peers what chunks we have
unhave- you deleted something you used to have
want- tell me when you
unwant- I no longer want these!
request- Get a single chunk of specifically indexed data.
data- actually send/receive a chunk!
channel - 0 for metadata, 1 for content
Using hashes of public keys during discovery rather than the public keys themselves. Avoids needing a bunch of key rotations.
Use per-file hashing (as per BitTorrent v2 as well)
Identities as cryptographic keys is great, but need some means of giving them petnames/shortnames.
Tree-only data structures make everything append-only!
The Random Access properties are really neat! (being able to read a specific 100MB chunk within a CSV) but they come with some tradeoffs!