Zero Net : P2P websites without central web server

ZeroNet is a very interesting distributed platform for content publishing and sharing. It has some strong properties, which are very interesting to bypass censorship and online hosting costs.

According to its proponents, it provides:

Open, free and uncensorable websites,
using Bitcoin cryptography and BitTorrent network

In this post we will see how ZeroNet can provide these properties.

Interesting features

  • Password-less BIP32 based authorization
    • Your account is protected by the same cryptography as your Bitcoin wallet.
    • You will need your private key to make changes to a given site
    • the site’s address is also BIP32-based
  • Real-time updated sites
    • visitors are notified of changes
    • and they update their browser contents if needed
  • Namecoin .bit domains support
    • for human intelligible site names
  • SQL Database support
    • runs locally
    • dedicated API for database access
  • Anonymity:
    • Full Tor network support with .onion hidden services instead of ipv4 addresses
  • TLS encrypted connections.
  • Automatic, uPnP port opening.
    • can be interesting for visitors using TCP/IP connections
  • Works with any browser/OS.

How does it work ?

According to its proponents, when you vist a ZeroNet site, this is what happens:

ad-hoc-and-p2p_1014841493414094001

ad-hoc-and-p2p_7615479204197627471

Digging a little bit deeper into detail, we find the following architecture:

screen-shot-02-05-17-at-06-51-pm-001

  1. Create a new site: a BIP-32 address and its respective public and private keys are created
  2. A BitTorrent compatible “.torrent” is created for announcing purposes
  3. The BIP-32 address is announced in the BitTorrent network trackers
  4. The BIP-32 address can be shared with the public
  5. A new visitor asks the BitTorrent trackers for other visitors serving the BIP-32 address
  6. The new visitor connects to the first ZeroNet server (using TCP/IP or TOR) and asks for “content.json“.
  7. After validating the “content.json” contents, it asks for the remaining site contents and validates its signature. It stores all the site’s contents locally
  8. The new visitor announces that it is also serving the site

Site Name

ZeroNet uses a BIP-32 address for the site name. This address can be concatenated to the local ZeroNet server.

For example:

Domain names

Namecoin .bit domains are supported: these addresses can be concatenated to the local ZeroNet server.

  • Example:
  • Other examples:

    screen-shot-11-02-16-at-12-16-pm

  • https://namecoin.org/
    • «Namecoin was the first fork of Bitcoin and still is one of the most innovative “altcoins”. It was first to implement merged mining and a decentralized DNS. Namecoin was also the first solution to Zooko’s Triangle, the long-standing problem of producing a naming system that is simultaneously secure, decentralized, and human-meaningful.»
    • «Namecoin is an experimental open-source technology which improves decentralization, security, censorship resistance, privacy, and speed of certain components of the Internet infrastructure such as DNS and identities.»

Other Benefits

Some additional benefits include:

  • Transparent sites: all the site code for all ZeroNet sites is available and can be audited if needed
  • One click cloning
    • it is very easy to clone an existing site and modify it
  • No back-end code
    • everything runs locally, there is no central server
      • even the database runs locally
    • this means the code runs very fast
    • and that there is no single point of failure or censorship
    • API for local database access
    • offline access is naturally supported
  • Instant CDN (Content Delivery Network)
    • the system behaves naturally as a CDN because it naturally produces multiple site copies (one for each visitor)
  • It is able to run on non-internet networks
    • Mesh-networks, Bluetooth, WiFi-direct, for example
  • Zero discrimination
    • everyone can build a site for a zero cost infrastructure
  • Zero trust
    • it is impossible to modify a site without the original private key (for the BIP32 address)
    • NOTE that it is not impossible to make a copy of the site, it’s just impossible to change a site at a given address without the private key

Other Features and limitations

  • No torrent-like file splitting for big file support
  • Support for anonymous communication (built-in full Tor support added)
  • File transactions are not compressed
  • Encrypted communication (TLS encryption added)
  • No private sites
  • Local ZeroNet service uses 15441 TCP/IP port (or a TOR hidden service) for inbound connections from other visitors

Demo

Step 0 – Install ZeroNet

  • Decompress the package and run the script:
    • ZeroNet.cmd (win),
    • ZeroNet(.app) (osx),
    • ZeroNet.sh (linux)
root@osboxes:~/Downloads# cd ZeroBundle/
root@osboxes:~/Downloads/ZeroBundle# ls -l
lib  
LICENSE.txt  
Python  
ssl  
ZeroNet.sh
root@osboxes:~/Downloads/ZeroBundle# ./ZeroNet.sh 
Downloading https://github.com/HelloZeroNet/ZeroNet/archive/master.zip to ZeroNet directory.
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Downloaded.
Extracting... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Done.
Starting ZeroNet/"start.py"...
- Starting ZeroNet...
[15:24:11] - OpenSSL loaded, version: 01000207F
[15:24:12] - Version: 0.4.1 r1536, Python 2.7.11 |Continuum Analytics, Inc.| (default, Dec  6 2015, 18:08:32) 
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)], Gevent: 1.0.2
[
15:24:12] - Creating FileServer....
[15:24:12] TorManager Tor controller connect error: AssertionError: Tor version >=0.2.7.5 required, found: 0.2.2.35 in TorManager.py line 159
[15:24:12] - Creating UiServer....
[15:24:12] Site:1Name2..hM9F Content.json not exist: data/1Name2NXVi1RDPDgf5617UoW7xA6YrhM9F/content.json
[15:24:12] - Removing old SSL certs...
[15:24:12] - Starting servers....
[15:24:12] Ui.UiServer --------------------------------------
[15:24:12] Ui.UiServer Web interface: http://127.0.0.1:43110/
[15:24:12] Ui.UiServer --------------------------------------
[15:24:12] - Opening browser: default_browser...
[15:24:12] Site:1Name2..hM9F Content.json not exist: data/1Name2NXVi1RDPDgf5617UoW7xA6YrhM9F/content.json
[15:24:16] FileServer Checking port 15441 using portchecker.co...
[15:24:21] FileServer [BAD :(] Port closed: Port 15441 is closed.
[15:24:21] FileServer Trying to open port using UpnpPunch...
[15:24:34] Site:1HeLLo..Tf3D Content.json not exist: data/1HeLLo4uzjaLetFx6NH3PMwFP3qbRbTf3D/content.json
[15:24:36] FileServer UpnpPunch run error: UpnpError: Failed to communicate with igd using port 15441 on local machine after 3 tries. in FileServer.py line 73 > UpnpPunch.py line 319 > UpnpPunch.py line 310


screen-shot-11-02-16-at-12-11-pm

screen-shot-11-02-16-at-12-12-pm

Step 1 – Create a new site

You can create an empty site through the command-line, as follows:

# zeronet.cmd siteCreate

- Starting ZeroNet...
 - OpenSSL loaded, version: 01000201F
 - Version: 0.4.1 r1536, Python 2.7.12 (v2.7.12:d33e0cf91556, Jun 27 2016, 15:19:
 22) [MSC v.1500 32 bit (Intel)], Gevent: 1.1.2
 - Generating new privatekey...
 - ----------------------------------------------------------------------
 - Site private key: 5KfnCgnyqNZWrRJ8tWeg1c5jaXXXXXXXXXXXXXXXXX
 -                   !!! ^ Save it now, required to modify the site ^ !!!
 - Site address:     1KVaAeeVhhW43FsYPWiEK7FSTuXXXXXXXX
 - ----------------------------------------------------------------------
 ? Have you secured your private key? (yes, no) >

Alternatively, you can clone an existing site (much easier option to start with).

Step 2 – View the new site contents

  • In a browser, we can access the new site with the following URL:
  • http://127.0.0.1:43110/1KVaAeeVhhW43FsYPWiEK7FSTuXXXXXXXX
  • Note that the URL to access a ZeroNet site is composed of the local service URL concatenated with site’s BIP-32 address

screen-shot-11-02-16-at-12-10-pm

Step 3 – Update the new site contents

ZeroNet provides some site templates which you can use to quickly add your content.

For example, the Blog and Talk templates allow the content files to be changed through the web interface without any coding. You can clone the ZeroBlog site to create a new blog.

screen-shot-11-02-16-at-01-24-pm

screen-shot-11-02-16-at-03-41-pm

Accessing the new site from another host:

screen-shot-11-02-16-at-04-01-pm

If you want to update your site files manually, you will have to perform some more steps from the command-line:

  • Update the site files located in data/[your site address key] 
    (eg:1KVaAeeVhhW43FsYPWiEK7FSTuXXXXXXXX).
  • When your site is ready run:
$ zeronet.py siteSign 1KVaAeeVhhW43FsYPWiEK7FSTuXXXXXXXX 
- Signing site: 1KVaAeeVhhW43FsYPWiEK7FSTuXXXXXXXX
... Private key (input hidden): 
<enter your private key>

  • in order to inform peers about the changes you made you need to run:
$ zeronet.py sitePublish 1KVaAeeVhhW43FsYPWiEK7FSTuXXXXXXXX
... Site:13DNDk..bhC2 Publishing to 3/10 peers... 
Site:13DNDk..bhC2 Successfuly published to 3 peers 
- Serving files.... 

Stats

You can check your local service’s stats using the following URL:

rev1536 | 88.210.64.173 | Opened: False | Crypt: ['tls-rsa'] | In: 7.23MB, Out: 0.20MB | Peerid: -ZN0041-sbW9XT0WjF9k |
Connections (4, total made: 109):
id proto type ip open crypt ping buff bad idle open delay out in last sent waiting version sites
86 v2 out 112.115.5.254:15441 True ECDHE-RSA-AES128-GCM-SHA256 0.515 0 0 716 6149 0.000 2kB 1kB listModified [] 0.4.1 r1536 3
88 v2 out 78.213.146.86:15441 True AES128-GCM-SHA256 0.140 0 0 716 6130 0.000 2kB 1kB pex [] 0.3.6 r915 3
94 v2 out 78.135.51.152:15441 True ECDHE-RSA-AES128-GCM-SHA256 0.374 0 0 716 6105 0.000 1kB 0kB listModified [] 0.4.1 r1536 2
96 v2 out 179.8.205.91:15441 True ECDHE-RSA-AES128-GCM-SHA256 0.359 0 0 613 6103 0.000 0kB 0kB listModified [] 0.4.1 r1536 2

Tor hidden services (status: OK (7 onion running)):
- 1KVaAeeVhhW43FsYPWiEK7FSTuHdGnbeoX: hyvex7lfikk277kq
- global : bkgn22ygvetxb3ze
- 1BLogC9LN4oPDcruNz3qo1ysa133E9AGg8: 2mtnv2xrii6rrym4
- 131a4VbWAo21oeLffGfty9DVy4SnnPHCND: xzykj3o5yi6vd2xc
- 1Q9xoVam2Kx4Y5r76QkPDCLVMq9t4jRqZc: bol2gf6sil5jbf3n
- 1Name2NXVi1RDPDgf5617UoW7xA6YrhM9F: 6j2u2jiecp3l2d3b
- 1HeLLo4uzjaLetFx6NH3PMwFP3qbRbTf3D: ifnzspvmmwuori33

Db:
- 128.979s: data/content.db
- 127.196s: data/1BLogC9LN4oPDcruNz3qo1ysa133E9AGg8/data/zeroblog.db
- 128.952s: data/1Q9xoVam2Kx4Y5r76QkPDCLVMq9t4jRqZc/data/zeroblog.db

Sites:
address connected peers content.json out in
131a4VbWAo21oeLffGfty9DVy4SnnPHCND [] 0/0/0 0 (loaded: 0) 0kB 0kB
1BLogC9LN4oPDcruNz3qo1ysa133E9AGg8 [88, 86] 2/2/376 465 (loaded: 52) 0kB 3458kB
1HeLLo4uzjaLetFx6NH3PMwFP3qbRbTf3D [94, 96, 86, 88] 4/4/630 1 (loaded: 1) 0kB 807kB
1KVaAeeVhhW43FsYPWiEK7FSTuHdGnbeoX [] 0/0/0 1 (loaded: 1) 0kB 0kB
1Name2NXVi1RDPDgf5617UoW7xA6YrhM9F [94, 96, 86, 88] 4/4/647 1 (loaded: 1) 0kB 818kB
1Q9xoVam2Kx4Y5r76QkPDCLVMq9t4jRqZc [] 0/0/0 3 (loaded: 3) 0kB 0kB

Final Remarks

ZeroNet is a promising technology for the Zero Marginal Cost Society, as it enables democratic access to publishing resources for creative commons sharing, without having to use any private enterprise infrastructure site to host your site content.

This means that ZeroNet is a very interesting technology  if you are worried that your site could be censored by any public authority or private entity, or if you would simple like to use it freely, as it does not imply any fee or payment to external entities.

References

Advertisements

BitTorrent Architecture Overview

According to its proponents, BitTorrent is “a free speech tool”.

This is indeed the case, as it allows its users to distribute content without a centralized authority, using mainly each user’s network and computing resources in a distributed fashion.

A user’s network and computing resources are somewhat shared with the other users (also called “peers” in this context), so that everyone can benefit from expanded availability and reduced censorship properties of the BitTorrent network. The content can be stored in multiple peers at the same time: so that if a peer goes down, the content can still be obtained from the remaining peers that have a complete or partial copy of the content.

Additionally, BitTorrent allows a user to download content from multiple peers, instead of downloading it from a single server. This feature can often enable faster download speeds for its users.

Privacy can also be enhanced by using BitTorrent with minimal or no logging, whenever possible. This contrasts with downloading content from a server, which is usually logged.

Main components/terms

BitTorrent has a complex terminology. So it will be interesting to clarify its meaning.

Let’s start with:

  • Original file/content
    • The content to be published/shared with other users
  • Downloader
    • Application that implements the BitTorrent protocols and specifications
      • there are several such applications
      • examples: uTorrent, qbittorrent
  • “.torrent” file (a.k.a. Metadata file)
    • a “summary” file that summarizes the original file contents
      • the user can share the metadata file with other users, so that they can download the shared content
    • “info_hash” is a SHA hash for the “info” section in the “.torrent” file
      • this hash is used to identify the torrent and for searching for peers seeding the torrent
  • Magnet URI
    • a URI for a metadata file
      • even simpler to share than a file
      • the user can share the magnet URI with other users, so that they can download the shared content
  • Piece
    • a segment of the original file
      • the original file is split in multiple “pieces” for downloading and uploading
  • Peer
    • another user with whom to share file “pieces”
  • Swarm
    • a group of peers sharing the same “torrent”
      • downloading
      • or uploading (also known as “seeding”)
    • identified by the torrrent’s “info_hash”
  • Seed
    • a peer that has the entire “torrent” contents
      • has already downloaded all the “pieces” (or is the original publisher)
  • Tracker
    • an auxiliary service that behaves as a kind of “name server”
    • maps torrent’s “info_hashes” in lists of peers that are seeding each torrent
    • peers “announce” that their are “interested” in the torrent identified by a given “info_hash” and simultaneously receive a list of peers that also “interested”
      • “interested” peers are willing to upload and download “pieces” of the torrent
    • trackers are setup and maintained by voluntary entities
  • DHT – Distributed Hash Table
    • another auxiliary service implemented mostly by the full set of BitTorrent nodes
    • also behaves as a kind of “name server”
    • peers “announce” that their are “interested” in the torrent identified by a given “info_hash” and simultaneously receive a list of peers that also “interested”

Step 1. Creating & sharing a new torrent + swarm

screen-shot-10-21-16-at-06-33-pm

In order to publish a new torrent, a user typically has to perform the following steps:

  • 1. Create the “.torrent” metadata file
    • select which content (local files) to include in the new “.torrent”
    • optionally select which tracker(s) will be used to announce the new torrent
    • optionally select if the torrent will be private (not announced in the DHT nor in public trackers) or if it will be public (announced in the DHT and in public trackers)
    • optionally select “web seeds” for the content
      • (these are just HTTP URLs pointing to some web server that is also serving the same content via HTTP)
  • 2. Save the “.torrent” metadata file and “Magnet URI” for later use
  • 3. Announce and Seed the new “.torrent”
    • Announce
      • In the “Trackers” defined in the metadata
      • In the DHT
      • For Local Peers (in the same LAN)
    • Seed
      • allow any interested peer to download from the initial seeder
    • the “info_hash” is used to uniquely identify the “.torrent” file
  • 4. Share the “.torrent” file or Magnet URI with the intended peer audience using means external to the BitTorrent network
    • web sites, emails, chat, sms, …

Example using uTorrent (Windows):

screen-shot-10-20-16-at-07-50-pm

Example using qbittorrent (Linux):

screen-shot-10-20-16-at-07-20-pm

Step 2. Finding and Joining peers for a given torrent/swarm

screen-shot-10-22-16-at-12-28-pm

In order to find and join a torrent, a user typically has to perform the following steps:

  • 1. Search for interesting “.torrent” or Magnet URIs using means external to the BitTorrent network
    • web sites, emails, chat, sms, …
    • NOTE: BitTorrent does not provide a “content search” mechanism, as some of its predecessors did1.
  • 2. Download and Save the “.torrent” file or Magnet URI
  • 3. Calculate the “info_hash” for the “.torrent” file or Magnet URI
  • 4. Query for peers seeding the torrent and Join the Swarm
    • using the calculated “info_hash
    • announce interest in the torrent
    • get lists of peers
    • through the Trackers, the DHT, Local Peer Discovery
    • ( and also through Peer Exchange, after some peers have been found through the other mechanisms )
  • 5. Download and Upload content “Pieces” from/to other Peers
  • 6. Reassemble the original file by assembling all downloaded pieces together
  • 7. Become a Seed
    • Keep seeding the contents
    • seeding all “pieces”

Adding a new torrent file example:

Screen Shot 10-22-16 at 01.18 PM.PNG

Screen Shot 10-22-16 at 01.19 PM.PNG

screen-shot-10-22-16-at-01-19-pm-001

Trackers example:

screen-shot-10-19-16-at-12-15-pm

DHT Example:

screen-shot-10-19-16-at-12-16-pm

Step 3. Downloading and Uploading torrent/swarm pieces

During the download and even after a full download, the “Downloader” also “seeds” the “pieces” it has already downloaded. This means that other peers can download these “pieces” from it. This allows for extra availability and extra bandwidth, when there are many peers in a swarm.

When one or more peers go offline, some of the torrent “pieces” may become “not  available”. This means that other peers which still don’t have those “pieces” will no be able to download the full torrent until those “pieces” become available again.

As previously mentioned:

  • 5. Download and Upload content “Pieces” from/to other Peers
  • 6. Reassemble the original file by assembling all downloaded pieces together
  • 7. Become a Seed
    • Keep seeding the contents
    • seeding all “pieces”

Download Examples (with uTorrent)

 

Screen Shot 10-23-16 at 12.27 PM.PNG

Download status for a specific torrent/file

 

screen-shot-10-21-16-at-05-07-pm

Peers in the swarm

screen-shot-10-21-16-at-05-06-pm

Peers in the swarm

screen-shot-10-19-16-at-12-21-pm

Fully downloaded torrent/file

screen-shot-10-19-16-at-12-18-pm

Peers in the swarm with decoded countries

screen-shot-10-19-16-at-12-20-pm

Fully downloaded torrent/file in “Seeding” status

 

Final remarks

BitTorrent is a very powerful and popular free speech tool.

I will be describing more protocol details in future posts.

References