SLEEP + general docs update + more introduction content (#57)

* big docs update

* fix sleep link

* separate sustainability header

* better into and reword

* clean up terms and other places

* add intro, overview, troubleshooting. update more content

* minor change to force new deploy

* more words

* more updates and some pictures

* wordsmith intro to be more about dat uniqueness

* syncing & publishing

* add link files description

* add new gifs

* change overview to remove tutorial

* add wip tutorial content

* update build with dat

* add http file

Author: Joe Hand

assets/dat_folder.png

@@ -1,19 +1,25 @@
 {
-  "Dat": {
-    "Introduction": "modules/dat.md",
-    "How Dat Works": "how-dat-works.md",
-    "Terminology": "terms.md",
-    "FAQ": "faq.md"
+  "Using Dat": {
+    "Introduction": "intro.md",
+    "Dat Concepts": "overview.md",
+    "Command Line": "modules/dat.md",
+    "FAQ": "faq.md",
+    "Troubleshooting": "troubleshooting.md",
+    "Terminology": "terms.md"
   },
   "Cookbook": {
-    "On a Server": "cookbook/server.md",
+    "Getting Started": "cookbook/tutorial.md",
+    "Sharing files over HTTP": "cookbook/http.md",
+    "Running a Dat Server": "cookbook/server.md",
     "In the Browser": "cookbook/browser.md",
-    "Under the Hood": "cookbook/diy-dat.md"
+    "Using Dat in JS Apps": "cookbook/diy-dat.md",
+    "Using Hyperdrive FS": "cookbook/using-fs.md"
   },
-  "Ecosystem": {
+  "Dat Technology": {
     "Overview": "ecosystem.md",
-    "SLEEP": "sleep.md",
-    "Hyperdrive": "modules/hyperdrive.md",
-    "Hypercore": "modules/hypercore.md"
+    "dat-node": "modules/dat-node.md",
+    "hyperdrive": "modules/hyperdrive.md",
+    "hypercore": "modules/hypercore.md",
+    "hyperdiscovery": "modules/hyperdiscovery.md"
   }
 }

assets/desktop_share.gif

@@ -2,132 +2,124 @@
 
 In this guide, we will show how to develop applications with the Dat ecosystem. The Dat ecosystem is very modular making it easy to develop custom applications using Dat.
 
+Dat comes with a built in javascript API we use in Dat Desktop and dat command line. For custom applications, or more control, you can also use the core Dat modules separately.
+
+Use Dat in your JS Application:
+
+1. `require('dat')`: use the [high-level Dat JS API](https://github.com/datproject/dat-node).
+2. Build your own!
+
+This tutorial will cover the second option and get you familiar with the core Dat modules.
+
+### The Dat Core Modules
+
 This tutorial will follow the steps for sharing and downloading files using Dat. In practice, we implement these in [dat-node](https://github.com/datproject/dat-node), a high-level module for using Dat that provides easy access to the core Dat modules.
 
-For any Dat application, there are three essential modules you will start with:
+For any Dat application, there are two essential modules you will start with:
 
 1. [hyperdrive](https://npmjs.org/hyperdrive) for file synchronization and versioning
-2. [hyperdrive-archive-swarm](https://npmjs.org/hyperdrive-archive-swarm) helps discover and connect to peers over local networks and the internet
-3. A [LevelDB](https://npmjs.org/level) compatible database for storing metadata.
+2. [hyperdiscovery](https://npmjs.org/hyperdiscovery) helps discover and connect to peers over local networks and the internet
 
-The [Dat CLI](https://npmjs.org/dat) module itself combines these modules and wraps them in a command-line API. These modules can be swapped out for a similarly compatible module, such as switching LevelDb for [MemDB](https://github.com/juliangruber/memdb) (which we do in the first example). More details on how these module work together are available in [How Dat Works](how-dat-works.md).
+The [Dat CLI](https://npmjs.org/dat) module itself combines these modules and wraps them in a command-line API. We also use the [dat-storage](https://github.com/datproject/dat-storage) module to handle file and key storage. These modules can be swapped out for a similarly compatible module, such as switching storage for [random-access-memory](https://github.com/mafintosh/random-access-memory).
 
 ## Getting Started
 
 You will need node and npm installed to build with Dat. [Read more](https://github.com/datproject/dat/blob/master/CONTRIBUTING.md#development-workflow) about our development work flow to learn how we manage our module dependencies during development.
 
 ## Download a File
 
-Our first module will download files from a Dat link entered by the user. View the code for this module on [Github](https://github.com/joehand/diy-dat-examples/tree/master/module-1).
+Our first module will download files from a Dat link entered by the user.
 
 ```bash
 mkdir module-1 && cd module-1
 npm init
-npm install --save hyperdrive memdb hyperdrive-archive-swarm
+npm install --save hyperdrive random-access-memory hyperdiscovery
 touch index.js
 ```
 
-For this example, we will use [memdb](https://github.com/juliangruber/memdb) for our database (keeping the metadata in memory rather than on the file system). In your `index.js` file, require the main modules and set them up:
+For this example, we will use random-access-memory for our database (keeping the metadata in memory rather than on the file system). In your `index.js` file, require the main modules and set them up:
 
 ```js
-var memdb = require('memdb')
-var Hyperdrive = require('hyperdrive')
-var Swarm = require('hyperdrive-archive-swarm')
+var ram = require('random-access-memory')
+var hyperdrive = require('hyperdrive')
+var discovery = require('hyperdiscovery')
 
 var link = process.argv[2] // user inputs the dat link
 
-var db = memdb()
-var drive = Hyperdrive(db)
-var archive = drive.createArchive(link)
-var swarm = Swarm(archive)
+var archive = hyperdrive(ram, link)
+archive.ready(function () {
+  discovery(archive)
+})
 ```
 
-Notice, the user will input the link for the second argument The easiest way to get a file from a hyperdrive archive is to make a read stream. `archive.createFileReadStream` accepts the index number of filename for the first argument. To display the file, we can create a file stream and pipe it to `process.stdout`.
+Notice, the user will input the link for the second argument The easiest way to get a file from a hyperdrive archive is to make a read stream. `archive.readFile` accepts the index number of filename for the first argument. To display the file, we can create a file stream and pipe it to `process.stdout`.
 
 ```js
-var stream = archive.createFileReadStream(0) // get the first file
-stream.pipe(process.stdout)
+// Make sure your archive has a dat.json file!
+var stream = archive.readFile('dat.json', 'utf-8', function (err, data) {
+  if (err) throw err
+  console.log(data)
+})
 ```
 
-Now, you can run the module! To download the first file from our docs Dat, run:
+Now, you can run the module! To download the `dat.json` file from an archive:
 
 ```
-node index.js 395e3467bb5b2fa083ee8a4a17a706c5574b740b5e1be6efd65754d4ab7328c2
+node index.js dat://<link>
 ```
 
-You should see the first file in our docs repo.
+You should see the `dat.json` file.
 
 #### Bonus: Display any file in the Dat
 
 With a few more lines of code, the user can enter a file to display from the Dat link.
 
-Challenge: create a module that will allow the user to input a Dat link and a filename: `node bonus.js <dat-link> <filename>`. The module will print out that file from the link, as we did above. To get a specific file you can change the file stream to use the filename instead of the index number:
+Challenge: create a module that will allow the user to input a Dat link and a filename: `node bonus.js <dat-link> <filename>`. The module will print out that file from the link, as we did above:
 
 ```js
-var stream = archive.createFileReadStream(fileName)
+var stream = archive.readFile(fileName)
 ```
 
 Once you are finished, see if you can view this file by running:
 
 ```bash
-node bonus.js 395e3467bb5b2fa083ee8a4a17a706c5574b740b5e1be6efd65754d4ab7328c2 cookbook/diy-dat.md
+node bonus.js 395e3467bb5b2fa083ee8a4a17a706c5574b740b5e1be6efd65754d4ab7328c2 readme.md
 ```
 
-[See how we coded it](https://github.com/joehand/diy-dat-examples/blob/master/module-1/bonus.js).
-
 ## Download all files to computer
 
-This module will build on the last module. Instead of displaying a single file, we will download all of the files from a Dat into a local directory. View the code for this module on [Github](https://github.com/joehand/diy-dat-examples/tree/master/module-2).
+This module will build on the last module. Instead of displaying a single file, we will download all of the files from a Dat into a local directory.
 
-To download the files to the file system, instead of to a database, we will use the `file` option in `hyperdrive` and the [random-access-file](http://npmjs.org/random-access-file) module. We will also learn two new archive functions that make handling all the files a bit easier than the file stream in module #1.
+To download the files to the file system, we are going to use [mirror-folder](https://github.com/mafintosh/mirror-folder). [Read more](/using-fs) about how mirror-folder works with hyperdrive.
 
-Setup will be the same as before (make sure you install random-access-file and stream-each this time):
+In practice, you should use [dat-storage](https://github.com/datproject/dat-storage) to do this as it'll be more efficient and keep the metadata on disk.
 
-```bash
-mkdir module-2 && cd module-2
-npm init
-npm install --save hyperdrive memdb hyperdrive-archive-swarm random-access-file stream-each
-touch index.js
-```
-
-The first part of the module will look the same. We will add random-access-file (and [stream-each](http://npmjs.org/stream-each) to make things easier). The only difference is that we have to specify the `file` option when creating our archive:
+Setup will be the same as before (make sure you install `mirror-folder`). The first part of the module will look the same.
 
 ```js
-var memdb = require('memdb')
-var Hyperdrive = require('hyperdrive')
-var Swarm = require('hyperdrive-archive-swarm')
-var raf = require('random-access-file') // this is new!
-var each = require('stream-each')
-
-var link = process.argv[2]
-
-var db = memdb()
-var drive = Hyperdrive(db)
-var archive = drive.createArchive(link, {
-  file: function (name) {
-    return raf(path.join('download', name)) // download into a "download" dir
-  }
-})
-var swarm = Swarm(archive)
-```
+var ram = require('random-access-memory')
+var hyperdrive = require('hyperdrive')
+var discovery = require('hyperdiscovery')
+var mirror = require('mirror-folder')
 
-Now that we are setup, we can work with the archive. The `archive.download` function downloads the file content (to wherever you specified in the file option). To download all the files, we will need a list of files and then we will call download on each of them. `archive.list` will give us the list of the files. We use the stream-each module to make it easy to iterate over each item in the archive, then exit when the stream is finished.
+var link = process.argv[2] // user inputs the dat link
+var dir = process.cwd() // download to cwd
 
-```js
-var stream = archive.list({live: false}) // Use {live: false} for now to make the stream easier to handle.
-each(stream, function (entry, next) {
-  archive.download(entry, function (err) {
-    if (err) return console.error(err)
-    console.log('downloaded', entry.name)
-    next()
+var archive = hyperdrive(ram, link)
+archive.ready(function () {
+  discovery(archive)
+
+  var progress = mirror({name: '/', fs: archive}, dir, function (err) {
+    console.log('done downloading!')
+  })
+  progress.on('put', function (src) {
+    console.log(src.name, 'downloaded')
   })
-}, function () {
-  process.exit(0)
 })
 ```
 
 You should be able to run the module and see all our docs files in the `download` folder:
 
 ```bash
-node index.js 395e3467bb5b2fa083ee8a4a17a706c5574b740b5e1be6efd65754d4ab7328c2
+node index.js dat://<link>
 ```

assets/install_cli.png

@@ -0,0 +1,61 @@
+# Sharing files over HTTP
+
+The Dat command line comes with a built in HTTP server. This is a cool demo because we can also see how version history works! The `--http` option works for files you are sharing *or* downloading.
+
+(If you don't have dat command line installed, run `npm install -g dat`, or [see more info](intro#installation).)
+
+## Serve over HTTP
+
+Serve dat files on http by adding the `--http` option. For example, you can sync an existing dat:
+
+```
+❯ dat sync --http
+dat://778f8d955175c92e4ced5e4f5563f69bfec0c86cc6f670352c457943666fe639
+Sharing dat: 2 files (1.4 MB)
+Serving files over http at http://localhost:8080
+
+2 connections | Download 0 B/s Upload 0 B/s
+```
+
+Now visit [http://localhost:8080]() to see the files in your browser! The default http port is 8080. You should see a directory listing:
+
+<img src="/assets/cli-http.png" alt="Dat HTTP viewer"/>
+
+If your dat has an `index.html` page, that will be shown instead.
+
+You can combine Dat's http support with our server tools to create a live updating website or place to share files publicly.
+
+## Built-in Versioning
+
+As you may know, Dat automatically versions all files. The HTTP display is an easy way to view version history:
+
+**Use [localhost:8080/?version=2]() to view a specific version.**
+
+## Live reloading
+
+The Dat http viewer also comes with live reloading. If it detects a new version it will automatically reload with the new directory listing or page (as long as you aren't viewing a specific version in the url).
+
+## Sparse Downloading
+
+Dat supports *sparse*, or partial downloads, of datasets. This is really useful if you only want a single file from a large dat. Unfortunately, we haven't quite built a user interface for this into our applications. So you can hack around it!
+
+This will allow you to download a single file from a larger dat, without downloading metadata or any other files.
+
+First, start downloading our demo dat, make sure you include both the flags (`--http`, `--sparse`).
+
+```
+❯ dat dat://778f8d955175c92e4ced5e4f5563f69bfec0c86cc6f670352c457943666fe639 ./demo --http --sparse
+Cloning: 2 files (1.4 MB)
+Serving files over http at http://localhost:8080
+
+3 connections | Download 0 B/s Upload 0 B/s
+```
+
+The `--sparse` option tells Dat to only download files you specifically request. See how it works:
+
+1. Check out your `./demo` folder, it should be empty.
+2. [Open the Dat](http://localhost:8080) in your browser.
+3. Click on a file to download.
+4. It should be in your folder now!
+
+Pretty cool! You can use this hack to download only specific files or even older versions of files (if they've been saved somewhere).

assets/install_desktop.png

@@ -0,0 +1,50 @@
+# Getting Started with Dat 
+
+In this tutorial we will go through the two main ways to use Dat, sharing data and downloading data. If possible, this is great to go through with a partner to see how Dat works across computers. Get Dat [installed](intro#installation) and get started!
+
+Dat Desktop makes it easy for anyone to get started using Dat with user-friendly interface. If you are comfortable with the command line then you can install dat via npm. You can always switch apps later and keep your dats the same. Dat can share your files to anyone, it does not matter how they are using Dat.
+
+## Command Line Tutorial
+
+### Downloading Data
+
+We made a demo folder we made just for this exercise. Inside the demo folder is a `dat.json` file and a gif. We shared these files via Dat and now you can download them with our dat key!
+
+Similar to git, you do download somebody's dat by running `dat clone <link>`. You can also specify the directory:
+
+```
+❯ dat clone dat://778f8d955175c92e4ced5e4f5563f69bfec0c86cc6f670352c457943666fe639 ~/Downloads/dat-demo
+dat v13.5.0
+Created new dat in /Users/joe/Downloads/dat-demo/.dat
+Cloning: 2 files (1.4 MB)
+
+2 connections | Download 614 KB/s Upload 0 B/s
+
+dat sync complete.
+Version 4
+```
+
+This will download our demo files to the `~/downloads/dat-demo` folder. These files are being shared by a server over Dat (to ensure high availability) but you may connect to any number of users also hosting the content.
+
+You can also also view the files online: [datproject.org/778f8d955175c92e4ced5e4f5563f69bfec0c86cc6f670352c457943666fe639](https://datproject.org/778f8d955175c92e4ced5e4f5563f69bfec0c86cc6f670352c457943666fe639/). datproject.org can download files over Dat and display them on http as long as someone is hosting it. The website temporarily caches data for any visited links (do not view your dat on datproject.org if you do not want us caching your data).
+
+### Sharing Data
+
+We'll be creating a dat from a folder on your computer. If you are with a friend you can sync these files to their computer. Otherwise you can view them online via datproject.org to see how viewing a dat online works. 
+
+Find a folder on your computer to share. Any kind of files work with Dat but for now, make sure it's something you want to share with your friends. Dat can handle all sorts of files (Dat works with really big folders too!). We like cat pictures.
+
+First, you can create a new dat inside that folder. Using the `dat create` command also walks us through making a `dat.json` file:
+
+```
+❯ dat create
+Welcome to dat program!
+You can turn any folder on your computer into a Dat.
+A Dat is a folder with some magic.
+```
+
+This will create a new (empty) dat. Dat will print a link, share this link to give others access to view your files.
+
+Once we have our dat, run `dat share` to scan your files and sync them to the network. Share the link with your friend to instantly start downloading files.
+
+You can also try viewing your files online. Go to [datproject.org](https://datproject.org/explore) and enter your link to preview on the top right. *(Some users, including me when writing this, may have trouble connecting to datproject.org initially. Don't be alarmed! It is something we are working on. Thanks.)*

assets/share_link.png

@@ -0,0 +1,58 @@
+# Using the Hyperdrive FS
+
+[Hyperdrive](https://github.com/mafintosh/hyperdrive), the core file system module in dat, exposes an API that mimics the Node fs API. This allows you to create modules that act on hyperdrive or a regular fs with the same API. We have several modules that we make use of this custom fs, such as [mirror-folder](https://github.com/mafintosh/mirror-folder). 
+
+Mirror folder can copy a regular directory to another regular directory:
+
+```js
+var mirror = require('mirror-folder')
+
+mirror('/source', '/dest', function (err) {
+  console.log('mirror complete')
+})
+```
+
+You can also copy a folder to an archive (this is how we do importing in Dat):
+
+```js
+var archive = hyperdrive('/dir')
+mirror('/source', {name: '/', fs: archive}, function (err) {
+  console.log('mirror complete')
+})
+```
+
+### Creating Custom FS Modules
+
+To create a module that uses a custom fs, you can default to the regular `fs` but also accept `fs` as an argument. For example, to print a file you could write this function:
+
+```js
+function printFile(file, fs) {
+  if (!fs) fs = require('fs')
+
+  fs.readFile(file, 'utf-8', function (err, data) {
+    console.log(data)
+  }) 
+}
+```
+
+Then you could use this to print a file from a regular fs:
+
+```js
+printFile('/data/hello-world.txt')
+```
+
+Or from a hyperdrive archive:
+
+```js
+var archive = hyperdrive('/data')
+printFile('/hello-world.txt', archive) // pass archive as the fs!
+```
+
+## Modules!
+
+See more examples of custom-fs modules:
+
+* [mirror-folder](https://github.com/mafintosh/mirror-folder) - copy files from one fs to another fs (regular or custom)
+* [count-files](https://github.com/joehand/count-files) - count files in regular or custom fs.
+* [ftpfs](https://github.com/maxogden/ftpfs) - custom fs for FTPs
+* [bagit-fs](https://github.com/joehand/bagit-fs) - custom fs module for the BagIt spec.