http: fix, tidy and rework ready for release

* Fix remaining problems
  * Refactor to make testing easier and add a test suite
  * Make path parsing more robust.
  * Add single file operations
  * Add MimeType reading for objects
  * Add documentation
  * Note go1.7+ is required to build

README.md

@@ -27,6 +27,7 @@ Rclone is a command line program to sync files and directories to and from
   * Yandex Disk
   * SFTP
   * FTP
+  * HTTP
   * The local filesystem
 
 Features

bin/make_manual.py

@@ -30,8 +30,9 @@ docs = [
     "b2.md",
     "yandex.md",
     "sftp.md",
-    "crypt.md",
     "ftp.md",
+    "http.md",
+    "crypt.md",
     "local.md",
     "changelog.md",
     "bugs.md",

cmd/cmd.go

@@ -53,6 +53,7 @@ from various cloud storage systems and using file transfer services, such as:
   * Yandex Disk
   * SFTP
   * FTP
+  * HTTP
   * The local filesystem
 
 Features

docs/content/about.md

@@ -25,6 +25,7 @@ Rclone is a command line program to sync files and directories to and from
   * Yandex Disk
   * SFTP
   * FTP
+  * HTTP
   * The local filesystem
 
 Features

docs/content/docs.md

@@ -32,6 +32,7 @@ See the following for detailed instructions for
   * [Yandex Disk](/yandex/)
   * [SFTP](/sftp/)
   * [FTP](/ftp/)
+  * [HTTP](/http/)
   * [Crypt](/crypt/) - to encrypt other remotes
 
 Usage

docs/content/http.md

@@ -0,0 +1,137 @@
+---
+title: "HTTP Remote"
+description: "Read only remote for HTTP servers"
+date: "2017-06-19"
+---
+
+<i class="fa fa-globe"></i> HTTP
+-------------------------------------------------
+
+The HTTP remote is a read only remote for reading files of a
+webserver.  The webserver should provide file listings which rclone
+will read and turn into a remote.  This has been tested with common
+webservers such as Apache/Nginx/Caddy and will likely work with file
+listings from most web servers.  (If it doesn't then please file an
+issue, or send a pull request!)
+
+Paths are specified as `remote:` or `remote:path/to/dir`.
+
+Here is an example of how to make a remote called `remote`.  First
+run:
+
+     rclone config
+
+This will guide you through an interactive setup process:
+
+```
+No remotes found - make a new one
+n) New remote
+s) Set configuration password
+q) Quit config
+n/s/q> n
+name> remote
+Type of storage to configure.
+Choose a number from below, or type in your own value
+ 1 / Amazon Drive
+   \ "amazon cloud drive"
+ 2 / Amazon S3 (also Dreamhost, Ceph, Minio)
+   \ "s3"
+ 3 / Backblaze B2
+   \ "b2"
+ 4 / Dropbox
+   \ "dropbox"
+ 5 / Encrypt/Decrypt a remote
+   \ "crypt"
+ 6 / FTP Connection
+   \ "ftp"
+ 7 / Google Cloud Storage (this is not Google Drive)
+   \ "google cloud storage"
+ 8 / Google Drive
+   \ "drive"
+ 9 / Hubic
+   \ "hubic"
+10 / Local Disk
+   \ "local"
+11 / Microsoft OneDrive
+   \ "onedrive"
+12 / Openstack Swift (Rackspace Cloud Files, Memset Memstore, OVH)
+   \ "swift"
+13 / SSH/SFTP Connection
+   \ "sftp"
+14 / Yandex Disk
+   \ "yandex"
+15 / http Connection
+   \ "http"
+Storage> http
+URL of http host to connect to
+Choose a number from below, or type in your own value
+ 1 / Connect to example.com
+   \ "https://example.com"
+url> https://beta.rclone.org
+Remote config
+--------------------
+[remote]
+url = https://beta.rclone.org
+--------------------
+y) Yes this is OK
+e) Edit this remote
+d) Delete this remote
+y/e/d> y
+Current remotes:
+
+Name                 Type
+====                 ====
+remote               http
+
+e) Edit existing remote
+n) New remote
+d) Delete remote
+r) Rename remote
+c) Copy remote
+s) Set configuration password
+q) Quit config
+e/n/d/r/c/s/q> q
+```
+
+This remote is called `remote` and can now be used like this
+
+See all the top level directories
+
+    rclone lsd remote:
+
+List the contents of a directory
+
+    rclone ls remote:directory
+
+Sync the remote `directory` to `/home/local/directory`, deleting any excess files.
+
+    rclone sync remote:directory /home/local/directory
+
+### Read only ###
+
+This remote is read only - you can't upload files to an HTTP server.
+
+### Modified time ###
+
+Most HTTP servers store time accurate to 1 second.
+
+### Checksum ###
+
+No checksums are stored.
+
+### Usage without a config file ###
+
+Note that since only two environment variable need to be set, it is
+easy to use without a config file like this.
+
+```
+RCLONE_CONFIG_ZZ_TYPE=http RCLONE_CONFIG_ZZ_URL=https://beta.rclone.org rclone lsd zz:
+```
+
+Or if you prefer
+
+```
+export RCLONE_CONFIG_ZZ_TYPE=http
+export RCLONE_CONFIG_ZZ_URL=https://beta.rclone.org
+rclone lsd zz:
+```

docs/content/overview.md

@@ -29,6 +29,7 @@ Here is an overview of the major features of each cloud storage system.
 | Yandex Disk            | MD5     | Yes     | No               | No              | R/W       |
 | SFTP                   | -       | Yes     | Depends          | No              | -         |
 | FTP                    | -       | No      | Yes              | No              | -         |
+| HTTP                   | -       | No      | Yes              | No              | R         |
 | The local filesystem   | All     | Yes     | Depends          | No              | -         |
 
 ### Hash ###
@@ -122,6 +123,7 @@ operations more efficient.
 | Yandex Disk            | Yes   | No   | No   | No      | No  [#575](https://github.com/ncw/rclone/issues/575) | Yes   |
 | SFTP                   | No    | No   | Yes  | Yes     | No      | No    |
 | FTP                    | No    | No   | Yes  | Yes     | No      | No    |
+| HTTP                   | No    | No   | No   | No      | No      | No    |
 | The local filesystem   | Yes   | No   | Yes  | Yes     | No      | No    |
 
 

docs/layouts/chrome/navbar.html

@@ -62,6 +62,7 @@
                     <li><a href="/yandex/"><i class="fa fa-space-shuttle"></i> Yandex Disk</a></li>
                     <li><a href="/sftp/"><i class="fa fa-server"></i> SFTP</a></li>
                     <li><a href="/ftp/"><i class="fa fa-file"></i> FTP</a></li>
+                    <li><a href="/http/"><i class="fa fa-globe"></i> HTTP</a></li>
                     <li><a href="/crypt/"><i class="fa fa-lock"></i> Crypt (encrypts the above)</a></li>
                   </ul>
                 </li>

http/http.go

@@ -1,18 +1,16 @@
 // Package http provides a filesystem interface using golang.org/net/http
 //
-// It treads HTML pages served from the endpoint as directory
+// It treats HTML pages served from the endpoint as directory
 // listings, and includes any links found as files.
 
-// +build !plan9
+// +build go1.7
 
 package http
 
 import (
-	"fmt"
 	"io"
 	"net/http"
 	"net/url"
-	"os"
 	"path"
 	"strconv"
 	"strings"
@@ -23,7 +21,10 @@ import (
 	"golang.org/x/net/html"
 )
 
-var errorReadOnly = errors.New("http remotes are read only")
+var (
+	errorReadOnly = errors.New("http remotes are read only")
+	timeUnset     = time.Unix(0, 0)
+)
 
 func init() {
 	fsi := &fs.RegInfo{
@@ -31,7 +32,7 @@ func init() {
 		Description: "http Connection",
 		NewFs:       NewFs,
 		Options: []fs.Option{{
-			Name:     "endpoint",
+			Name:     "url",
 			Help:     "URL of http host to connect to",
 			Optional: false,
 			Examples: []fs.OptionExample{{
@@ -54,49 +55,86 @@ type Fs struct {
 
 // Object is a remote object that has been stat'd (so it exists, but is not necessarily open for reading)
 type Object struct {
-	fs     *Fs
+	fs          *Fs
-	remote string
+	remote      string
-	info   os.FileInfo
+	size        int64
+	modTime     time.Time
+	contentType string
 }
 
-// ObjectReader holds the File interface to a remote http file opened for reading
+// Join a URL and a path returning a new URL
-type ObjectReader struct {
+func urlJoin(base *url.URL, path string) *url.URL {
-	object   *Object
+	rel, err := url.Parse(path)
-	httpFile io.ReadCloser
+	if err != nil {
-}
+		fs.Errorf(nil, "Error parsing %q as URL: %v", path, err)
-
-func urlJoin(u *url.URL, paths ...string) string {
-	r := u
-	for _, p := range paths {
-		if p == "/" {
-			continue
-		}
-		rel, _ := url.Parse(p)
-		r = r.ResolveReference(rel)
 	}
-	return r.String()
+	return base.ResolveReference(rel)
+}
+
+// statusError returns an error if the res contained an error
+func statusError(res *http.Response, err error) error {
+	if err != nil {
+		return err
+	}
+	if res.StatusCode < 200 || res.StatusCode > 299 {
+		_ = res.Body.Close()
+		return errors.Errorf("HTTP Error %d: %s", res.StatusCode, res.Status)
+	}
+	return nil
 }
 
 // NewFs creates a new Fs object from the name and root. It connects to
 // the host specified in the config file.
 func NewFs(name, root string) (fs.Fs, error) {
-	endpoint := fs.ConfigFileGet(name, "endpoint")
+	endpoint := fs.ConfigFileGet(name, "url")
+	if !strings.HasSuffix(endpoint, "/") {
+		endpoint += "/"
+	}
 
-	u, err := url.Parse(endpoint)
+	// Parse the endpoint and stick the root onto it
+	base, err := url.Parse(endpoint)
+	if err != nil {
+		return nil, err
+	}
+	rootURL, err := url.Parse(root)
+	if err != nil {
+		return nil, err
+	}
+	u := base.ResolveReference(rootURL)
+
+	client := fs.Config.Client()
+
+	var isFile = false
+	if !strings.HasSuffix(u.String(), "/") {
+		// Make a client which doesn't follow redirects so the server
+		// doesn't redirect http://host/dir to http://host/dir/
+		noRedir := *client
+		noRedir.CheckRedirect = func(req *http.Request, via []*http.Request) error {
+			return http.ErrUseLastResponse
+		}
+		// check to see if points to a file
+		res, err := noRedir.Head(u.String())
+		err = statusError(res, err)
+		if err == nil {
+			isFile = true
+		}
+	}
+
+	newRoot := u.String()
+	if isFile {
+		// Point to the parent if this is a file
+		newRoot, _ = path.Split(u.String())
+	} else {
+		if !strings.HasSuffix(newRoot, "/") {
+			newRoot += "/"
+		}
+	}
+
+	u, err = url.Parse(newRoot)
 	if err != nil {
 		return nil, err
 	}
 
-	if !strings.HasSuffix(root, "/") && root != "" {
-		root += "/"
-	}
-
-	client := fs.Config.Client()
-
-	_, err = client.Head(urlJoin(u, root))
-	if err != nil {
-		return nil, errors.Wrap(err, "couldn't connect http")
-	}
 	f := &Fs{
 		name:       name,
 		root:       root,
@@ -104,6 +142,9 @@ func NewFs(name, root string) (fs.Fs, error) {
 		endpoint:   u,
 	}
 	f.features = (&fs.Features{}).Fill(f)
+	if isFile {
+		return f, fs.ErrorIsFile
+	}
 	return f, nil
 }
 
@@ -119,7 +160,7 @@ func (f *Fs) Root() string {
 
 // String returns the URL for the filesystem
 func (f *Fs) String() string {
-	return urlJoin(f.endpoint, f.root)
+	return f.endpoint.String()
 }
 
 // Features returns the optional features of this Fs
@@ -145,51 +186,6 @@ func (f *Fs) NewObject(remote string) (fs.Object, error) {
 	return o, nil
 }
 
-// dirExists returns true,nil if the directory exists, false, nil if
-// it doesn't or false, err
-func (f *Fs) dirExists(dir string) (bool, error) {
-	res, err := f.httpClient.Head(urlJoin(f.endpoint, dir))
-	if err != nil {
-		return false, err
-	}
-	if res.StatusCode == http.StatusOK {
-		return true, nil
-	}
-	return false, nil
-}
-
-type entry struct {
-	name  string
-	url   string
-	size  int64
-	mode  os.FileMode
-	mtime int64
-}
-
-func (e *entry) Name() string {
-	return e.name
-}
-
-func (e *entry) Size() int64 {
-	return e.size
-}
-
-func (e *entry) Mode() os.FileMode {
-	return os.FileMode(e.mode)
-}
-
-func (e *entry) ModTime() time.Time {
-	return time.Unix(e.mtime, 0)
-}
-
-func (e *entry) IsDir() bool {
-	return e.mode&os.ModeDir != 0
-}
-
-func (e *entry) Sys() interface{} {
-	return nil
-}
-
 func parseInt64(s string) int64 {
 	n, e := strconv.ParseInt(s, 10, 64)
 	if e != nil {
@@ -198,84 +194,95 @@ func parseInt64(s string) int64 {
 	return n
 }
 
-func parseBool(s string) bool {
+// parseName turns a name as found in the page into a remote path or returns false
-	b, e := strconv.ParseBool(s)
+func parseName(base *url.URL, val string) (string, bool) {
-	if e != nil {
+	name, err := url.QueryUnescape(val)
-		return false
+	if err != nil {
+		return "", false
 	}
-	return b
+	u := urlJoin(base, name)
-}
+	uStr := u.String()
-
+	if strings.Index(uStr, "?") >= 0 {
-func prepareTimeString(ts string) string {
+		return "", false
-	return strings.Trim(strings.Join(strings.SplitN(strings.Trim(ts, "\t "), " ", 3)[0:2], " "), "\r\n\t ")
-}
-
-func parseTime(n *html.Node) (t time.Time) {
-	if ts := prepareTimeString(n.Data); ts != "" {
-		t, _ = time.Parse("2-Jan-2006 15:04", ts)
 	}
-	return t
+	baseStr := base.String()
+	// check has URL prefix
+	if !strings.HasPrefix(uStr, baseStr) {
+		return "", false
+	}
+	// check has path prefix
+	if !strings.HasPrefix(u.Path, base.Path) {
+		return "", false
+	}
+	// calculate the name relative to the base
+	name = u.Path[len(base.Path):]
+	// musn't be empty
+	if name == "" {
+		return "", false
+	}
+	// mustn't contain a /
+	slash := strings.Index(name, "/")
+	if slash >= 0 && slash != len(name)-1 {
+		return "", false
+	}
+	return name, true
 }
 
-func (f *Fs) readDir(p string) ([]*entry, error) {
+// Parse turns HTML for a directory into names
-	entries := make([]*entry, 0)
+// base should be the base URL to resolve any relative names from
-	res, err := f.httpClient.Get(urlJoin(f.endpoint, p))
+func parse(base *url.URL, in io.Reader) (names []string, err error) {
+	doc, err := html.Parse(in)
 	if err != nil {
 		return nil, err
 	}
-	if res.Body == nil || res.StatusCode != http.StatusOK {
+	var walk func(*html.Node)
-		//return nil, errors.Errorf("directory listing failed with error: % (%d)", res.Status, res.StatusCode)
+	walk = func(n *html.Node) {
-		return nil, nil
+		if n.Type == html.ElementNode && n.Data == "a" {
+			for _, a := range n.Attr {
+				if a.Key == "href" {
+					name, ok := parseName(base, a.Val)
+					if ok {
+						names = append(names, name)
+					}
+					break
+				}
+			}
+		}
+		for c := n.FirstChild; c != nil; c = c.NextSibling {
+			walk(c)
+		}
+	}
+	walk(doc)
+	return names, nil
+}
+
+// Read the directory passed in
+func (f *Fs) readDir(dir string) (names []string, err error) {
+	u := urlJoin(f.endpoint, dir)
+	if !strings.HasSuffix(u.String(), "/") {
+		return nil, errors.Errorf("internal error: readDir URL %q didn't end in /", u.String())
+	}
+	res, err := f.httpClient.Get(u.String())
+	if err == nil && res.StatusCode == http.StatusNotFound {
+		return nil, fs.ErrorDirNotFound
+	}
+	err = statusError(res, err)
+	if err != nil {
+		return nil, errors.Wrap(err, "failed to readDir")
 	}
 	defer fs.CheckClose(res.Body, &err)
 
-	switch strings.SplitN(res.Header.Get("Content-Type"), ";", 2)[0] {
+	contentType := strings.SplitN(res.Header.Get("Content-Type"), ";", 2)[0]
+	switch contentType {
 	case "text/html":
-		doc, err := html.Parse(res.Body)
+		names, err = parse(u, res.Body)
 		if err != nil {
-			return nil, err
+			return nil, errors.Wrap(err, "readDir")
 		}
-		var walk func(*html.Node)
+	default:
-		walk = func(n *html.Node) {
+		return nil, errors.Errorf("Can't parse content type %q", contentType)
-			if n.Type == html.ElementNode && n.Data == "a" {
-				for _, a := range n.Attr {
-					if a.Key == "href" {
-						name, err := url.QueryUnescape(a.Val)
-						if err != nil {
-							continue
-						}
-						if name == "../" || name == "./" || name == ".." {
-							break
-						}
-						if strings.Index(name, "?") >= 0 || strings.HasPrefix(name, "http") {
-							break
-						}
-						u, err := url.Parse(name)
-						if err != nil {
-							break
-						}
-						name = path.Clean(u.Path)
-						e := &entry{
-							name: strings.TrimRight(name, "/"),
-							url:  name,
-						}
-						if a.Val[len(a.Val)-1] == '/' {
-							e.mode = os.FileMode(0555) | os.ModeDir
-						} else {
-							e.mode = os.FileMode(0444)
-						}
-						entries = append(entries, e)
-						break
-					}
-				}
-			}
-			for c := n.FirstChild; c != nil; c = c.NextSibling {
-				walk(c)
-			}
-		}
-		walk(doc)
 	}
-	return entries, nil
+	return names, nil
 }
 
 // List the objects and directories in dir into entries.  The
@@ -288,36 +295,21 @@ func (f *Fs) readDir(p string) ([]*entry, error) {
 // This should return ErrDirNotFound if the directory isn't
 // found.
 func (f *Fs) List(dir string) (entries fs.DirEntries, err error) {
-	endpoint := path.Join(f.root, dir)
+	if !strings.HasSuffix(dir, "/") && dir != "" {
-	if !strings.HasSuffix(dir, "/") {
+		dir += "/"
-		endpoint += "/"
 	}
-	ok, err := f.dirExists(endpoint)
+	names, err := f.readDir(dir)
-	if err != nil {
-		return nil, errors.Wrap(err, "List failed")
-	}
-	if !ok {
-		return nil, fs.ErrorDirNotFound
-	}
-	httpDir := path.Join(f.root, dir)
-	if !strings.HasSuffix(dir, "/") {
-		httpDir += "/"
-	}
-	infos, err := f.readDir(httpDir)
 	if err != nil {
 		return nil, errors.Wrapf(err, "error listing %q", dir)
 	}
-	for _, info := range infos {
+	for _, name := range names {
-		remote := ""
+		isDir := name[len(name)-1] == '/'
-		if dir != "" {
+		name = strings.TrimRight(name, "/")
-			remote = dir + "/" + info.Name()
+		remote := path.Join(dir, name)
-		} else {
+		if isDir {
-			remote = info.Name()
-		}
-		if info.IsDir() {
 			dir := &fs.Dir{
 				Name:  remote,
-				When:  info.ModTime(),
+				When:  timeUnset,
 				Bytes: 0,
 				Count: 0,
 			}
@@ -326,7 +318,6 @@ func (f *Fs) List(dir string) (entries fs.DirEntries, err error) {
 			file := &Object{
 				fs:     f,
 				remote: remote,
-				info:   info,
 			}
 			if err = file.stat(); err != nil {
 				continue
@@ -371,12 +362,12 @@ func (o *Object) Hash(r fs.HashType) (string, error) {
 
 // Size returns the size in bytes of the remote http file
 func (o *Object) Size() int64 {
-	return o.info.Size()
+	return o.size
 }
 
 // ModTime returns the modification time of the remote http file
 func (o *Object) ModTime() time.Time {
-	return o.info.ModTime()
+	return o.modTime
 }
 
 // path returns the native path of the object
@@ -386,37 +377,19 @@ func (o *Object) path() string {
 
 // stat updates the info field in the Object
 func (o *Object) stat() error {
-	endpoint := urlJoin(o.fs.endpoint, o.fs.root, o.remote)
+	endpoint := urlJoin(o.fs.endpoint, o.remote).String()
-	if o.info.IsDir() {
-		endpoint += "/"
-	}
 	res, err := o.fs.httpClient.Head(endpoint)
+	err = statusError(res, err)
 	if err != nil {
-		return err
+		return errors.Wrap(err, "failed to stat")
 	}
-	if res.StatusCode != http.StatusOK {
-		return errors.New("failed to stat")
-	}
-	var mtime int64
 	t, err := http.ParseTime(res.Header.Get("Last-Modified"))
 	if err != nil {
-		mtime = 0
+		t = timeUnset
-	} else {
-		mtime = t.Unix()
 	}
-	size := parseInt64(res.Header.Get("Content-Length"))
+	o.size = parseInt64(res.Header.Get("Content-Length"))
-	e := &entry{
+	o.modTime = t
-		name:  o.remote,
+	o.contentType = res.Header.Get("Content-Type")
-		size:  size,
-		mtime: mtime,
-		mode:  os.FileMode(0444),
-	}
-	if strings.HasSuffix(o.remote, "/") {
-		e.mode = os.FileMode(0555) | os.ModeDir
-		e.size = 0
-		e.name = o.remote[:len(o.remote)-1]
-	}
-	o.info = e
 	return nil
 }
 
@@ -429,52 +402,29 @@ func (o *Object) SetModTime(modTime time.Time) error {
 
 // Storable returns whether the remote http file is a regular file (not a directory, symbolic link, block device, character device, named pipe, etc)
 func (o *Object) Storable() bool {
-	return o.info.Mode().IsRegular()
+	return true
-}
-
-// Read from a remote http file object reader
-func (file *ObjectReader) Read(p []byte) (n int, err error) {
-	n, err = file.httpFile.Read(p)
-	return n, err
-}
-
-// Close a reader of a remote http file
-func (file *ObjectReader) Close() (err error) {
-	return file.httpFile.Close()
 }
 
 // Open a remote http file object for reading. Seek is supported
 func (o *Object) Open(options ...fs.OpenOption) (in io.ReadCloser, err error) {
-	var offset int64
+	endpoint := urlJoin(o.fs.endpoint, o.remote).String()
-	endpoint := urlJoin(o.fs.endpoint, o.fs.root, o.remote)
-	offset = 0
-	for _, option := range options {
-		switch x := option.(type) {
-		case *fs.SeekOption:
-			offset = x.Offset
-		default:
-			if option.Mandatory() {
-				fs.Logf(o, "Unsupported mandatory option: %v", option)
-			}
-		}
-	}
-
 	req, err := http.NewRequest("GET", endpoint, nil)
 	if err != nil {
 		return nil, errors.Wrap(err, "Open failed")
 	}
-	if offset > 0 {
+
-		req.Header.Set("Range", fmt.Sprintf("bytes=%d-", offset))
+	// Add optional headers
+	for k, v := range fs.OpenOptionHeaders(options) {
+		req.Header.Add(k, v)
 	}
+
+	// Do the request
 	res, err := o.fs.httpClient.Do(req)
+	err = statusError(res, err)
 	if err != nil {
 		return nil, errors.Wrap(err, "Open failed")
 	}
-	in = &ObjectReader{
+	return res.Body, nil
-		object:   o,
-		httpFile: res.Body,
-	}
-	return in, nil
 }
 
 // Hashes returns fs.HashNone to indicate remote hashing is unavailable
@@ -502,8 +452,14 @@ func (o *Object) Update(in io.Reader, src fs.ObjectInfo, options ...fs.OpenOptio
 	return errorReadOnly
 }
 
+// MimeType of an Object if known, "" otherwise
+func (o *Object) MimeType() string {
+	return o.contentType
+}
+
 // Check the interfaces are satisfied
 var (
-	_ fs.Fs     = &Fs{}
+	_ fs.Fs        = &Fs{}
-	_ fs.Object = &Object{}
+	_ fs.Object    = &Object{}
+	_ fs.MimeTyper = &Object{}
 )

http/http_internal_test.go

@@ -0,0 +1,308 @@
+// +build go1.7
+
+package http
+
+import (
+	"fmt"
+	"io/ioutil"
+	"net/http"
+	"net/http/httptest"
+	"net/url"
+	"os"
+	"path/filepath"
+	"sort"
+	"testing"
+	"time"
+
+	"github.com/ncw/rclone/fs"
+	"github.com/ncw/rclone/fstest"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+var (
+	remoteName = "TestHTTP"
+	testPath   = "test"
+	filesPath  = filepath.Join(testPath, "files")
+)
+
+// prepareServer the test server and return a function to tidy it up afterwards
+func prepareServer(t *testing.T) func() {
+	// file server for test/files
+	fileServer := http.FileServer(http.Dir(filesPath))
+
+	// Make the test server
+	ts := httptest.NewServer(fileServer)
+
+	// Configure the remote
+	fs.LoadConfig()
+	// fs.Config.LogLevel = fs.LogLevelDebug
+	// fs.Config.DumpHeaders = true
+	// fs.Config.DumpBodies = true
+	fs.ConfigFileSet(remoteName, "type", "http")
+	fs.ConfigFileSet(remoteName, "url", ts.URL)
+
+	// return a function to tidy up
+	return ts.Close
+}
+
+// prepare the test server and return a function to tidy it up afterwards
+func prepare(t *testing.T) (fs.Fs, func()) {
+	tidy := prepareServer(t)
+
+	// Instantiate it
+	f, err := NewFs(remoteName, "")
+	require.NoError(t, err)
+
+	return f, tidy
+}
+
+func testListRoot(t *testing.T, f fs.Fs) {
+	entries, err := f.List("")
+	require.NoError(t, err)
+
+	sort.Sort(entries)
+
+	require.Equal(t, 4, len(entries))
+
+	e := entries[0]
+	assert.Equal(t, "four", e.Remote())
+	assert.Equal(t, int64(0), e.Size())
+	_, ok := e.(*fs.Dir)
+	assert.True(t, ok)
+
+	e = entries[1]
+	assert.Equal(t, "one.txt", e.Remote())
+	assert.Equal(t, int64(6), e.Size())
+	_, ok = e.(*Object)
+	assert.True(t, ok)
+
+	e = entries[2]
+	assert.Equal(t, "three", e.Remote())
+	assert.Equal(t, int64(0), e.Size())
+	_, ok = e.(*fs.Dir)
+	assert.True(t, ok)
+
+	e = entries[3]
+	assert.Equal(t, "two.html", e.Remote())
+	assert.Equal(t, int64(7), e.Size())
+	_, ok = e.(*Object)
+	assert.True(t, ok)
+}
+
+func TestListRoot(t *testing.T) {
+	f, tidy := prepare(t)
+	defer tidy()
+	testListRoot(t, f)
+}
+
+func TestListSubDir(t *testing.T) {
+	f, tidy := prepare(t)
+	defer tidy()
+
+	entries, err := f.List("three")
+	require.NoError(t, err)
+
+	sort.Sort(entries)
+
+	assert.Equal(t, 1, len(entries))
+
+	e := entries[0]
+	assert.Equal(t, "three/underthree.txt", e.Remote())
+	assert.Equal(t, int64(9), e.Size())
+	_, ok := e.(*Object)
+	assert.True(t, ok)
+}
+
+func TestNewObject(t *testing.T) {
+	f, tidy := prepare(t)
+	defer tidy()
+
+	o, err := f.NewObject("four/underfour.txt")
+	require.NoError(t, err)
+
+	assert.Equal(t, "four/underfour.txt", o.Remote())
+	assert.Equal(t, int64(9), o.Size())
+	_, ok := o.(*Object)
+	assert.True(t, ok)
+
+	// Test the time is correct on the object
+
+	tObj := o.ModTime()
+
+	fi, err := os.Stat(filepath.Join(filesPath, "four", "underfour.txt"))
+	require.NoError(t, err)
+	tFile := fi.ModTime()
+
+	dt, ok := fstest.CheckTimeEqualWithPrecision(tObj, tFile, time.Second)
+	assert.True(t, ok, fmt.Sprintf("%s: Modification time difference too big |%s| > %s (%s vs %s) (precision %s)", o.Remote(), dt, time.Second, tObj, tFile, time.Second))
+}
+
+func TestOpen(t *testing.T) {
+	f, tidy := prepare(t)
+	defer tidy()
+
+	o, err := f.NewObject("four/underfour.txt")
+	require.NoError(t, err)
+
+	// Test normal read
+	fd, err := o.Open()
+	require.NoError(t, err)
+	data, err := ioutil.ReadAll(fd)
+	require.NoError(t, fd.Close())
+	assert.Equal(t, "beetroot\n", string(data))
+
+	// Test with range request
+	fd, err = o.Open(&fs.RangeOption{Start: 1, End: 5})
+	require.NoError(t, err)
+	data, err = ioutil.ReadAll(fd)
+	require.NoError(t, fd.Close())
+	assert.Equal(t, "eetro", string(data))
+}
+
+func TestMimeType(t *testing.T) {
+	f, tidy := prepare(t)
+	defer tidy()
+
+	o, err := f.NewObject("four/underfour.txt")
+	require.NoError(t, err)
+
+	do, ok := o.(fs.MimeTyper)
+	require.True(t, ok)
+	assert.Equal(t, "text/plain; charset=utf-8", do.MimeType())
+}
+
+func TestIsAFileRoot(t *testing.T) {
+	tidy := prepareServer(t)
+	defer tidy()
+
+	f, err := NewFs(remoteName, "one.txt")
+	assert.Equal(t, err, fs.ErrorIsFile)
+
+	testListRoot(t, f)
+}
+
+func TestIsAFileSubDir(t *testing.T) {
+	tidy := prepareServer(t)
+	defer tidy()
+
+	f, err := NewFs(remoteName, "three/underthree.txt")
+	assert.Equal(t, err, fs.ErrorIsFile)
+
+	entries, err := f.List("")
+	require.NoError(t, err)
+
+	sort.Sort(entries)
+
+	assert.Equal(t, 1, len(entries))
+
+	e := entries[0]
+	assert.Equal(t, "underthree.txt", e.Remote())
+	assert.Equal(t, int64(9), e.Size())
+	_, ok := e.(*Object)
+	assert.True(t, ok)
+}
+
+func TestParseName(t *testing.T) {
+	for i, test := range []struct {
+		base   string
+		val    string
+		wantOK bool
+		want   string
+	}{
+		{"http://example.com/", "potato", true, "potato"},
+		{"http://example.com/dir/", "potato", true, "potato"},
+		{"http://example.com/dir/", "../dir/potato", true, "potato"},
+		{"http://example.com/dir/", "..", false, ""},
+		{"http://example.com/dir/", "http://example.com/", false, ""},
+		{"http://example.com/dir/", "http://example.com/dir/", false, ""},
+		{"http://example.com/dir/", "http://example.com/dir/potato", true, "potato"},
+		{"http://example.com/dir/", "/dir/", false, ""},
+		{"http://example.com/dir/", "/dir/potato", true, "potato"},
+		{"http://example.com/dir/", "subdir/potato", false, ""},
+	} {
+		u, err := url.Parse(test.base)
+		require.NoError(t, err)
+		got, gotOK := parseName(u, test.val)
+		what := fmt.Sprintf("test %d base=%q, val=%q", i, test.base, test.val)
+		assert.Equal(t, test.wantOK, gotOK, what)
+		assert.Equal(t, test.want, got, what)
+	}
+}
+
+// Load HTML from the file given and parse it, checking it against the entries passed in
+func parseHTML(t *testing.T, name string, base string, want []string) {
+	in, err := os.Open(filepath.Join(testPath, "index_files", name))
+	require.NoError(t, err)
+	defer func() {
+		require.NoError(t, in.Close())
+	}()
+	if base == "" {
+		base = "http://example.com/"
+	}
+	u, err := url.Parse(base)
+	require.NoError(t, err)
+	entries, err := parse(u, in)
+	require.NoError(t, err)
+	assert.Equal(t, want, entries)
+}
+
+func TestParseEmpty(t *testing.T) {
+	parseHTML(t, "empty.html", "", []string(nil))
+}
+
+func TestParseApache(t *testing.T) {
+	parseHTML(t, "apache.html", "http://example.com/nick/pub/", []string{
+		"SWIG-embed.tar.gz",
+		"avi2dvd.pl",
+		"cambert.exe",
+		"cambert.gz",
+		"fedora_demo.gz",
+		"gchq-challenge/",
+		"mandelterm/",
+		"pgp-key.txt",
+		"pymath/",
+		"rclone",
+		"readdir.exe",
+		"rush_hour_solver_cut_down.py",
+		"snake-puzzle/",
+		"stressdisk/",
+		"timer-test",
+		"words-to-regexp.pl",
+	})
+}
+
+func TestParseMemstore(t *testing.T) {
+	parseHTML(t, "memstore.html", "", []string{
+		"test/",
+		"v1.35/",
+		"v1.36-01-g503cd84/",
+		"rclone-beta-latest-freebsd-386.zip",
+		"rclone-beta-latest-freebsd-amd64.zip",
+		"rclone-beta-latest-windows-amd64.zip",
+	})
+}
+
+func TestParseNginx(t *testing.T) {
+	parseHTML(t, "nginx.html", "", []string{
+		"deltas/",
+		"objects/",
+		"refs/",
+		"state/",
+		"config",
+		"summary",
+	})
+}
+
+func TestParseCaddy(t *testing.T) {
+	parseHTML(t, "caddy.html", "", []string{
+		"mimetype.zip",
+		"rclone-delete-empty-dirs.py",
+		"rclone-show-empty-dirs.py",
+		"stat-windows-386.zip",
+		"v1.36-155-gcf29ee8b-team-driveβ/",
+		"v1.36-156-gca76b3fb-team-driveβ/",
+		"v1.36-156-ge1f0e0f5-team-driveβ/",
+		"v1.36-22-g06ea13a-ssh-agentβ/",
+	})
+}

http/http_unsupported.go

@@ -0,0 +1,6 @@
+// Build for mount for unsupported platforms to stop go complaining
+// about "no buildable Go source files "
+
+// +build !go1.7
+
+package http

http/test/files/four/underfour.txt

@@ -0,0 +1 @@
+beetroot

http/test/files/one.txt

@@ -0,0 +1 @@
+hello

http/test/files/three/underthree.txt

@@ -0,0 +1 @@
+rutabaga

http/test/files/two.html

@@ -0,0 +1 @@
+potato

http/test/index_files/apache.html

@@ -0,0 +1,28 @@
+<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
+<html>
+ <head>
+  <title>Index of /nick/pub</title>
+ </head>
+ <body>
+<h1>Index of /nick/pub</h1>
+<table><tr><th><img src="/icons/blank.gif" alt="[ICO]"></th><th><a href="?C=N;O=D">Name</a></th><th><a href="?C=M;O=A">Last modified</a></th><th><a href="?C=S;O=A">Size</a></th><th><a href="?C=D;O=A">Description</a></th></tr><tr><th colspan="5"><hr></th></tr>
+<tr><td valign="top"><img src="/icons/back.gif" alt="[DIR]"></td><td><a href="/nick/">Parent Directory</a></td><td>&nbsp;</td><td align="right">  - </td><td>&nbsp;</td></tr>
+<tr><td valign="top"><img src="/icons/compressed.gif" alt="[   ]"></td><td><a href="SWIG-embed.tar.gz">SWIG-embed.tar.gz</a></td><td align="right">29-Nov-2005 16:27  </td><td align="right">2.3K</td><td>&nbsp;</td></tr>
+<tr><td valign="top"><img src="/icons/text.gif" alt="[TXT]"></td><td><a href="avi2dvd.pl">avi2dvd.pl</a></td><td align="right">14-Apr-2010 23:07  </td><td align="right"> 17K</td><td>&nbsp;</td></tr>
+<tr><td valign="top"><img src="/icons/binary.gif" alt="[   ]"></td><td><a href="cambert.exe">cambert.exe</a></td><td align="right">15-Dec-2006 18:07  </td><td align="right"> 54K</td><td>&nbsp;</td></tr>
+<tr><td valign="top"><img src="/icons/compressed.gif" alt="[   ]"></td><td><a href="cambert.gz">cambert.gz</a></td><td align="right">14-Apr-2010 23:07  </td><td align="right"> 18K</td><td>&nbsp;</td></tr>
+<tr><td valign="top"><img src="/icons/compressed.gif" alt="[   ]"></td><td><a href="fedora_demo.gz">fedora_demo.gz</a></td><td align="right">08-Jun-2007 11:01  </td><td align="right">1.0M</td><td>&nbsp;</td></tr>
+<tr><td valign="top"><img src="/icons/folder.gif" alt="[DIR]"></td><td><a href="gchq-challenge/">gchq-challenge/</a></td><td align="right">24-Dec-2016 15:24  </td><td align="right">  - </td><td>&nbsp;</td></tr>
+<tr><td valign="top"><img src="/icons/folder.gif" alt="[DIR]"></td><td><a href="mandelterm/">mandelterm/</a></td><td align="right">13-Jul-2013 22:22  </td><td align="right">  - </td><td>&nbsp;</td></tr>
+<tr><td valign="top"><img src="/icons/text.gif" alt="[TXT]"></td><td><a href="pgp-key.txt">pgp-key.txt</a></td><td align="right">14-Apr-2010 23:07  </td><td align="right">400 </td><td>&nbsp;</td></tr>
+<tr><td valign="top"><img src="/icons/folder.gif" alt="[DIR]"></td><td><a href="pymath/">pymath/</a></td><td align="right">24-Dec-2016 15:24  </td><td align="right">  - </td><td>&nbsp;</td></tr>
+<tr><td valign="top"><img src="/icons/unknown.gif" alt="[   ]"></td><td><a href="rclone">rclone</a></td><td align="right">09-May-2017 17:15  </td><td align="right"> 22M</td><td>&nbsp;</td></tr>
+<tr><td valign="top"><img src="/icons/binary.gif" alt="[   ]"></td><td><a href="readdir.exe">readdir.exe</a></td><td align="right">21-Oct-2016 14:47  </td><td align="right">1.6M</td><td>&nbsp;</td></tr>
+<tr><td valign="top"><img src="/icons/text.gif" alt="[TXT]"></td><td><a href="rush_hour_solver_cut_down.py">rush_hour_solver_cut_down.py</a></td><td align="right">23-Jul-2009 11:44  </td><td align="right"> 14K</td><td>&nbsp;</td></tr>
+<tr><td valign="top"><img src="/icons/folder.gif" alt="[DIR]"></td><td><a href="snake-puzzle/">snake-puzzle/</a></td><td align="right">25-Sep-2016 20:56  </td><td align="right">  - </td><td>&nbsp;</td></tr>
+<tr><td valign="top"><img src="/icons/folder.gif" alt="[DIR]"></td><td><a href="stressdisk/">stressdisk/</a></td><td align="right">08-Nov-2016 14:25  </td><td align="right">  - </td><td>&nbsp;</td></tr>
+<tr><td valign="top"><img src="/icons/unknown.gif" alt="[   ]"></td><td><a href="timer-test">timer-test</a></td><td align="right">09-May-2017 17:05  </td><td align="right">1.5M</td><td>&nbsp;</td></tr>
+<tr><td valign="top"><img src="/icons/text.gif" alt="[TXT]"></td><td><a href="words-to-regexp.pl">words-to-regexp.pl</a></td><td align="right">01-Mar-2005 20:43  </td><td align="right">6.0K</td><td>&nbsp;</td></tr>
+<tr><th colspan="5"><hr></th></tr>
+</table>
+</body></html>

http/test/index_files/caddy.html

@@ -0,0 +1,378 @@
+<!DOCTYPE html>
+<html>
+	<head>
+		<title>/</title>
+		<meta charset="utf-8">
+		<meta name="viewport" content="width=device-width, initial-scale=1.0">
+<style>
+* { padding: 0; margin: 0; }
+
+body {
+	font-family: sans-serif;
+	text-rendering: optimizespeed;
+}
+
+a {
+	color: #006ed3;
+	text-decoration: none;
+}
+
+a:hover,
+h1 a:hover {
+	color: #319cff;
+}
+
+header,
+#summary {
+	padding-left: 5%;
+	padding-right: 5%;
+}
+
+th:first-child,
+td:first-child {
+	padding-left: 5%;
+}
+
+th:last-child,
+td:last-child {
+	padding-right: 5%;
+}
+
+header {
+	padding-top: 25px;
+	padding-bottom: 15px;
+	background-color: #f2f2f2;
+}
+
+h1 {
+	font-size: 20px;
+	font-weight: normal;
+	white-space: nowrap;
+	overflow-x: hidden;
+	text-overflow: ellipsis;
+}
+
+h1 a {
+	color: inherit;
+}
+
+h1 a:hover {
+	text-decoration: underline;
+}
+
+main {
+	display: block;
+}
+
+.meta {
+	font-size: 12px;
+	font-family: Verdana, sans-serif;
+	border-bottom: 1px solid #9C9C9C;
+	padding-top: 10px;
+	padding-bottom: 10px;
+}
+
+.meta-item {
+	margin-right: 1em;
+}
+
+#filter {
+	padding: 4px;
+	border: 1px solid #CCC;
+}
+
+table {
+	width: 100%;
+	border-collapse: collapse;
+}
+
+tr {
+	border-bottom: 1px dashed #dadada;
+}
+
+tbody tr:hover {
+	background-color: #ffffec;
+}
+
+th,
+td {
+	text-align: left;
+	padding: 10px 0;
+}
+
+th {
+	padding-top: 15px;
+	padding-bottom: 15px;
+	font-size: 16px;
+	white-space: nowrap;
+}
+
+th a {
+	color: black;
+}
+
+th svg {
+	vertical-align: middle;
+}
+
+td {
+	font-size: 14px;
+}
+
+td:first-child {
+	width: 50%;
+}
+
+th:last-child,
+td:last-child {
+	text-align: right;
+}
+
+td:first-child svg {
+	position: absolute;
+}
+
+td .name,
+td .goup {
+	margin-left: 1.75em;
+	word-break: break-all;
+	overflow-wrap: break-word;
+	white-space: pre-wrap;
+}
+
+footer {
+	padding: 40px 20px;
+	font-size: 12px;
+	text-align: center;
+}
+
+@media (max-width: 600px) {
+	.hideable {
+		display: none;
+	}
+
+	td:first-child {
+		width: auto;
+	}
+
+	th:nth-child(2),
+	td:nth-child(2) {
+		padding-right: 5%;
+		text-align: right;
+	}
+}
+</style>
+	</head>
+	<body>
+		<svg version="1.1" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" height="0" width="0" style="position: absolute;">
+			<defs>
+				<!-- Folder -->
+				<linearGradient id="f" y2="640" gradientUnits="userSpaceOnUse" x2="244.84" gradientTransform="matrix(.97319 0 0 1.0135 -.50695 -13.679)" y1="415.75" x1="244.84">
+					<stop stop-color="#b3ddfd" offset="0"/>
+					<stop stop-color="#69c" offset="1"/>
+				</linearGradient>
+				<linearGradient id="e" y2="571.06" gradientUnits="userSpaceOnUse" x2="238.03" gradientTransform="translate(0,2)" y1="346.05" x1="236.26">
+					<stop stop-color="#ace" offset="0"/>
+					<stop stop-color="#369" offset="1"/>
+				</linearGradient>
+				<g id="folder" transform="translate(-266.06 -193.36)">
+					<g transform="matrix(.066019 0 0 .066019 264.2 170.93)">
+						<g transform="matrix(1.4738 0 0 1.4738 -52.053 -166.93)">
+							<path fill="#69c" d="m98.424 343.78c-11.08 0-20 8.92-20 20v48.5 33.719 105.06c0 11.08 8.92 20 20 20h279.22c11.08 0 20-8.92 20-20v-138.78c0-11.08-8.92-20-20-20h-117.12c-7.5478-1.1844-9.7958-6.8483-10.375-11.312v-5.625-11.562c0-11.08-8.92-20-20-20h-131.72z"/>
+							<rect rx="12.885" ry="12.199" height="227.28" width="366.69" y="409.69" x="54.428" fill="#369"/>
+							<path fill="url(#e)" d="m98.424 345.78c-11.08 0-20 8.92-20 20v48.5 33.719 105.06c0 11.08 8.92 20 20 20h279.22c11.08 0 20-8.92 20-20v-138.78c0-11.08-8.92-20-20-20h-117.12c-7.5478-1.1844-9.7958-6.8483-10.375-11.312v-5.625-11.562c0-11.08-8.92-20-20-20h-131.72z"/>
+							<rect rx="12.885" ry="12.199" height="227.28" width="366.69" y="407.69" x="54.428" fill="url(#f)"/>
+						</g>
+					</g>
+				</g>
+
+				<!-- File -->
+				<linearGradient id="a">
+					<stop stop-color="#cbcbcb" offset="0"/>
+					<stop stop-color="#f0f0f0" offset=".34923"/>
+					<stop stop-color="#e2e2e2" offset="1"/>
+				</linearGradient>
+				<linearGradient id="d" y2="686.15" xlink:href="#a" gradientUnits="userSpaceOnUse" y1="207.83" gradientTransform="matrix(.28346 0 0 .31053 -608.52 485.11)" x2="380.1" x1="749.25"/>
+				<linearGradient id="c" y2="287.74" xlink:href="#a" gradientUnits="userSpaceOnUse" y1="169.44" gradientTransform="matrix(.28342 0 0 .31057 -608.52 485.11)" x2="622.33" x1="741.64"/>
+				<linearGradient id="b" y2="418.54" gradientUnits="userSpaceOnUse" y1="236.13" gradientTransform="matrix(.29343 0 0 .29999 -608.52 485.11)" x2="330.88" x1="687.96">
+					<stop stop-color="#fff" offset="0"/>
+					<stop stop-color="#fff" stop-opacity="0" offset="1"/>
+				</linearGradient>
+				<g id="file" transform="translate(-278.15 -216.59)">
+					<g fill-rule="evenodd" transform="matrix(.19775 0 0 .19775 381.05 112.68)">
+						<path d="m-520.17 525.5v36.739 36.739 36.739 36.739h33.528 33.528 33.528 33.528v-36.739-36.739-36.739l-33.528-36.739h-33.528-33.528-33.528z" stroke-opacity=".36478" stroke-width=".42649" fill="#fff"/>
+						<g>
+							<path d="m-520.11 525.68v36.739 36.739 36.739 36.739h33.528 33.528 33.528 33.528v-36.739-36.739-36.739l-33.528-36.739h-33.528-33.528-33.528z" stroke-opacity=".36478" stroke="#000" stroke-width=".42649" fill="url(#d)"/>
+							<path d="m-386 562.42c-10.108-2.9925-23.206-2.5682-33.101-0.86253 1.7084-10.962 1.922-24.701-0.4271-35.877l33.528 36.739z" stroke-width=".95407pt" fill="url(#c)"/>
+							<path d="m-519.13 537-0.60402 134.7h131.68l0.0755-33.296c-2.9446 1.1325-32.692-40.998-70.141-39.186-37.483 1.8137-27.785-56.777-61.006-62.214z" stroke-width="1pt" fill="url(#b)"/>
+						</g>
+					</g>
+				</g>
+
+				<!-- Up arrow -->
+				<g id="up-arrow" transform="translate(-279.22 -208.12)">
+					<path transform="matrix(.22413 0 0 .12089 335.67 164.35)" stroke-width="0" d="m-194.17 412.01h-28.827-28.827l14.414-24.965 14.414-24.965 14.414 24.965z"/>
+				</g>
+
+				<!-- Down arrow -->
+				<g id="down-arrow" transform="translate(-279.22 -208.12)">
+					<path transform="matrix(.22413 0 0 -.12089 335.67 257.93)" stroke-width="0" d="m-194.17 412.01h-28.827-28.827l14.414-24.965 14.414-24.965 14.414 24.965z"/>
+				</g>
+			</defs>
+		</svg>
+
+		<header>
+			<h1>
+				<a href="/">/</a>
+			</h1>
+		</header>
+		<main>
+			<div class="meta">
+				<div id="summary">
+					<span class="meta-item"><b>4</b> directories</span>
+					<span class="meta-item"><b>4</b> files</span>
+					<span class="meta-item"><input type="text" placeholder="filter" id="filter" onkeyup='filter()'></span>
+				</div>
+			</div>
+			<div class="listing">
+				<table aria-describedby="summary">
+					<thead>
+					<tr>
+						<th>
+							<a href="?sort=name&order=desc">Name <svg width="1em" height=".4em" version="1.1" viewBox="0 0 12.922194 6.0358899"><use xlink:href="#up-arrow"></use></svg></a>
+						</th>
+						<th>
+							<a href="?sort=size&order=asc">Size</a>
+						</th>
+						<th class="hideable">
+							<a href="?sort=time&order=asc">Modified</a>
+						</th>
+					</tr>
+					</thead>
+					<tbody>
+					<tr class="file">
+						<td>
+							<a href="./mimetype.zip">
+								<svg width="1.5em" height="1em" version="1.1" viewBox="0 0 26.604381 29.144726"><use xlink:href="#file"></use></svg>
+								<span class="name">mimetype.zip</span>
+							</a>
+						</td>
+						<td data-order="783696">765 KiB</td>
+						<td class="hideable"><time datetime="2016-04-04T15:36:49Z">04/04/2016 03:36:49 PM +00:00</time></td>
+					</tr>
+					<tr class="file">
+						<td>
+							<a href="./rclone-delete-empty-dirs.py">
+								<svg width="1.5em" height="1em" version="1.1" viewBox="0 0 26.604381 29.144726"><use xlink:href="#file"></use></svg>
+								<span class="name">rclone-delete-empty-dirs.py</span>
+							</a>
+						</td>
+						<td data-order="1271">1.2 KiB</td>
+						<td class="hideable"><time datetime="2016-10-26T16:05:08Z">10/26/2016 04:05:08 PM +00:00</time></td>
+					</tr>
+					<tr class="file">
+						<td>
+							<a href="./rclone-show-empty-dirs.py">
+								<svg width="1.5em" height="1em" version="1.1" viewBox="0 0 26.604381 29.144726"><use xlink:href="#file"></use></svg>
+								<span class="name">rclone-show-empty-dirs.py</span>
+							</a>
+						</td>
+						<td data-order="868">868 B</td>
+						<td class="hideable"><time datetime="2016-10-26T09:29:34Z">10/26/2016 09:29:34 AM +00:00</time></td>
+					</tr>
+					<tr class="file">
+						<td>
+							<a href="./stat-windows-386.zip">
+								<svg width="1.5em" height="1em" version="1.1" viewBox="0 0 26.604381 29.144726"><use xlink:href="#file"></use></svg>
+								<span class="name">stat-windows-386.zip</span>
+							</a>
+						</td>
+						<td data-order="704960">688 KiB</td>
+						<td class="hideable"><time datetime="2016-08-14T20:44:58Z">08/14/2016 08:44:58 PM +00:00</time></td>
+					</tr>
+					<tr class="file">
+						<td>
+							<a href="./v1.36-155-gcf29ee8b-team-drive%CE%B2/">
+								<svg width="1.5em" height="1em" version="1.1" viewBox="0 0 35.678803 28.527945"><use xlink:href="#folder"></use></svg>
+								<span class="name">v1.36-155-gcf29ee8b-team-driveβ</span>
+							</a>
+						</td>
+						<td data-order="-1">&mdash;</td>
+						<td class="hideable"><time datetime="2017-06-01T21:28:09Z">06/01/2017 09:28:09 PM +00:00</time></td>
+					</tr>
+					<tr class="file">
+						<td>
+							<a href="./v1.36-156-gca76b3fb-team-drive%CE%B2/">
+								<svg width="1.5em" height="1em" version="1.1" viewBox="0 0 35.678803 28.527945"><use xlink:href="#folder"></use></svg>
+								<span class="name">v1.36-156-gca76b3fb-team-driveβ</span>
+							</a>
+						</td>
+						<td data-order="-1">&mdash;</td>
+						<td class="hideable"><time datetime="2017-06-04T08:53:04Z">06/04/2017 08:53:04 AM +00:00</time></td>
+					</tr>
+					<tr class="file">
+						<td>
+							<a href="./v1.36-156-ge1f0e0f5-team-drive%CE%B2/">
+								<svg width="1.5em" height="1em" version="1.1" viewBox="0 0 35.678803 28.527945"><use xlink:href="#folder"></use></svg>
+								<span class="name">v1.36-156-ge1f0e0f5-team-driveβ</span>
+							</a>
+						</td>
+						<td data-order="-1">&mdash;</td>
+						<td class="hideable"><time datetime="2017-06-02T10:38:05Z">06/02/2017 10:38:05 AM +00:00</time></td>
+					</tr>
+					<tr class="file">
+						<td>
+							<a href="./v1.36-22-g06ea13a-ssh-agent%CE%B2/">
+								<svg width="1.5em" height="1em" version="1.1" viewBox="0 0 35.678803 28.527945"><use xlink:href="#folder"></use></svg>
+								<span class="name">v1.36-22-g06ea13a-ssh-agentβ</span>
+							</a>
+						</td>
+						<td data-order="-1">&mdash;</td>
+						<td class="hideable"><time datetime="2017-04-10T13:58:02Z">04/10/2017 01:58:02 PM +00:00</time></td>
+					</tr>
+					</tbody>
+				</table>
+			</div>
+		</main>
+		<footer>
+			Served with <a rel="noopener noreferrer" href="https://caddyserver.com">Caddy</a>
+		</footer>
+		<script>
+			var filterEl = document.getElementById('filter');
+			function filter() {
+				var q = filterEl.value.trim().toLowerCase();
+				var elems = document.querySelectorAll('tr.file');
+				elems.forEach(function(el) {
+					if (!q) {
+						el.style.display = '';
+						return;
+					}
+					var nameEl = el.querySelector('.name');
+					var nameVal = nameEl.textContent.trim().toLowerCase();
+					if (nameVal.indexOf(q) !== -1) {
+						el.style.display = '';
+					} else {
+						el.style.display = 'none';
+					}
+				});
+			}
+
+			function localizeDatetime(e, index, ar) {
+				if (e.textContent === undefined) {
+					return;
+				}
+				var d = new Date(e.getAttribute('datetime'));
+				if (isNaN(d)) {
+					d = new Date(e.textContent);
+					if (isNaN(d)) {
+						return;
+					}
+				}
+				e.textContent = d.toLocaleString();
+			}
+			var timeList = Array.prototype.slice.call(document.getElementsByTagName("time"));
+			timeList.forEach(localizeDatetime);
+		</script>
+	</body>
+</html>
+

http/test/index_files/memstore.html

@@ -0,0 +1,77 @@
+<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
+<html xmlns="http://www.w3.org/1999/xhtml">
+<head>
+  <meta http-equiv="content-type" content="text/html; charset=utf-8" />
+  <meta name="robots" content="noindex" />
+  <title>Index of /</title>
+</head>
+<body>
+<div id="content">
+  <h1>Index of /</h1>
+
+  <table>
+  <thead>
+      <tr>
+        <th>Name</th>
+        <th>Type</th>
+        <th>Size</th>
+        <th>Last modified</th>
+        <th>MD5</th>
+      </tr>
+  </thead>
+  <tbody>
+  
+      <tr>
+        <td><a href="test/">test/</a></td>
+        <td>application/directory</td>
+        <td>0 bytes</td>
+        <td>-</td>
+        <td>-</td>
+      </tr>
+  
+      <tr>
+        <td><a href="v1.35/">v1.35/</a></td>
+        <td>application/directory</td>
+        <td>0 bytes</td>
+        <td>-</td>
+        <td>-</td>
+      </tr>
+  
+      <tr>
+        <td><a href="v1.36-01-g503cd84/">v1.36-01-g503cd84/</a></td>
+        <td>application/directory</td>
+        <td>0 bytes</td>
+        <td>-</td>
+        <td>-</td>
+      </tr>
+
+      <tr>
+        <td><a href="rclone-beta-latest-freebsd-386.zip">rclone-beta-latest-freebsd-386.zip</a></td>
+        <td>application/zip</td>
+        <td>4.6 MB</td>
+        <td>2017-06-19 14:04:52</td>
+        <td>e747003c69c81e675f206a715264bfa8</td>
+      </tr>
+  
+      <tr>
+        <td><a href="rclone-beta-latest-freebsd-amd64.zip">rclone-beta-latest-freebsd-amd64.zip</a></td>
+        <td>application/zip</td>
+        <td>5.0 MB</td>
+        <td>2017-06-19 14:04:53</td>
+        <td>ff30b5e9bf2863a2373069142e6f2b7f</td>
+      </tr>
+  
+      <tr>
+        <td><a href="rclone-beta-latest-windows-amd64.zip">rclone-beta-latest-windows-amd64.zip</a></td>
+        <td>application/x-zip-compressed</td>
+        <td>4.9 MB</td>
+        <td>2017-06-19 13:56:02</td>
+        <td>851a5547a0495cbbd94cbc90a80ed6f5</td>
+      </tr>
+  
+  </tbody>
+  </table>
+  <p class="right"><a href="http://www.memset.com/"><img src="http://www.memset.com/images/Memset_logo_2010.gif" alt="Memset Ltd." /></a></p>
+</div>
+</body>
+</html>

http/test/index_files/nginx.html

@@ -0,0 +1,12 @@
+<html>
+<head><title>Index of /atomic/fedora/</title></head>
+<body bgcolor="white">
+<h1>Index of /atomic/fedora/</h1><hr><pre><a href="../">../</a>
+<a href="deltas/">deltas/</a>                                            04-May-2017 21:37                   -
+<a href="objects/">objects/</a>                                           04-May-2017 20:44                   -
+<a href="refs/">refs/</a>                                              04-May-2017 20:42                   -
+<a href="state/">state/</a>                                             04-May-2017 21:36                   -
+<a href="config">config</a>                                             04-May-2017 20:42                 118
+<a href="summary">summary</a>                                            04-May-2017 21:36                 806
+</pre><hr></body>
+</html>