Complete SFTP remote #521

* Add unit tests
  * Fix up remote so it passes tests
  * Add docs

README.md

@@ -23,6 +23,7 @@ Rclone is a command line program to sync files and directories to and from
   * Hubic
   * Backblaze B2
   * Yandex Disk
+  * SFTP
   * The local filesystem
 
 Features

bin/make_manual.py

@@ -29,6 +29,7 @@ docs = [
     "hubic.md",
     "b2.md",
     "yandex.md",
+    "sftp.md",
     "crypt.md",
     "local.md",
     "changelog.md",

docs/content/about.md

@@ -23,6 +23,7 @@ Rclone is a command line program to sync files and directories to and from
   * Hubic
   * Backblaze B2
   * Yandex Disk
+  * SFTP
   * The local filesystem
 
 Features

docs/content/docs.md

@@ -30,6 +30,7 @@ See the following for detailed instructions for
   * [Hubic](/hubic/)
   * [Microsoft One Drive](/onedrive/)
   * [Yandex Disk](/yandex/)
+  * [SFTP](/sftp/)
   * [Crypt](/crypt/) - to encrypt other remotes
 
 Usage

docs/content/overview.md

@@ -27,6 +27,7 @@ Here is an overview of the major features of each cloud storage system.
 | Hubic                  | MD5     | Yes     | No               | No              | R/W       |
 | Backblaze B2           | SHA1    | Yes     | No               | No              | R/W       |
 | Yandex Disk            | MD5     | Yes     | No               | No              | R/W       |
+| SFTP                   | -       | Yes     | Depends          | No              | -         |
 | The local filesystem   | All     | Yes     | Depends          | No              | -         |
 
 ### Hash ###
@@ -60,7 +61,8 @@ This can cause problems when syncing between a case insensitive
 system and a case sensitive system.  The symptom of this is that no
 matter how many times you run the sync it never completes fully.
 
-The local filesystem may or may not be case sensitive depending on OS.
+The local filesystem and SFTP may or may not be case sensitive
+depending on OS.
 
   * Windows - usually case insensitive, though case is preserved
   * OSX - usually case insensitive, though it is possible to format case sensitive
@@ -113,6 +115,7 @@ operations more efficient.
 | Hubic                  | Yes † | Yes  | No   | No      | No      |
 | Backblaze B2           | No    | No   | No   | No      | Yes     |
 | Yandex Disk            | Yes   | No   | No   | No      | No  [#575](https://github.com/ncw/rclone/issues/575) |
+| SFTP                   | No    | No   | Yes  | Yes     | No      |
 | The local filesystem   | Yes   | No   | Yes  | Yes     | No      |
 
 

docs/content/sftp.md

@@ -0,0 +1,128 @@
+---
+title: "SFTP"
+description: "SFTP"
+date: "2017-02-01"
+---
+
+<i class="fa fa-server"></i> SFTP
+----------------------------------------
+
+SFTP is the [Secure (or SSH) File Transfer
+Protocol](https://en.wikipedia.org/wiki/SSH_File_Transfer_Protocol).
+
+It runs over SSH v2 and is standard with most modern SSH
+installations.
+
+Paths are specified as `remote:path`. If the path does not begin with
+a `/` it is relative to the home directory of the user.  An empty path
+`remote:` refers to the users home directory.
+
+Here is an example of making a SFTP configuration.  First run
+
+    rclone config
+
+This will guide you through an interactive setup process.  You will
+need your account number (a short hex number) and key (a long hex
+number) which you can get from the SFTP control panel.
+```
+No remotes found - make a new one
+n) New remote
+r) Rename remote
+c) Copy remote
+s) Set configuration password
+q) Quit config
+n/r/c/s/q> n
+name> remote
+Type of storage to configure.
+Choose a number from below, or type in your own value
+ 1 / Amazon Drive
+   \ "amazon cloud drive"
+ 2 / Amazon S3 (also Dreamhost, Ceph, Minio)
+   \ "s3"
+ 3 / Backblaze B2
+   \ "b2"
+ 4 / Dropbox
+   \ "dropbox"
+ 5 / Encrypt/Decrypt a remote
+   \ "crypt"
+ 6 / Google Cloud Storage (this is not Google Drive)
+   \ "google cloud storage"
+ 7 / Google Drive
+   \ "drive"
+ 8 / Hubic
+   \ "hubic"
+ 9 / Local Disk
+   \ "local"
+10 / Microsoft OneDrive
+   \ "onedrive"
+11 / Openstack Swift (Rackspace Cloud Files, Memset Memstore, OVH)
+   \ "swift"
+12 / SSH/SFTP Connection
+   \ "sftp"
+13 / Yandex Disk
+   \ "yandex"
+Storage> 12  
+SSH host to connect to
+Choose a number from below, or type in your own value
+ 1 / Connect to example.com
+   \ "example.com"
+host> example.com
+SSH username, leave blank for current username, ncw
+user> 
+SSH port
+port> 
+SSH password, leave blank to use ssh-agent
+y) Yes type in my own password
+g) Generate random password
+n) No leave this optional password blank
+y/g/n> n
+Remote config
+--------------------
+[remote]
+host = example.com
+user = 
+port = 
+pass = 
+--------------------
+y) Yes this is OK
+e) Edit this remote
+d) Delete this remote
+y/e/d> y
+```
+
+This remote is called `remote` and can now be used like this
+
+See all directories in the home directory
+
+    rclone lsd remote:
+
+Make a new directory
+
+    rclone mkdir remote:path/to/directory
+
+List the contents of a directory
+
+    rclone ls remote:path/to/directory
+
+Sync `/home/local/directory` to the remote directory, deleting any
+excess files in the directory.
+
+    rclone sync /home/local/directory remote:directory
+
+### Modified time ###
+
+Modified times are stored on the server to 1 second precision.
+
+Modified times are used in syncing and are fully supported.
+
+### Limitations ###
+
+SFTP does not support any checksums.
+
+SFTP isn't supported under plan9 until [this
+issue](https://github.com/pkg/sftp/issues/156) is fixed.
+
+Note that since SFTP isn't HTTP based the following flags don't work
+with it: `--dump-headers`, `--dump-bodies`, `--dump-auth`
+
+Note that `--timeout` isn't supported (but `--contimeout` is).

docs/layouts/chrome/navbar.html

@@ -60,6 +60,7 @@
                     <li><a href="/b2/"><i class="fa fa-fire"></i> Backblaze B2</a></li>
                     <li><a href="/local/"><i class="fa fa-file"></i> Local</a></li>
                     <li><a href="/yandex/"><i class="fa fa-space-shuttle"></i> Yandex Disk</a></li>
+                    <li><a href="/sftp/"><i class="fa fa-server"></i> SFTP</a></li>
                     <li><a href="/crypt/"><i class="fa fa-lock"></i> Crypt (encrypts the above)</a></li>
                   </ul>
                 </li>

fs/test_all.go

@@ -32,6 +32,7 @@ var (
 		"TestHubic:",
 		"TestOneDrive:",
 		"TestS3:",
+		"TestSftp:",
 		"TestSwift:",
 		"TestYandex:",
 	}

fstest/fstests/gen_tests.go

@@ -140,5 +140,6 @@ func main() {
 	generateTestProgram(t, fns, "Yandex", "")
 	generateTestProgram(t, fns, "Crypt", "")
 	generateTestProgram(t, fns, "Crypt", "2")
+	generateTestProgram(t, fns, "Sftp", "")
 	log.Printf("Done")
 }

sftp/sftp.go

@@ -1,4 +1,7 @@
 // Package sftp provides a filesystem interface using github.com/pkg/sftp
+
+// +build !plan9
+
 package sftp
 
 import (
@@ -14,6 +17,7 @@ import (
 	"golang.org/x/crypto/ssh/agent"
 
 	"github.com/ncw/rclone/fs"
+	"github.com/pkg/errors"
 	"github.com/pkg/sftp"
 )
 
@@ -48,14 +52,6 @@ func init() {
 	fs.Register(fsi)
 }
 
-type stringer interface {
-	String() string
-}
-
-func debug(o stringer, msg string) {
-	fs.Debug(o, msg)
-}
-
 // Fs stores the interface to the remote SFTP files
 type Fs struct {
 	name       string
@@ -86,9 +82,6 @@ func NewFs(name, root string) (fs.Fs, error) {
 	host := fs.ConfigFileGet(name, "host")
 	port := fs.ConfigFileGet(name, "port")
 	pass := fs.ConfigFileGet(name, "pass")
-	if root == "" {
-		root = "."
-	}
 	if user == "" {
 		user = os.Getenv("USER")
 	}
@@ -98,19 +91,31 @@ func NewFs(name, root string) (fs.Fs, error) {
 	config := &ssh.ClientConfig{
 		User:    user,
 		Auth:    []ssh.AuthMethod{},
+		Timeout: fs.Config.ConnectTimeout,
 	}
 	if pass == "" {
-		if sshAgent, err := net.Dial("unix", os.Getenv("SSH_AUTH_SOCK")); err == nil {
+		authSock := os.Getenv("SSH_AUTH_SOCK")
+		if authSock == "" {
+			return nil, errors.New("SSH_AUTH_SOCK is unset so can't connect to ssh-agent")
+		}
+		sshAgent, err := net.Dial("unix", authSock)
+		if err != nil {
+			return nil, errors.Wrap(err, "couldn't connect to ssh-agent")
+		}
 		sshAgentClient := agent.NewClient(sshAgent)
-			signers, _ := sshAgentClient.Signers()
+		signers, err := sshAgentClient.Signers()
+		if err != nil {
+			return nil, errors.Wrap(err, "couldn't read ssh agent signers")
+		}
+		/*
 			for i, signer := range signers {
 				if 2*i < len(signers) {
 					signers[i] = signers[len(signers)-i-1]
 					signers[len(signers)-i-1] = signer
 				}
 			}
+		*/
 		config.Auth = append(config.Auth, ssh.PublicKeys(signers...))
-		}
 	} else {
 		clearpass, err := fs.Reveal(pass)
 		if err != nil {
@@ -118,12 +123,15 @@ func NewFs(name, root string) (fs.Fs, error) {
 		}
 		config.Auth = append(config.Auth, ssh.Password(clearpass))
 	}
-	if sshClient, err := ssh.Dial("tcp", host+":"+port, config); err != nil {
+	sshClient, err := ssh.Dial("tcp", host+":"+port, config)
-		return nil, err
+	if err != nil {
-	} else if sftpClient, err := sftp.NewClient(sshClient); err != nil {
+		return nil, errors.Wrap(err, "couldn't connect ssh")
+	}
+	sftpClient, err := sftp.NewClient(sshClient)
+	if err != nil {
 		_ = sshClient.Close()
-		return nil, err
+		return nil, errors.Wrap(err, "couldn't initialise SFTP")
-	} else {
+	}
 	f := &Fs{
 		name:       name,
 		root:       root,
@@ -132,8 +140,31 @@ func NewFs(name, root string) (fs.Fs, error) {
 		url:        "sftp://" + user + "@" + host + ":" + port + "/" + root,
 	}
 	f.features = (&fs.Features{}).Fill(f)
+	if root != "" {
+		// Check to see if the root actually an existing file
+		remote := path.Base(root)
+		f.root = path.Dir(root)
+		if f.root == "." {
+			f.root = ""
+		}
+		_, err := f.NewObject(remote)
+		if err != nil {
+			if err == fs.ErrorObjectNotFound {
+				// File doesn't exist so return old f
+				f.root = root
 				return f, nil
 			}
+			return nil, err
+		}
+		// return an error with an fs which points to the parent
+		return f, fs.ErrorIsFile
+	}
+	go func() {
+		// FIXME re-open the connection here...
+		err := f.sshClient.Conn.Wait()
+		fs.ErrorLog(f, "SSH connection closed: %v", err)
+	}()
+	return f, nil
 }
 
 // Name returns the configured name of the file system
@@ -163,129 +194,243 @@ func (f *Fs) Precision() time.Duration {
 
 // NewObject creates a new remote sftp file object
 func (f *Fs) NewObject(remote string) (fs.Object, error) {
-	debug(f, "New '"+remote+"'")
+	o := &Object{
-	info, err := f.sftpClient.Stat(f.sftpClient.Join(f.root, remote))
-	if err != nil {
-		return nil, err
-	}
-	object := &Object{
 		fs:     f,
 		remote: remote,
-		info:   info,
 	}
-	return object, nil
+	err := o.stat()
-}
-
-func (f *Fs) list(out fs.ListOpts, dirs string, name string, info os.FileInfo, level int, done *sync.WaitGroup) {
-	debug(f, "list '"+f.sftpClient.Join(dirs, name)+"'")
-	defer done.Done()
-	if info.IsDir() {
-		if out.IncludeDirectory(info.Name()) {
-			dir := &fs.Dir{
-				Name:  info.Name(),
-				When:  info.ModTime(),
-				Bytes: -1,
-				Count: -1,
-			}
-			if level >= out.Level() {
-				return
-			}
-			if infos, err := f.sftpClient.ReadDir(f.sftpClient.Join(f.root, dirs, name)); err == nil {
-				dir.Count = int64(len(infos))
-				out.AddDir(dir)
-				done.Add(len(infos))
-				for _, newInfo := range infos {
-					go f.list(out, f.sftpClient.Join(dirs, name), newInfo.Name(), newInfo, level+1, done)
-				}
-			}
-		}
-	} else {
-		file := &Object{
-			fs:     f,
-			remote: f.sftpClient.Join(dirs, info.Name()),
-			info:   info,
-		}
-		out.Add(file)
-	}
-}
-
-// List the files and directories starting at <dir>
-func (f *Fs) List(out fs.ListOpts, dir string) {
-	debug(f, "List '"+dir+"'")
-	if dir == "" {
-		dir = "."
-	}
-	var done sync.WaitGroup
-	if info, _ := f.sftpClient.Stat(f.sftpClient.Join(f.root, dir)); info != nil {
-		done.Add(1)
-		f.list(out, "", ".", info, 0, &done)
-	}
-	debug(f, "List--waiting")
-	done.Wait()
-	out.Finished()
-}
-
-// Put data from <in> into a new remote sftp file object described by <src.Remote()> and <src.ModTime()>
-func (f *Fs) Put(in io.Reader, src fs.ObjectInfo) (fs.Object, error) {
-	debug(f, "Put '"+src.Remote()+"'")
-	_ = f.mkdir(f.sftpClient.Join(f.root, filepath.Dir(src.Remote())))
-	file, err := f.sftpClient.Create(f.sftpClient.Join(f.root, src.Remote()))
-	if err != nil {
-		return nil, err
-	}
-	_, err = file.ReadFrom(in)
-	if err != nil {
-		return nil, err
-	}
-	o, err := f.NewObject(src.Remote())
-	if err != nil {
-		return nil, err
-	}
-	err = o.SetModTime(src.ModTime())
 	if err != nil {
 		return nil, err
 	}
 	return o, nil
 }
 
-func (f *Fs) mkdir(path string) error {
+// dirExists returns true,nil if the directory exists, false, nil if
-	debug(f, "mkdir '"+path+"'")
+// it doesn't or false, err
-	parent := filepath.Dir(path)
+func (f *Fs) dirExists(dir string) (bool, error) {
-	if parent != "." && parent != "/" {
+	if dir == "" {
-		_ = f.mkdir(parent)
+		dir = "."
 	}
-	return f.sftpClient.Mkdir(path)
+	info, err := f.sftpClient.Stat(dir)
+	if err != nil {
+		if os.IsNotExist(err) {
+			return false, nil
+		}
+		return false, errors.Wrap(err, "dirExists stat failed")
+	}
+	if !info.IsDir() {
+		return false, fs.ErrorIsFile
+	}
+	return true, nil
+}
+
+func (f *Fs) list(out fs.ListOpts, dir string, level int, wg *sync.WaitGroup, tokens chan struct{}) {
+	defer wg.Done()
+	// take a token
+	<-tokens
+	// return it when done
+	defer func() {
+		tokens <- struct{}{}
+	}()
+	sftpDir := path.Join(f.root, dir)
+	if sftpDir == "" {
+		sftpDir = "."
+	}
+	infos, err := f.sftpClient.ReadDir(sftpDir)
+	if err != nil {
+		err = errors.Wrapf(err, "error listing %q", dir)
+		fs.ErrorLog(f, "Listing failed: %v", err)
+		out.SetError(err)
+		return
+	}
+	for _, info := range infos {
+		remote := path.Join(dir, info.Name())
+		if info.IsDir() {
+			if out.IncludeDirectory(remote) {
+				dir := &fs.Dir{
+					Name:  remote,
+					When:  info.ModTime(),
+					Bytes: -1,
+					Count: -1,
+				}
+				out.AddDir(dir)
+				if level < out.Level() {
+					wg.Add(1)
+					go f.list(out, remote, level+1, wg, tokens)
+				}
+			}
+		} else {
+			file := &Object{
+				fs:     f,
+				remote: remote,
+				info:   info,
+			}
+			out.Add(file)
+		}
+	}
+}
+
+// List the files and directories starting at <dir>
+func (f *Fs) List(out fs.ListOpts, dir string) {
+	root := path.Join(f.root, dir)
+	ok, err := f.dirExists(root)
+	if err != nil {
+		out.SetError(errors.Wrap(err, "List failed"))
+		return
+	}
+	if !ok {
+		out.SetError(fs.ErrorDirNotFound)
+		return
+	}
+	// tokens to control the concurrency
+	tokens := make(chan struct{}, fs.Config.Checkers)
+	for i := 0; i < fs.Config.Checkers; i++ {
+		tokens <- struct{}{}
+	}
+	wg := new(sync.WaitGroup)
+	wg.Add(1)
+	f.list(out, dir, 1, wg, tokens)
+	wg.Wait()
+	out.Finished()
+}
+
+// Put data from <in> into a new remote sftp file object described by <src.Remote()> and <src.ModTime()>
+func (f *Fs) Put(in io.Reader, src fs.ObjectInfo) (fs.Object, error) {
+	err := f.mkParentDir(src.Remote())
+	if err != nil {
+		return nil, errors.Wrap(err, "Put mkParentDir failed")
+	}
+	// Temporary object under construction
+	o := &Object{
+		fs:     f,
+		remote: src.Remote(),
+	}
+	err = o.Update(in, src)
+	if err != nil {
+		return nil, err
+	}
+	return o, nil
+}
+
+// mkParentDir makes the parent of remote if necessary and any
+// directories above that
+func (f *Fs) mkParentDir(remote string) error {
+	parent := path.Dir(remote)
+	return f.mkdir(path.Join(f.root, parent))
+}
+
+// mkdir makes the directory and parents using native paths
+func (f *Fs) mkdir(path string) error {
+	if path == "." || path == "/" {
+		return nil
+	}
+	ok, err := f.dirExists(path)
+	if err != nil {
+		return errors.Wrap(err, "mkdir dirExists failed")
+	}
+	if ok {
+		return nil
+	}
+	parent := filepath.Dir(path)
+	err = f.mkdir(parent)
+	if err != nil {
+		return err
+	}
+	err = f.sftpClient.Mkdir(path)
+	if err != nil {
+		return errors.Wrapf(err, "mkdir %q failed", path)
+	}
+	return nil
 }
 
 // Mkdir makes the root directory of the Fs object
 func (f *Fs) Mkdir(dir string) error {
 	root := path.Join(f.root, dir)
-	debug(f, "Mkdir '"+root+"'")
-	o, _ := f.NewObject("")
-	if o == nil {
 	return f.mkdir(root)
 }
-	return nil
-}
 
 // Rmdir removes the root directory of the Fs object
 func (f *Fs) Rmdir(dir string) error {
 	root := path.Join(f.root, dir)
-	debug(f, "Rmdir '"+root+"'")
+	return f.sftpClient.Remove(root)
-	return f.sftpClient.Remove(f.root)
 }
 
 // Move renames a remote sftp file object
 func (f *Fs) Move(src fs.Object, remote string) (fs.Object, error) {
-	debug(f, "Move '"+src.Remote()+"' to '"+remote+"'")
+	srcObj, ok := src.(*Object)
-	err := f.sftpClient.Rename(
+	if !ok {
-		f.sftpClient.Join(f.root, src.Remote()),
+		fs.Debug(src, "Can't move - not same remote type")
-		f.sftpClient.Join(f.root, remote))
+		return nil, fs.ErrorCantMove
+	}
+	err := f.mkParentDir(remote)
 	if err != nil {
-		return nil, err
+		return nil, errors.Wrap(err, "Move mkParentDir failed")
+	}
+	err = f.sftpClient.Rename(
+		srcObj.path(),
+		path.Join(f.root, remote),
+	)
+	if err != nil {
+		return nil, errors.Wrap(err, "Move Rename failed")
 	}
 	dstObj, err := f.NewObject(remote)
-	return dstObj, err
+	if err != nil {
+		return nil, errors.Wrap(err, "Move NewObject failed")
+	}
+	return dstObj, nil
+}
+
+// DirMove moves src directory to this remote using server side move
+// operations.
+//
+// Will only be called if src.Fs().Name() == f.Name()
+//
+// If it isn't possible then return fs.ErrorCantDirMove
+//
+// If destination exists then return fs.ErrorDirExists
+func (f *Fs) DirMove(src fs.Fs) error {
+	srcFs, ok := src.(*Fs)
+	if !ok {
+		fs.Debug(srcFs, "Can't move directory - not same remote type")
+		return fs.ErrorCantDirMove
+	}
+
+	// Check if destination exists
+	ok, err := f.dirExists(f.root)
+	if err != nil {
+		return errors.Wrap(err, "DirMove dirExists dst failed")
+	}
+	if ok {
+		return fs.ErrorDirExists
+	}
+
+	// Refuse to move to or from the root
+	if f.root == "" || srcFs.root == "" {
+		fs.Debug(src, "DirMove error: Can't move root")
+		return errors.New("can't move root directory")
+	}
+
+	// Make sure the parent directory exists
+	// err = f.mkParentDir(f.root)
+	// if err != nil {
+	// 	return errors.Wrap(err, "DirMove mkParentDir dst failed")
+	// }
+
+	// Make sure the source directory exists
+	err = srcFs.mkdir(srcFs.root)
+	if err != nil {
+		return errors.Wrap(err, "DirMove mkdir src failed")
+	}
+
+	// Do the move
+	err = f.sftpClient.Rename(
+		srcFs.root,
+		f.root,
+	)
+	if err != nil {
+		return errors.Wrapf(err, "DirMove Rename(%q,%q) failed", srcFs.root, f.root)
+	}
+	return nil
 }
 
 // Hashes returns fs.HashNone to indicate remote hashing is unavailable
@@ -303,7 +448,7 @@ func (o *Object) String() string {
 	if o == nil {
 		return "<nil>"
 	}
-	return o.fs.url + "/" + o.remote
+	return o.remote
 }
 
 // Remote the name of the remote SFTP file, relative to the fs root
@@ -313,56 +458,76 @@ func (o *Object) Remote() string {
 
 // Hash returns "" since SFTP (in Go or OpenSSH) doesn't support remote calculation of hashes
 func (o *Object) Hash(r fs.HashType) (string, error) {
-	debug(o.fs, "Hash '"+o.remote+"'")
 	return "", fs.ErrHashUnsupported
 }
 
 // Size returns the size in bytes of the remote sftp file
 func (o *Object) Size() int64 {
-	debug(o.fs, "Size '"+o.remote+"'")
 	return o.info.Size()
 }
 
 // ModTime returns the modification time of the remote sftp file
 func (o *Object) ModTime() time.Time {
-	debug(o.fs, "ModTime '"+o.remote+"'")
 	return o.info.ModTime()
 }
 
-// SetModTime sets the modification and access time to the specified time
+// path returns the native path of the object
-func (o *Object) SetModTime(modTime time.Time) error {
+func (o *Object) path() string {
-	debug(o.fs, "SetModTime '"+o.remote+"'")
+	return path.Join(o.fs.root, o.remote)
-	err := o.fs.sftpClient.Chtimes(o.fs.sftpClient.Join(o.fs.root, o.remote), modTime, modTime)
-	if err != nil {
-		return err
 }
-	o.info, err = o.fs.sftpClient.Stat(o.fs.sftpClient.Join(o.fs.root, o.remote))
+
-	return err
+// stat updates the info field in the Object
+func (o *Object) stat() error {
+	info, err := o.fs.sftpClient.Stat(o.path())
+	if err != nil {
+		if os.IsNotExist(err) {
+			return fs.ErrorObjectNotFound
+		}
+		return errors.Wrap(err, "stat failed")
+	}
+	if info.IsDir() {
+		// FIXME this error message doesn't seem right, but it
+		// is necessary for the NewFS code
+		return fs.ErrorObjectNotFound
+	}
+	o.info = info
+	return nil
+}
+
+// SetModTime sets the modification and access time to the specified time
+//
+// it also updates the info field
+func (o *Object) SetModTime(modTime time.Time) error {
+	err := o.fs.sftpClient.Chtimes(o.path(), modTime, modTime)
+	if err != nil {
+		return errors.Wrap(err, "SetModTime failed")
+	}
+	err = o.stat()
+	if err != nil {
+		return errors.Wrap(err, "SetModTime failed")
+	}
+	return nil
 }
 
 // Storable returns whether the remote sftp file is a regular file (not a directory, symbolic link, block device, character device, named pipe, etc)
 func (o *Object) Storable() bool {
-	debug(o.fs, "Storable '"+o.remote+"'?")
 	return o.info.Mode().IsRegular()
 }
 
 // Read from a remote sftp file object reader
 func (file *ObjectReader) Read(p []byte) (n int, err error) {
-	debug(file.object.fs, "Read '"+file.object.remote+"'")
 	n, err = file.sftpFile.Read(p)
 	return n, err
 }
 
 // Close a reader of a remote sftp file
 func (file *ObjectReader) Close() (err error) {
-	debug(file.object.fs, "Close '"+file.object.remote+"'")
 	err = file.sftpFile.Close()
 	return err
 }
 
 // Open a remote sftp file object for reading. Seek is supported
 func (o *Object) Open(options ...fs.OpenOption) (in io.ReadCloser, err error) {
-	debug(o.fs, "Open '"+o.remote+"'")
 	var offset int64
 	offset = 0
 	for _, option := range options {
@@ -375,14 +540,14 @@ func (o *Object) Open(options ...fs.OpenOption) (in io.ReadCloser, err error) {
 			}
 		}
 	}
-	sftpFile, err := o.fs.sftpClient.Open(o.fs.sftpClient.Join(o.fs.root, o.remote))
+	sftpFile, err := o.fs.sftpClient.Open(o.path())
 	if err != nil {
-		return nil, err
+		return nil, errors.Wrap(err, "Open failed")
 	}
 	if offset > 0 {
 		off, err := sftpFile.Seek(offset, 0)
 		if err != nil || off != offset {
-			return nil, err
+			return nil, errors.Wrap(err, "Open Seek failed")
 		}
 	}
 	in = &ObjectReader{
@@ -394,28 +559,45 @@ func (o *Object) Open(options ...fs.OpenOption) (in io.ReadCloser, err error) {
 
 // Update a remote sftp file using the data <in> and ModTime from <src>
 func (o *Object) Update(in io.Reader, src fs.ObjectInfo) error {
-	debug(o.fs, "Update '"+o.remote+"'")
+	file, err := o.fs.sftpClient.Create(o.path())
-	file, err := o.fs.sftpClient.Create(o.fs.sftpClient.Join(o.fs.root, o.remote))
+	if err != nil {
-	if err == nil {
+		return errors.Wrap(err, "Update Create failed")
+	}
+	// remove the file if upload failed
+	remove := func() {
+		removeErr := o.fs.sftpClient.Remove(o.path())
+		if removeErr != nil {
+			fs.Debug(src, "Failed to remove: %v", removeErr)
+		} else {
+			fs.Debug(src, "Removed after failed upload: %v", err)
+		}
+	}
 	_, err = file.ReadFrom(in)
 	if err != nil {
-			return err
+		remove()
+		return errors.Wrap(err, "Update ReadFrom failed")
+	}
+	err = file.Close()
+	if err != nil {
+		remove()
+		return errors.Wrap(err, "Update Close failed")
 	}
 	err = o.SetModTime(src.ModTime())
-		return err
+	if err != nil {
+		return errors.Wrap(err, "Update SetModTime failed")
 	}
-	return err
+	return nil
 }
 
 // Remove a remote sftp file object
 func (o *Object) Remove() error {
-	debug(o.fs, "Remove '"+o.remote+"'")
+	return o.fs.sftpClient.Remove(o.path())
-	return o.fs.sftpClient.Remove(o.fs.sftpClient.Join(o.fs.root, o.remote))
 }
 
 // Check the interfaces are satisfied
 var (
 	_ fs.Fs       = &Fs{}
 	_ fs.Mover    = &Fs{}
+	_ fs.DirMover = &Fs{}
 	_ fs.Object   = &Object{}
 )

sftp/sftp_test.go

@@ -0,0 +1,65 @@
+// Test Sftp filesystem interface
+//
+// Automatically generated - DO NOT EDIT
+// Regenerate with: make gen_tests
+
+// +build !plan9
+
+package sftp_test
+
+import (
+	"testing"
+
+	"github.com/ncw/rclone/fs"
+	"github.com/ncw/rclone/fstest/fstests"
+	"github.com/ncw/rclone/sftp"
+)
+
+func TestSetup(t *testing.T) {
+	fstests.NilObject = fs.Object((*sftp.Object)(nil))
+	fstests.RemoteName = "TestSftp:"
+}
+
+// Generic tests for the Fs
+func TestInit(t *testing.T)                { fstests.TestInit(t) }
+func TestFsString(t *testing.T)            { fstests.TestFsString(t) }
+func TestFsRmdirEmpty(t *testing.T)        { fstests.TestFsRmdirEmpty(t) }
+func TestFsRmdirNotFound(t *testing.T)     { fstests.TestFsRmdirNotFound(t) }
+func TestFsMkdir(t *testing.T)             { fstests.TestFsMkdir(t) }
+func TestFsMkdirRmdirSubdir(t *testing.T)  { fstests.TestFsMkdirRmdirSubdir(t) }
+func TestFsListEmpty(t *testing.T)         { fstests.TestFsListEmpty(t) }
+func TestFsListDirEmpty(t *testing.T)      { fstests.TestFsListDirEmpty(t) }
+func TestFsNewObjectNotFound(t *testing.T) { fstests.TestFsNewObjectNotFound(t) }
+func TestFsPutFile1(t *testing.T)          { fstests.TestFsPutFile1(t) }
+func TestFsPutError(t *testing.T)          { fstests.TestFsPutError(t) }
+func TestFsPutFile2(t *testing.T)          { fstests.TestFsPutFile2(t) }
+func TestFsUpdateFile1(t *testing.T)       { fstests.TestFsUpdateFile1(t) }
+func TestFsListDirFile2(t *testing.T)      { fstests.TestFsListDirFile2(t) }
+func TestFsListDirRoot(t *testing.T)       { fstests.TestFsListDirRoot(t) }
+func TestFsListSubdir(t *testing.T)        { fstests.TestFsListSubdir(t) }
+func TestFsListLevel2(t *testing.T)        { fstests.TestFsListLevel2(t) }
+func TestFsListFile1(t *testing.T)         { fstests.TestFsListFile1(t) }
+func TestFsNewObject(t *testing.T)         { fstests.TestFsNewObject(t) }
+func TestFsListFile1and2(t *testing.T)     { fstests.TestFsListFile1and2(t) }
+func TestFsCopy(t *testing.T)              { fstests.TestFsCopy(t) }
+func TestFsMove(t *testing.T)              { fstests.TestFsMove(t) }
+func TestFsDirMove(t *testing.T)           { fstests.TestFsDirMove(t) }
+func TestFsRmdirFull(t *testing.T)         { fstests.TestFsRmdirFull(t) }
+func TestFsPrecision(t *testing.T)         { fstests.TestFsPrecision(t) }
+func TestObjectString(t *testing.T)        { fstests.TestObjectString(t) }
+func TestObjectFs(t *testing.T)            { fstests.TestObjectFs(t) }
+func TestObjectRemote(t *testing.T)        { fstests.TestObjectRemote(t) }
+func TestObjectHashes(t *testing.T)        { fstests.TestObjectHashes(t) }
+func TestObjectModTime(t *testing.T)       { fstests.TestObjectModTime(t) }
+func TestObjectMimeType(t *testing.T)      { fstests.TestObjectMimeType(t) }
+func TestObjectSetModTime(t *testing.T)    { fstests.TestObjectSetModTime(t) }
+func TestObjectSize(t *testing.T)          { fstests.TestObjectSize(t) }
+func TestObjectOpen(t *testing.T)          { fstests.TestObjectOpen(t) }
+func TestObjectOpenSeek(t *testing.T)      { fstests.TestObjectOpenSeek(t) }
+func TestObjectUpdate(t *testing.T)        { fstests.TestObjectUpdate(t) }
+func TestObjectStorable(t *testing.T)      { fstests.TestObjectStorable(t) }
+func TestFsIsFile(t *testing.T)            { fstests.TestFsIsFile(t) }
+func TestFsIsFileNotFound(t *testing.T)    { fstests.TestFsIsFileNotFound(t) }
+func TestObjectRemove(t *testing.T)        { fstests.TestObjectRemove(t) }
+func TestObjectPurge(t *testing.T)         { fstests.TestObjectPurge(t) }
+func TestFinalise(t *testing.T)            { fstests.TestFinalise(t) }

sftp/sftp_unsupported.go

@@ -0,0 +1,6 @@
+// Build for sftp for unsupported platforms to stop go complaining
+// about "no buildable Go source files "
+
+// +build plan9
+
+package sftp