0537791d14
Betweeen rclone v1.54 and v1.55 there was an approx 3x performance regression when transferring to distant SFTP servers (in particular rsync.net). This turned out to be due to the library github.com/pkg/sftp rclone uses. Concurrent writes used to be enabled in this library by default (for v1.12.0 as used in rclone v1.54) but they are no longer enabled (for v1.13.0 as used in rclone v1.55) for safety reasons and it is necessary to enable them specifically. The safety concerns are due to the uncertainty as to whether writes come in order and whether a half completed file might have holes in it. This isn't a problem for rclone since a) it doesn't restart uploads and b) it has a post-transfer checksum test. This change introduces a new flag `--sftp-disable-concurrent-writes` to control the feature which defaults to false, meaning that concurrent writes are enabled as in v1.54. However this isn't quite enough to fix the problem as the sftp library needs to be able to sniff the size of the stream from the reader passed in, so this also adds a `Size` interface to the reader to enable this. This involved a patch to the library. The library was reverted to v1.12.0 for v1.55.1 - this patch installs v1.13.0+master to fix the Size interface problem. See: https://github.com/pkg/sftp/issues/426 |
||
|---|---|---|
| .. | ||
| sftp.go | ||
| sftp_internal_test.go | ||
| sftp_test.go | ||
| sftp_unsupported.go | ||
| stringlock.go | ||
| stringlock_test.go | ||