Commit graph

173 commits

Author SHA1 Message Date
themylogin
7d2861ead6 Adjust S3 upload concurrency with --s3-upload-concurrency 2018-06-14 16:15:17 +01:00
remusb
aaa8591661 cache: add non cached dirs on notifications - #2155 2018-06-13 23:57:26 +03:00
remusb
4df1794932 cache: fix panic when running without plex configs 2018-06-13 15:06:14 +03:00
Nick Craig-Wood
d18928962c dropbox: make dropbox for business folders accessible #2003
Paths prefixed with / on a dropbox for business plan will now start at
the root instead of the users home directory.
2018-06-13 11:03:34 +01:00
remusb
339fbf0df5 cache: reconnect plex websocket on failures 2018-06-12 22:58:15 +03:00
remusb
13ccb39819 cache: allow root to be expired from rc - #2237 2018-06-12 22:19:03 +03:00
remusb
f9a1a7e700 cache: fix root folder caching 2018-06-10 21:54:20 +03:00
Nick Craig-Wood
4d793b8ee8 drive: remove part of workaround for #1675
Now that https://issuetracker.google.com/issues/64468406 has been
fixed, we can remove part of the workaround which fixed #1675 -
019adc35609c2136

This will make queries marginally more efficient.  We still need the
other part of the workaround since the `=` operator is case
insensitive.
2018-06-10 15:28:32 +01:00
Nick Craig-Wood
9289aead9b drive: Add --drive-acknowledge-abuse to download flagged files - fixes #2317
Also if rclone gets the cannotDownloadAbusiveFile suggest using the
--drive-acknowledge-abuse flag.
2018-06-10 15:25:21 +01:00
Nick Craig-Wood
1053d7e123 local: fix symlink/junction point directory handling under Windows
Before this commit rclone's handling of symlinks and junction points
under Windows was broken.  rclone treated them as files and attempted
to transfer them which gave the error "The handle is invalid".

Ultimately the cause of this was 3e43ff7414 which was a
workaround so files with reparse points (which are a kind of symlink)
would transfer correctly.

The solution implemented is to revert the above commit which will mean
that #614 will break again.  However there is now a work-around (which
will be signaled by rclone) to use the -L flag which wasn't available
when the original commit was made.

Fixes #2336
2018-06-10 12:25:03 +01:00
Nick Craig-Wood
017297af70 s3: Fix --s3-chunk-size which was always using the minimum - fixes #2345 2018-06-10 12:22:30 +01:00
remusb
4e8e5fed7d cache: clean remaining empty folders from temp upload path 2018-06-09 14:52:31 +03:00
remusb
c0f772bc14 cache: update internal tests and small fixes 2018-06-08 23:34:38 +03:00
Nick Craig-Wood
05edb5f501 drive: Fix change list polling with team drives - fixes #2330
In the drive v3 conversion we forgot the IncludeTeamDriveItems
parameter when calling the changes API.  Adding it fixes the changes
polling with team drives.
2018-06-07 11:35:55 +01:00
Henning Surmeier
f1269dc06a onedrive: errorHandler for business requests
This implementation hopefully can handle all error requests from the
onedrive for business authentication.
I have only tested it with the "domain in unmanaged state" error.
2018-06-06 09:54:21 +01:00
remusb
8a84975993 cache: cache lists using batch writes 2018-06-04 21:04:45 +03:00
Nick Craig-Wood
d4213c0ac5 sftp: Fix slow downloads for long latency connections - fixes #1158
This was caused by using the sftp.File.Read method which resets the
streaming window after each call.  Replacing it with sftp.File.WriteTo
and an io.Pipe fixes the problem bringing the speed to the same as the
sftp binary.
2018-05-24 15:10:28 +01:00
John Clayton
71b4f1ccab cache: use secure websockets for HTTPS Plex addresses 2018-05-22 11:47:57 +01:00
Nick Craig-Wood
e5ff375948 Use config.FileGet instead of fs.ConfigFileGet 2018-05-22 09:43:24 +01:00
Nick Craig-Wood
512f4b4487 Update error checking on fmt.Fprint* after errcheck update
Now we need to check or ignore errors on fmt.Fprint* explicitly -
previously errcheck just ignored them for us.
2018-05-22 09:41:13 +01:00
Nick Craig-Wood
9697754707 drive: Don't attempt to choose Team Drives when using rclone config create 2018-05-16 09:10:09 +01:00
Nick Craig-Wood
e52ecba295 crypt: check the crypted hash of files when uploading #2303
This checks the checksum of the streamed encrypted data against the
checksum of the encrypted object returned from the remote and returns
an error if it is different.
2018-05-15 14:50:36 +01:00
Nick Craig-Wood
826975c341 fs: add Optional ID() method to Object and implement it in backends
ID() shows the internal ID of the Object if available.
2018-05-13 12:17:55 +01:00
Nick Craig-Wood
cdde8fa75a opendrive: finish off #1026
* Fix errcheck and golint warnings
  * Remove unused constants and fix comments
  * Parse error responses properly
  * Fix Open with RangeOption
  * Fix Move, Copy and DirMove
  * Implement DirCacheFlush
  * Check interfaces are correct
  * Remove debugs and update overview
  * Correct feature flags
  * Pare replacement characters down to the minimum set
  * Add to the integration tests
2018-05-12 10:10:46 +01:00
Jakub Karlicek
53292527bb opendrive: fill out the functionality #1026
* Add Mkdir, Rmdir, Purge, Delete, SetModTime, Copy, Move, DirMove
 * Update file size after upload
 * Add Open seek
 * Set private permission for new folder and uploaded file
 * Add docs
 * Update List function
 * Fix UserSessionInfo struct
 * Fix socket leaks
 * Don’t close resp.Body in Open method
 * Get hash when listing files
2018-05-12 10:07:25 +01:00
Nick Craig-Wood
63f413f477 webdav: show all available information when printing errors 2018-05-11 08:43:53 +01:00
Nick Craig-Wood
d85b9bc9d6 webdav: workarounds for biz.mail.ru
* Add "Depth: 1" on read metadata PROPFIND call
  * Accept 406 to mean directory already exists
2018-05-11 08:43:53 +01:00
Nick Craig-Wood
b07e51cf73 webdav: read the body of messages into the error if XML parse fails 2018-05-11 08:43:53 +01:00
Nick Craig-Wood
f073db81b1 drive: add --drive-alternate-export to fix large doc export - fixes #2243
The official drive APIs seem to have trouble downloading large
documents sometimes.

This commit adds a --drive-alternate-export flag to use an different,
unofficial set of export URLS which seem to download large files OK.
2018-05-10 10:04:39 +01:00
Nick Craig-Wood
9698a2babb gcs: low level retry all operations if necessary
Google cloud storage doesn't normally need retries, however certain
things (eg bucket creation and removal) are rate limited and do
generate 429 errors.

Before this change the integration tests would regularly blow up with
errors from GCS rate limiting bucket creation and removal.

After this change we low level retry all operations using the same
exponential backoff strategy as used in the google drive backend.
2018-05-10 09:24:09 +01:00
Nick Craig-Wood
aa5c5ec5d3 build: mask linter errors we can't fix 2018-05-05 17:32:41 +01:00
Nick Craig-Wood
e80ae4e09c build: remove unused struct fields spotted by structcheck 2018-05-05 17:32:41 +01:00
Nick Craig-Wood
1320e84bc2 build: remove unused code spotted by the deadcode linter 2018-05-05 17:32:41 +01:00
Nick Craig-Wood
cb5bd47e61 build: fix errors spotted by ineffassign linter
These were mostly caused by shadowing err and a good fraction of them
will have caused errors not to be propagated properly.
2018-05-05 17:32:41 +01:00
Nick Craig-Wood
7ea68f1fc6 sftp: require go1.9+ after golang.org/x/crypto/ssh update 2018-05-05 16:23:47 +01:00
Fabian Möller
3b5e70c8c6 drive: fix ChangeNotify for folders 2018-05-05 12:54:03 +01:00
Nick Craig-Wood
fe25cb9c54 drive: fix about (and df on a mount) for team drives - fixes #2288
Before this fix team drives would return the drive quota which is
incorrect and mis-leading.

Team drives don't appear to have an API for reading the bytes used or
the quota so we now return that the quota and usage are unknown.
2018-05-03 08:59:14 +01:00
Rodrigo
b7d2048032 WebDAV: Ignore Reason-Phrase in status line #2281 2018-04-30 17:36:38 +01:00
Piotr Oleszczyk
76f2cbeb94 sftp: Add --ssh-path-override flag #1474
The flag allows calculation of checksums on systems using
different paths for SSH and SFTP, like synology NAS boxes.
2018-04-30 17:05:10 +01:00
Nick Craig-Wood
a3bf6b9c2c drive, gcs: fix service account authentication - fixes #2279
This fixes a problem introduced in b78af517de where it would
attempt to read a non-existent service account file.
2018-04-28 09:33:43 +01:00
Matt Holt
b78af517de Add service_account_credentials for Google Cloud and Drive 2018-04-27 16:07:37 +01:00
Nick Craig-Wood
ae562b5a4f ftp: more workarounds for FTP servers to fix mkParentDir - fixes #2181 2018-04-26 14:58:04 +01:00
Nick Craig-Wood
c01177bc28 ftp: work around strange response from box FTP server
The Box FTP server seems to send 450 instead of 550 - work around that.

See: https://forum.rclone.org/t/using-box-com-over-ftp-problems/5313
2018-04-26 14:58:04 +01:00
Nick Craig-Wood
9ab2521ef2 rc: autogenerate and tidy the docs and commands
* Rename rc/pid -> core/pid
  * Sort the output of `rc list`
  * Make a script to autogenerate the docs
  * Tidy docs
2018-04-23 20:57:17 +01:00
Nick Craig-Wood
870c58f7f8 sftp: fail soft with a debug on hash failure #1474
If md5sum/sha1sum fails we debug what it outputed on stderr and return
an empty hash indicating we didn't have a hash, rather than
hash.ErrUnsupported indicating that we don't support this hash type.

This fixes lots of ERROR messages for sftp and synology NAS which,
while it supports md5sum the SFTP paths and the SSH paths are
different so md5sum doesn't work.

We also stop disabling md5sum/sha1sum on errors since typically Hashes
is only checked at the start of a sync run and isn't expected to
change dynamically.
2018-04-21 09:02:53 +01:00
Nick Craig-Wood
311a962011 s3: Look in S3 named profile files for credentials - fixes #2243 2018-04-21 09:00:20 +01:00
Nick Craig-Wood
da7a77ef2e ftp: Fix no error on listing non-existent directory 2018-04-20 23:22:46 +01:00
hensur
8fe3037301 webdav: support SharePoint cookie authentication
This enables the use of the SharePoint webdav endpoint provided by
OneDrive for Business or Office365 Education Accounts. It enables
unverified accounts to be accessed with rclone via webdav as it isn't
possible through the normal onedrive backend.

This integrates the https://github.com/hensur/onedrive-cookie-test
package to fetch the required cookies to authorize against the
SharePoint webdav endpoint.
2018-04-20 21:43:54 +01:00
Nick Craig-Wood
dc59836021 webdav: strip leading and trailing / off root - fixes #2257 2018-04-20 21:43:54 +01:00
Nick Craig-Wood
1a3fb21a77 onedrive: add QuickXorHash support for OneDrive for business - fixes #2262 2018-04-20 21:03:03 +01:00