TrueCloudLab/rclone
16
0
Fork 2
Code Issues 1 Pull requests 1 Projects Releases Packages Wiki Activity Actions 1
4933 commits 181 branches 114 tags 335 MiB
Commit graph

986 commits

Author SHA1 Message Date
Nick Craig-Wood
bcbe393af3 sftp: implement Shutdown method 2020-11-27 17:35:01 +00:00
Nick Craig-Wood
47aada16a0 fs: add Shutdown optional method for backends 2020-11-27 17:35:01 +00:00
Nick Craig-Wood
dfadd98969 azureblob,memory,pcloud: fix setting of mime types 
Before this change the backend was reading the mime type of the
destination object instead of the source object when uploading.

This changes fixes the problem and introduces an integration test for
it.

See: https://forum.rclone.org/t/is-there-a-way-to-get-rclone-copy-to-preserve-metadata/20682/2
 2020-11-27 14:40:05 +00:00
Nick Craig-Wood
9d574c0d63 fshttp: read config from ctx not passed in ConfigInfo #4685 2020-11-26 16:40:12 +00:00
Nick Craig-Wood
2e21c58e6a fs: deglobalise the config #4685 
This is done by making fs.Config private and attaching it to the
context instead.

The Config should be obtained with fs.GetConfig and fs.AddConfig
should be used to get a new mutable config that can be changed.
 2020-11-26 16:40:12 +00:00
Nick Craig-Wood
979bb07c86 filefabric: Implement the Enterprise File Fabric backend 
Missing features
- M-Stream support
- Oauth-like flow (soon being changed to oauth)
 2020-11-25 21:11:29 +00:00
Nick Craig-Wood
dfeae0e70a Revert "sharefile: fix backend due to API swapping integers for strings" 
The API seems to have reverted to what it was before

This reverts commit 095c7bd801.
 2020-11-25 20:52:57 +00:00
Nick Craig-Wood
f43a9ac17e pcloud: only use SHA1 hashes in EU region 
Apparently only SHA1 hashes are supported in the EU region for
pcloud. This has been confirmed by pCloud support. The EU regions also
support SHA256 hashes which we don't support yet.

https://forum.rclone.org/t/pcloud-to-local-no-hashes-in-common/19440
 2020-11-25 20:46:38 +00:00
Nick Craig-Wood
76ee3060d1 s3: Add MD5 metadata to objects uploaded with SSE-AWS/SSE-C 
Before this change, small objects uploaded with SSE-AWS/SSE-C would
not have MD5 sums.

This change adds metadata for these objects in the same way that the
metadata is stored for multipart uploaded objects.

See: #1824 #2827
 2020-11-25 12:28:02 +00:00
Nick Craig-Wood
4bb241c435 s3: store md5 in the Object rather than the ETag 
This enables us to set the md5 to cache it.

See: #1824 #2827
 2020-11-25 12:28:02 +00:00
Nick Craig-Wood
a06f4c2514 s3: fix hashes on small files with aws:kms and sse-c 
If rclone is configured for server side encryption - either aws:kms or
sse-c (but not sse-s3) then don't treat the ETags returned on objects
as MD5 hashes.

This fixes being able to upload small files.

Fixes #1824
 2020-11-25 12:28:02 +00:00
Nick Craig-Wood
53aa03cc44 s3: complete sse-c implementation 
This now can complete all operations with SSE-C enabled.

Fixes #2827
See: https://forum.rclone.org/t/issues-with-aws-s3-sse-c-getting-strange-log-entries-and-errors/20553
 2020-11-25 12:28:02 +00:00
Nick Craig-Wood
f0905499e3 random: seed math/rand in one place with crypto strong seed #4783 
This shouldn't be read as encouraging the use of math/rand instead of
crypto/rand in security sensitive contexts, rather as a safer default
if that does happen by accident.
 2020-11-18 17:48:44 +00:00
Nick Craig-Wood
095c7bd801 sharefile: fix backend due to API swapping integers for strings 
For some reason the API started returning some integers as strings in
JSON. This is probably OK in Javascript but it upsets Go.

This is easily fixed with the `json:"name,size"` struct tag.
 2020-11-13 14:37:43 +00:00
Nick Craig-Wood
23469c9c7c ftp: add --ftp-disable-msld option to ignore MLSD for really old servers 
This is useful for servers which advertise MLSD (eg some versions of
Serv-U) but don't support it properly.

See: https://forum.rclone.org/t/double-folder-names-on-target-destination-paths-ftp/18822
See: https://github.com/jlaffaye/ftp/pull/196
 2020-11-13 11:25:34 +00:00
buengese
636fb5344a drive: implement CleanUp workaround for team drives - fixes #2418 2020-11-13 03:30:28 +01:00
buengese
bc4282e49e compress: added experimental compression remote - implements #2098, #1356, #675 
This remote implements transparent compression using gzip. Uses JSON as a for storing metadata.

Co-authored-by: id01 <gaviniboom@gmail.com>
 2020-11-13 02:31:59 +01:00
Manish Gupta
95d0410baa
local: continue listing files/folders when a circular symlink is detected 
Before this change a circular symlink would cause rclone to error out from the listings.

After this change rclone will skip a circular symlink and carry on the listing,
producing an error at the end.

Fixes #4743
 2020-11-12 11:32:55 +00:00
Nick Craig-Wood
f7efce594b config: add context.Context #3257 #4685 
This add config to the Config callback in the backends and the related
config functions.
 2020-11-09 18:05:54 +00:00
Nick Craig-Wood
1fb6ad700f accounting: add context.Context #3257 #4685 2020-11-09 18:05:54 +00:00
Nick Craig-Wood
8b96933e58 fs: Add context to fs.Features.Fill & fs.Features.Mask #3257 #4685 2020-11-09 18:05:54 +00:00
Nick Craig-Wood
d846210978 fs: Add context to NewFs #3257 #4685 
This adds a context.Context parameter to NewFs and related calls.

This is necessary as part of reading config from the context -
backends need to be able to read the global config.
 2020-11-09 18:05:54 +00:00
Nick Craig-Wood
bedf6e90d2 onedrive: warn on gateway timeout errors 
It seems that when doing chunked uploads to onedrive, if the chunks
take more than 3 minutes or so to upload then they may timeout with
error 504 Gateway Timeout.

This change produces an error (just once) suggesting lowering
`--onedrive-chunk-size` or decreasing `--transfers`.

This is easy to replicate with:

    rclone copy -Pvv --bwlimit 0.05M 20M onedrive:20M

See: https://forum.rclone.org/t/default-onedrive-chunk-size-does-not-work/20010/
 2020-11-02 16:53:35 +00:00
Nick Craig-Wood
1973fc1ecc azureblob: update lib from v0.10.0 to v0.11.0 and fix API breakage 
See: https://github.com/Azure/azure-storage-blob-go/issues/226
 2020-10-29 13:34:39 +00:00
Josh Soref
0a6196716c docs: style: avoid double-nesting parens 
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
 2020-10-28 18:16:23 +00:00
Josh Soref
a15f50254a docs: grammar: if, then 
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
 2020-10-28 18:16:23 +00:00
Josh Soref
5d4f77a022 docs: grammar: Oxford comma 
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
 2020-10-28 18:16:23 +00:00
Josh Soref
a089de0964 docs: grammar: uncountable: links 
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
 2020-10-28 18:16:23 +00:00
Josh Soref
3068ae8447 docs: grammar: count agreement: files 
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
 2020-10-28 18:16:23 +00:00
Josh Soref
67ff153b0c docs: grammar: article: a-file 
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
 2020-10-28 18:16:23 +00:00
Josh Soref
3e1cb8302a docs: spelling: etc. 
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
 2020-10-28 18:16:23 +00:00
Josh Soref
e4a87f772f docs: spelling: e.g. 
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
 2020-10-28 18:16:23 +00:00
Josh Soref
d4f38d45a5 docs: spelling: high-speed 
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
 2020-10-28 18:16:23 +00:00
Josh Soref
bbe7eb35f1 docs: spelling: server-side 
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
 2020-10-28 18:16:23 +00:00
edwardxml
87e54f2dde
ftp: update wording for flags 
Minor wording change to help for explicit and implicit FTPS flags. More consistent between flags. Add 's' to request because only one 'client' mentioned.
 2020-10-28 15:45:52 +00:00
Ivan Andreev
be6a888e50 chunker: skip long local hashing, hash in-transit (fixes #4021) 
PR 4614
 2020-10-26 20:18:07 +03:00
Ivan Andreev
dad8447423 mailru: avoid prehashing of large local files 
PR 4617
 2020-10-26 20:16:52 +03:00
Ivan Andreev
65ff109065 mailru: accept special folders eg camera-upload 
Fixes #4025
PR 4690
 2020-10-26 20:04:31 +03:00
Nick Craig-Wood
cf0bdad5de union: create root directories if none exist 
This fixes the TestUnion: integration test if the /tmp/union[123] dirs
don't exist.
 2020-10-25 18:10:49 +00:00
albertony
ffdd0719e7 jottacloud: avoid double url escaping of device/mountpoint - fixes #4697 2020-10-20 17:43:49 +02:00
Anagh Kumar Baranwal
5b09599a23 drive: Added flag --drive-stop-on-download-limit to stop transfers when the download limit is exceeded 
Signed-off-by: Anagh Kumar Baranwal <6824881+darthShadow@users.noreply.github.com>
 2020-10-14 15:51:01 +01:00
Josh Soref
d0888edc0a Spelling fixes 
Fix spelling of: above, already, anonymous, associated,
authentication, bandwidth, because, between, blocks, calculate,
candidates, cautious, changelog, cleaner, clipboard, command,
completely, concurrently, considered, constructs, corrupt, current,
daemon, dependencies, deprecated, directory, dispatcher, download,
eligible, ellipsis, encrypter, endpoint, entrieslist, essentially,
existing writers, existing, expires, filesystem, flushing, frequently,
hierarchy, however, implementation, implements, inaccurate,
individually, insensitive, longer, maximum, metadata, modified,
multipart, namedirfirst, nextcloud, obscured, opened, optional,
owncloud, pacific, passphrase, password, permanently, persimmon,
positive, potato, protocol, quota, receiving, recommends, referring,
requires, revisited, satisfied, satisfies, satisfy, semver,
serialized, session, storage, strategies, stringlist, successful,
supported, surprise, temporarily, temporary, transactions, unneeded,
update, uploads, wrapped

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
 2020-10-14 15:21:31 +01:00
Anagh Kumar Baranwal
fc5b14b620
s3: Added --s3-disable-http2 to disable http/2 
Fixes #4673

Signed-off-by: Anagh Kumar Baranwal <6824881+darthShadow@users.noreply.github.com>
 2020-10-13 17:11:22 +01:00
Stephen Harris
bbddadbd04 sftp: remember entered password in AskPass mode 
As reported in

  https://github.com/rclone/rclone/issues/4660#issuecomment-705502792

After switching to a password callback function, if the ssh connection
aborts and needs to be reconnected then the user is-reprompted for their
password.  Instead we now remember the password they entered and just give
that back.  We do lose the ability for them to correct mistakes, but that's
the situation from before switching to callbacks.  We keep the benefits
of not asking for passwords until the SSH connection succeeds (right
known_hosts entry, for example).

This required a small refactor of how `f := &Fs{}` was built, so we can
store the saved password in the Fs object
 2020-10-13 16:53:11 +01:00
Nick Craig-Wood
7428e47ebc local: fix sizes and syncing with --links option on Windows - fixes #4581 
Before this change rclone returned the size from the Stat call of the
link. On Windows this reads as 0 always, however on unix it reads as
the length of the text in the link. This caused errors like this when
syncing:

    Failed to copy: corrupted on transfer: sizes differ 0 vs 13

This change causes Windows platforms to read the link and use that as
the size of the link instead of 0 which fixes the problem.
 2020-10-13 16:29:56 +01:00
Dan Hipschman
70f92fd6b3 crypt: small simplification, no functionality change 2020-10-12 17:20:39 +01:00
Nick Craig-Wood
0906f8dd3b onedrive: fix disk usage for sharepoint 
Some onedrive sharepoints appear to return all 0s for quota

    "quota":{"deleted":0,"remaining":0,"total":0,"used":0}

This commit detects this and returns unknown for all quota parts.

See: https://forum.rclone.org/t/zero-size-volume-when-mounting-onedrive-sharepoint/19597
 2020-10-09 14:11:56 +01:00
buengese
664213cedb jottacloud: remove clientSecret from config when upgrading to token based authentication - #4645 2020-10-08 11:51:17 +02:00
Stephen Harris
9e925becb6 sftp: defer asking for user passwords until the SSH connection succeeds 
Issue: 4660
    https://github.com/rclone/rclone/issues/4660

Unexpected side effect: a wrong password allows for the user to retry!
 2020-10-07 12:01:17 +01:00
Anagh Kumar Baranwal
e3a5bb9b48 s3: Add missing regions for AWS 
Signed-off-by: Anagh Kumar Baranwal <6824881+darthShadow@users.noreply.github.com>
 2020-10-06 16:54:42 +01:00