Commit graph

5395 commits

Author SHA1 Message Date
Nick Craig-Wood
c81311722e ftp: close idle connections after --ftp-idle-timeout (1m by default)
This fixes a problem where ftp backends live on forever when using
the rc and use more and more connections.
2021-02-16 12:39:05 +00:00
Nick Craig-Wood
843ddd9136 ftp: implement Shutdown method 2021-02-16 12:39:05 +00:00
Nick Craig-Wood
a3fcadddc8 sftp: close idle connections after --sftp-idle-timeout (1m by default)
This fixes a problem where sftp backends live on forever when using
the rc and use more and more connections.

Fixes #4883
2021-02-16 12:39:05 +00:00
Nick Craig-Wood
a63e1f1383 Add Miron Veryanskiy to contributors 2021-02-16 12:39:05 +00:00
Nick Craig-Wood
5b84adf3b9 test: add "rclone test histogram" for file name distribution stats 2021-02-13 14:24:43 +00:00
Nick Craig-Wood
f890965020 test: add makefiles test command (converted from script) 2021-02-13 14:24:43 +00:00
Nick Craig-Wood
f88a5542cf test: move test commands under "rclone test" and make them visible 2021-02-13 14:24:43 +00:00
Miron Veryanskiy
fd94b3a473 docs: replace #file-caching with #vfs-file-caching
The documentation had dead links pointing to #file-caching. They've been
moved to point to #vfs-file-caching.
2021-02-13 12:56:25 +00:00
Nick Craig-Wood
2aebeb6061 accounting: fix --bwlimit when up or down is off - fixes #5019
Before this change the core bandwidth limit was limited to upload or
download value if the other value was off.

This fix only applies a core bandwidth limit when both values are set.
2021-02-13 12:45:12 +00:00
Nick Craig-Wood
e779cacc82 fshttp: fix bandwidth limiting after bad merge
Reapply missing bwlimiting which was inserted in

0a932dc1f2 Add --bwlimit for upload and download #1873

But accidentally removed when merging

edfe183ba2 fshttp: add DSCP support with --dscp for QoS with differentiated services
2021-02-13 12:45:12 +00:00
Nick Craig-Wood
37e630178e dropbox: add scopes to oauth request and optionally "members.read"
This change adds the scopes rclone wants during the oauth request.
Previously rclone left these blank to get a default set.

This allows rclone to add the "members.read" scope which is necessary
for "impersonate" to work, but only when it is in use as it require
authorisation from a Team Admin.

See: https://forum.rclone.org/t/dropbox-no-members-read/22223/3
2021-02-13 12:35:24 +00:00
Nick Craig-Wood
2cdc071b85 Add Ankur Gupta to contributors 2021-02-13 12:35:24 +00:00
Nick Craig-Wood
496e32fd8a Add cynthia kwok to contributors 2021-02-13 12:35:24 +00:00
Nick Craig-Wood
bf3ba50a0f Add David Sze to contributors 2021-02-13 12:35:24 +00:00
Nick Craig-Wood
22c226b152 Add Alexey Tabakman to contributors 2021-02-13 12:35:23 +00:00
Klaus Post
5ca7f1fe87 encoder/filename: Wrap scsu package 2021-02-12 11:39:39 +00:00
Klaus Post
f14220ef1e encoder/filename: Add 2 more tables and tests. 2021-02-12 11:39:39 +00:00
Klaus Post
424aaac2e1 encoder/filename: Add SCSU as tables
Instead of only adding SCSU, add it as an existing table.

Allow direct SCSU and add a, perhaps, reasonable table as well.

Add byte interfaces that doesn't base64 encode the URL as well with `EncodeBytes` and `DecodeBytes`.

Fuzz tested and decode tests added.
2021-02-12 11:39:39 +00:00
Ankur Gupta
47b69d6300 operations: Made copy and sync operations obey a RetryAfterError 2021-02-11 17:47:34 +00:00
cynthia kwok
c0c2505977
build: add an rclone user to the Docker image but don't use it by default
partially addresses #4831

Co-authored-by: cynful <cynful@users.noreply.github.com>
2021-02-11 17:45:44 +00:00
David Sze
2d7afe8690 local: Add flag --no-preallocate - #3207
Some virtual filesystems (such as Google Drive File Stream) may
incorrectly set the actual file size equal to the preallocated space,
causing checksum and file size checks to fail.

This flag can be used to disable preallocation for local backends of
this type.
2021-02-11 17:25:28 +00:00
Nick Craig-Wood
92187a3b33 cmount: fix unicode issues with accented characters on macOS
This adds

    -o modules=iconv,from_code=UTF-8,to_code=UTF-8-MAC

To the mount options if it isn't already present which fixes mounting
issues on macOS with accented characters in the finder.
2021-02-11 15:13:19 +00:00
Nick Craig-Wood
53aa4b87fd b2: fix failed to create file system with application key limited to a prefix
Before this change, if an application key limited to a prefix was in
use, with trailing `/` marking the folders then rclone would HEAD the
path without a trailing `/` to work out if it was a file or a folder.
This returned a permission denied error, which rclone returned to the
user.

    Failed to create file system for "b2:bucket/path/":
        failed to HEAD for download: Unknown 401  (401 unknown)

This change assumes any errors on HEAD will make rclone assume the
object does not exist and the path is referring to a directory.

See: https://forum.rclone.org/t/b2-error-on-application-key-limited-to-a-prefix/22159/
2021-02-11 15:13:19 +00:00
Max Sum
edfe183ba2 fshttp: add DSCP support with --dscp for QoS with differentiated services 2021-02-10 18:29:18 +00:00
edwardxml
dfc63eb8f1
docs: update filtering docs
Typos from prior major rewrite
2021-02-10 18:21:41 +00:00
edwardxml
f21f2529a3
docs: fix nesting of brackets and backticks in ftp docs 2021-02-10 18:18:01 +00:00
edwardxml
1efb543ad8
docs: add a Windows example to the filtering docs
Add an example pinched from rclone forum

https://forum.rclone.org/t/need-help-to-understand-filtering-commands/22196

Credit to @asdffdsa
2021-02-10 18:09:48 +00:00
edwardxml
92e36fcfc5
docs: update filtering time formats
Correction per @x0b from 
https://forum.rclone.org/t/max-age-min-age-rfc3339-format-rejected/22204
2021-02-10 18:08:25 +00:00
Alexey Tabakman
bf8542c670
docs: update information about Disk-O: desktop client #4988 (#4988) 2021-02-09 21:23:45 +03:00
albertony
cc5a1e90d8 mount: improved handling of relative paths on windows 2021-02-08 20:55:23 +00:00
albertony
b39fa54ab2 mount: allow mounting to root directory on windows 2021-02-08 20:55:23 +00:00
Nick Craig-Wood
f1147fe1dd rc: sync,copy,move: document createEmptySrcDirs parameter - fixes #4489 2021-02-08 12:25:40 +00:00
Nick Craig-Wood
8897377a54 filter: Make --exclude "dir/" equivalent to --exclude "dir/**"
Rclone uses directory exclusions to cut down the listing it has to do,
so before this fix `--exclude dir/` would make sure nothing in `dir/`
was scanned, **except** if --fast-list was used, in which case only
the directory was excluded and everything within it was included.

This is rather unexpected, so this patch makes `--exclude dir/` be
equivalent to `--exclude dir/**`, meaning that excluding a directory
excludes it and its contents.

We can't do the same for --include without changing the semantics of
filtering slightly.

Fixes #3375
2021-02-07 17:29:16 +00:00
Nick Craig-Wood
f50b4e51ed build: make a macOS ARM64 build to support Apple Silicon - Fixes #4786
- add `-macos-sdk` and `-macos-arch` to adjust CGO_CFLAGS and CGO_LDFLAGS
    - select macOS SDK 11.1 and arch arm64 when building
- add -cgo-cflags and -cgo-ldflags to set CGO_CFLAGS and CGO_LDFLAGS
    - add back /usr/local to pickup fuse headers and library
- add `-env` to cross-compile
- add macOS/arm64 to download matrix
2021-02-07 14:59:53 +00:00
Nick Craig-Wood
f135acbdfb build: install macfuse 4.x instead of osxfuse 3.x
The macfuse has been renamed, but brew is still picking up the old
version under the old name.

This corrects the name to macfuse which brings in v4.x which should
support Apple Silicon.
2021-02-07 14:59:53 +00:00
Nick Craig-Wood
cdd99a6f39 fs/accounting: fix occasionally failing test on macOS 2021-02-07 14:59:53 +00:00
Nick Craig-Wood
6ecb5794bc rc: add _config parameter to set global config for just this rc call 2021-02-07 14:56:41 +00:00
Nick Craig-Wood
9a21aff4ed rc: add options/local to see the options configured in the context 2021-02-07 14:56:41 +00:00
Nick Craig-Wood
8574a7bd67 rc: factor async/sync job handing into rc/jobs from rc/rcserver
This fixes async jobs with `rclone rc --loopback` which isn't very
important but sets the stage for _config setting.
2021-02-07 14:56:41 +00:00
Nick Craig-Wood
a0fc10e41a rc: factor out duplicate code in job creation 2021-02-07 14:56:41 +00:00
Nick Craig-Wood
ae3963e4b4 fs: Add string alternatives for setting options over the rc
Before this change options were read and set in native format. This
means for example nanoseconds for durations or an integer for
enumerated types, which isn't very convenient for humans.

This change enables these types to be set with a string with the
syntax as used in the command line instead, so `"10s"` rather than
`10000000000` or `"DEBUG"` rather than `8` for log level.
2021-02-07 14:56:41 +00:00
Nick Craig-Wood
e32f08f37b drive: refer to Shared Drives instead of Team Drives 2021-02-07 12:30:21 +00:00
Nick Craig-Wood
fea4b753b2 Add Alex JOST to contributors 2021-02-07 12:30:21 +00:00
Nick Craig-Wood
b2b5b7598c build: ensure go1.16 go gets the correct versions of tools
go1.16 uses GO111MODULE=on by default which meant we were picking up
v1 of nfpm instead of the intended v2.
2021-02-04 17:34:30 +00:00
Nick Craig-Wood
5f943aabc8 build: Use GO386=softfloat instead of deprecated GO386=387 for 386 builds 2021-02-04 16:37:23 +00:00
Nick Craig-Wood
84c785bc36 build: use go1.16-rc1 as default compiler 2021-02-04 16:08:51 +00:00
Nick Craig-Wood
993146375e build: update all dependencies 2021-02-03 21:34:38 +00:00
Nick Craig-Wood
bbe791a886 swift: update github.com/ncw/swift to v2.0.0
The update to v2 of the swift library introduces a context parameter
to each function. This required a lot of mostly mechanical changes
adding context parameters.

See: https://github.com/ncw/swift/issues/159
See: https://github.com/ncw/swift/issues/161
2021-02-03 20:23:37 +00:00
Nick Craig-Wood
1545ace8f2 build: remove go1.13 build constraints 2021-02-03 17:46:08 +00:00
Nick Craig-Wood
bcac8fdc83 Use http.NewRequestWithContext where possible after go1.13 minimum version 2021-02-03 17:41:27 +00:00