TrueCloudLab/rclone
16
0
Fork 2
Code Issues Pull requests 1 Projects Releases Packages Wiki Activity Actions 1
6554 commits 181 branches 114 tags 335 MiB
Commit graph

257 commits

Author SHA1 Message Date
Nick Craig-Wood
617c5d5e1b rcat: preserve metadata when Copy falls back to Rcat 
Before this change if we copied files of unknown size, then they lost
their metadata.

This was particularly noticeable using --s3-decompress.

This change adds metadata to Rcat and RcatSized and changes Copy to
pass the metadata in when it calls Rcat for an unknown sized input.

Fixes #6546
 2022-11-10 12:04:35 +00:00
Nick Craig-Wood
be6f29930b dedupe: make dedupe obey the filters 
See: https://forum.rclone.org/t/dial-tcp-lookup-api-pcloud-com-no-such-host/33910/
 2022-11-10 09:56:02 +00:00
Nick Craig-Wood
653bc23728 dedupe: count Checks in the stats while scanning for duplicates 
This allows the user to see rclone has not hung.

See: https://forum.rclone.org/t/dial-tcp-lookup-api-pcloud-com-no-such-host/33910/
 2022-11-10 09:56:02 +00:00
albertony
5d6b8141ec Replace deprecated ioutil 
As of Go 1.16, the same functionality is now provided by package io or
package os, and those implementations should be preferred in new code.
 2022-11-07 11:41:47 +00:00
Josh Soref
ce3b65e6dc all: fix spelling across the project 
* abcdefghijklmnopqrstuvwxyz
* accounting
* additional
* allowed
* almost
* already
* appropriately
* arise
* bandwidth
* behave
* bidirectional
* brackets
* cached
* characters
* cloud
* committing
* concatenating
* configured
* constructs
* current
* cutoff
* deferred
* different
* directory
* disposition
* dropbox
* either way
* error
* excess
* experiments
* explicitly
* externally
* files
* github
* gzipped
* hierarchies
* huffman
* hyphen
* implicitly
* independent
* insensitive
* integrity
* libraries
* literally
* metadata
* mimics
* missing
* modification
* multipart
* multiple
* nightmare
* nonexistent
* number
* obscure
* ourselves
* overridden
* potatoes
* preexisting
* priority
* received
* remote
* replacement
* represents
* reproducibility
* response
* satisfies
* sensitive
* separately
* separator
* specifying
* string
* successful
* synchronization
* syncing
* šenfeld
* take
* temporarily
* testcontents
* that
* the
* themselves
* throttling
* timeout
* transaction
* transferred
* unnecessary
* using
* webbrowser
* which
* with
* workspace

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>
 2022-08-30 11:16:26 +02:00
Nick Craig-Wood
6fd9e3d717 build: reformat comments to pass go1.19 vet 
See: https://go.dev/doc/go1.19#go-doc
 2022-08-05 16:35:41 +01:00
Nick Craig-Wood
a875320e37 sync,operations: optimise --copy-dest and --compare-dest 
Before this change --compare-dest and --copy-dest would check to see
if the compare/copy object existed first, before seeing if the
destination object was present.

This is inefficient, because in most --copy-dest syncs the destination
will be present and the compare/copy object need never be tested.
--compare-dest syncs may also be speeded up if they are done to the
same directory repeatedly.

This fixes the problem by re-arranging the logic so if the transfer is
not needed then the compare/copy object is never tested.

See: https://forum.rclone.org/t/union-with-copy-dest-enabled-is-slower-than-expected/32172
 2022-08-03 17:44:47 +01:00
Nick Naumann
11be920e90 sync: add filter-sensitivity to --backup-dir option 
The old Overlapping function and corresponding tests have been removed, as it has been completely replaced by the OverlappingFilterCheck function.
 2022-07-13 16:04:53 +01:00
Nick Craig-Wood
55bbff6346 operations: add --server-side-across-configs global flag for any backend 2022-07-11 18:17:42 +01:00
Nick Craig-Wood
2e54b56a01 rcat: check checksums by default like copy does #6305 
Before this change we were calculating the checksum for an rcat
transfer but never checking it.

See: https://forum.rclone.org/t/optimize-rclone-on-raspberry-pi-4-8gb/31741
 2022-07-07 16:09:09 +01:00
albertony
5b579cea47 staticcheck: use golang.org/x/text/cases instead of deprecated strings.Title 
strings.Title has been deprecated since Go 1.18 and an alternative has been
available since Go 1.0. The rule Title uses for word boundaries does not handle
Unicode punctuation properly. Use golang.org/x/text/cases instead.
 2022-07-04 11:24:59 +02:00
albertony
0772cae314 staticcheck: use result of type assertion to simplify cases 2022-07-04 11:24:59 +02:00
zzr93
060c8dfff0
operations: use correct src/dst in some log messages 
Most of the time this will make no difference to user logs, however
the difference may be visible in JSON logs and on the rare occasions
src and dst are pointing to different file names.
 2022-07-04 10:18:04 +01:00
Nick Craig-Wood
776a083892 lsf: add metadata support with M flag 2022-06-29 14:29:36 +01:00
Nick Craig-Wood
d823a38ce5 lsjson: add --metadata/-M flag 
Note that this removes the `-M` flag from `--encrypted` as it
conflicted with the global flag and adds it to `--metadata`.
 2022-06-29 14:29:36 +01:00
Nick Craig-Wood
c4451bc43a fs: add --metadata-set flag to specify metadata for uploads 2022-06-29 14:29:36 +01:00
Nick Craig-Wood
0652ec95db fs: implement MetadataInfo to show info about metadata in help and rc 
Info about this will appear in operations/fsinfo and in the backend
help (`rclone help backend s3`).
 2022-06-29 14:29:36 +01:00
Nick Craig-Wood
6a0e021dac fs: implement optional Metadata interface for Objects #111 
This implements integration tests for the feature also.
 2022-06-29 11:21:29 +01:00
Nick Craig-Wood
461d041c4d fstest: remove spurious contents return from PutTestContents and friends 2022-06-29 11:18:02 +01:00
albertony
fdd2f8e6d2 Error strings should not be capitalized 
Reported by staticcheck 2022.1.2 (v0.3.2)

See: staticcheck.io
 2022-06-23 23:26:02 +02:00
albertony
de5ccaab8e docs: cross link doc pages for related commands 2022-06-21 23:43:00 +02:00
buengese
ac0dc9922e copyurl: add tests for the option to honor the HTTP header filename directive 2022-06-20 12:06:09 +01:00
J-P Treen
8b8802a078 copyurl: Add option to honor the HTTP header filename directive. 
Implemented --header-filename for use with copyurl.

For specifically setting preferred download filenames for HTTP requests, RFC 6226
specifies a 'filename' directive, available within 'Content-Disposition'
header. We can handle with 'mime.ParseMediaType'.

See below for details:
https://httpwg.org/specs/rfc6266.html#disposition.parameter.filename
https://httpwg.org/specs/rfc6266.html#advice.generating

Co-authored-by: buengese <buengese@protonmail.com>
 2022-06-20 12:06:09 +01:00
Nick Craig-Wood
115f1c2cc9 operations: speed up hash checking by aborting the other hash if first returns nothing 
This speeds up hash checks when a Hash() function returns "" - this
means that the hash can be canceled for the other side.

In the common case of local hash vs remote hash empty this saves a lot
of time.

See: https://forum.rclone.org/t/rclone-s3-backend-copy-is-2x-slower-than-aws-s3-cp/27321/9
 2022-06-16 22:13:50 +01:00
Nick Craig-Wood
4f94b27800 check: implement --no-traverse and --no-unicode-normalization 
See: https://forum.rclone.org/t/rclone-check-head-or-list-object-from-source/30400
 2022-06-16 22:13:50 +01:00
Nick Craig-Wood
cfe0911e0d sync: fix tests for overlapping with filter 
In commit

3ccf222acb sync: overlap check is now filter-sensitive

The tests were attempting to write invalid objects on some backends
due to a leading / on the object name.

This fix also adds a few more test cases and makes sure the tests can
be run individually.
 2022-06-09 14:26:43 +01:00
albertony
f4f0e444bf filter: allow multiple --exclude-if-present flags - fixes #6219 2022-06-08 17:11:52 +02:00
Nick
3ccf222acb
sync: overlap check is now filter-sensitive 
Previously, the overlap check was based on simple prefix checks of the source and destination paths. Now it actually checks whether the destination is excluded via any filter rule or a "--exclude-if-present"-file.
 2022-06-01 18:24:54 +01:00
Eng Zer Jun
4f0ddb60e7 refactor: replace strings.Replace with strings.ReplaceAll 
strings.ReplaceAll(s, old, new) is a wrapper function for
strings.Replace(s, old, new, -1). But strings.ReplaceAll is more
readable and removes the hardcoded -1.

Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
 2022-05-17 11:08:37 +01:00
albertony
86bd5f6922 size: warn about inaccurate results when objects with unknown size 2022-04-26 19:37:15 +02:00
albertony
07f53aebdc touch: fix issue where directory is created instead of file 
Detected on ftp, sftp and Dropbox backends.

Fixes #5952
 2022-01-28 20:29:12 +01:00
albertony
237daa8aaf dedupe: add quit as a choice in interactive mode 
Fixes #5881
 2022-01-14 19:57:48 +01:00
Ole Frost
3c2252b7c0 fs/operations: add server-side moves to stats 
Fixes #5430
 2021-11-18 12:20:56 +00:00
Nick Craig-Wood
ba6730720d Fix repeated error messages after pkg/errors removal 2021-11-15 17:58:40 +00:00
albertony
29abbd2032 hashsum: support creating hash from data received on stdin 
See #5781
 2021-11-11 15:16:45 +01:00
Nick Craig-Wood
f36d6d01b5 rc: fix operations/publiclink default for expires parameter 
Before this change the expires parameter was defaulting to 0 if not
provided.

This change makes it default to fs.DurationOff which is the same as
the `rclone link` command.

See: https://forum.rclone.org/t/operations-publiclink-from-dropbox-error-not-autorized/27374
 2021-11-11 11:14:22 +00:00
database64128
a7a8372976
🧪 fstest: fix time tests on Windows and add convenience methods to check local and remote fs with precision 
Previously only the fs being checked on gets passed to
GetModifyWindow(). However, in most tests, the test files are
generated in the local fs and transferred to the remote fs. So the
local fs time precision has to be taken into account.

This meant that on Windows the time tests failed because the
local fs has a time precision of 100ns. Checking remote items uploaded
from local fs on Windows also requires a modify window of 100ns.
 2021-11-09 11:43:36 +00:00
Nick Craig-Wood
e43b5ce5e5 Remove github.com/pkg/errors and replace with std library version 
This is possible now that we no longer support go1.12 and brings
rclone into line with standard practices in the Go world.

This also removes errors.New and errors.Errorf from lib/errors and
prefers the stdlib errors package over lib/errors.
 2021-11-07 11:53:30 +00:00
Atílio Antônio
c08d48a50d
docs: improve grammar and fix typos (#5361) 
This alters some comments in source files, but is interested mainly in documentation files and help messages.
 2021-11-04 12:50:43 +01:00
albertony
4f05ece39e test: fix touchdir test on backends without modtime support 2021-10-22 13:37:34 +02:00
albertony
a70c20fe6b touch: improve error message from recursive touch 2021-10-22 13:37:34 +02:00
albertony
e2f47ecdeb docs: punctuation cleanup 
See #5538
 2021-10-20 22:56:19 +02:00
albertony
41876dd669 touch: add support for touching files in directory, with options for recursive, filtering and dry-run/interactive 
Fixes #5301
 2021-10-20 19:24:57 +02:00
Nick Craig-Wood
54da6154c4 operations: fix lsjson --stat on root directories of bucket based backends 2021-10-16 10:38:24 +01:00
Nick Craig-Wood
edf8978d15 operations: fix HashSum tests after removing ERROR and UNSUPPORTED 
This was caused by

7a1cab57b6 cmd/hashsum: dont put ERROR or UNSUPPORTED in output

And was picked up in the integration tests.

This patch no longer calls the HashLister for unsupported hash types.
 2021-10-15 10:51:08 +01:00
Nick Craig-Wood
f529c02446 lsjson: add --stat flag and operations/stat api 
This enables information about single files to be efficiently
retrieved.
 2021-10-14 17:15:50 +01:00
Ivan Andreev
16fb608bee hashsum: treat hash values in sum file as case insensitive 
Also warn duplicate file paths in sum files.

Fixes https://forum.rclone.org/t/rclone-check-sum/25566/45
 2021-10-13 18:21:34 +03:00
Ivan Andreev
7a1cab57b6 cmd/hashsum: dont put ERROR or UNSUPPORTED in output 2021-10-08 14:26:27 +03:00
albertony
721a9786a7 ls: introduce and global option to print human-readable sizes and consider it for ls commands 
Fixes #1890
 2021-08-31 09:57:27 +02:00
Nathan Collins
c32d5dd1f3 fs: move with --ignore-existing will not delete skipped files - #5463 2021-08-01 17:46:45 +01:00