Commit graph

6548 commits

Author SHA1 Message Date
Nick Craig-Wood
a692bd2cd4 s3: change metadata storage to normal map with lowercase keys 2022-06-29 14:29:36 +01:00
Nick Craig-Wood
776a083892 lsf: add metadata support with M flag 2022-06-29 14:29:36 +01:00
Nick Craig-Wood
d823a38ce5 lsjson: add --metadata/-M flag
Note that this removes the `-M` flag from `--encrypted` as it
conflicted with the global flag and adds it to `--metadata`.
2022-06-29 14:29:36 +01:00
Nick Craig-Wood
78d52882ca fs: add --metadata/-M flag to control whether metadata is copied 2022-06-29 14:29:36 +01:00
Nick Craig-Wood
c4451bc43a fs: add --metadata-set flag to specify metadata for uploads 2022-06-29 14:29:36 +01:00
Nick Craig-Wood
0652ec95db fs: implement MetadataInfo to show info about metadata in help and rc
Info about this will appear in operations/fsinfo and in the backend
help (`rclone help backend s3`).
2022-06-29 14:29:36 +01:00
Nick Craig-Wood
6a0e021dac fs: implement optional Metadata interface for Objects #111
This implements integration tests for the feature also.
2022-06-29 11:21:29 +01:00
Nick Craig-Wood
461d041c4d fstest: remove spurious contents return from PutTestContents and friends 2022-06-29 11:18:02 +01:00
Nick Craig-Wood
35f24d5b84 Add vyloy to contributors 2022-06-29 10:55:03 +01:00
Nick Craig-Wood
370c8fa220 Add mirekphd to contributors 2022-06-29 10:55:03 +01:00
Nick Craig-Wood
0fca4d2c86 Add buda to contributors 2022-06-29 10:55:03 +01:00
Martin Czygan
5de9278650
fs/cache: make sure we call the Shutdown method on backends
This change ensures we call the Shutdown method on backends when
they drop out of the fs/cache and at program exit.

Some backends implement the optional fs.Shutdowner interface. Until now,
Shutdown is only checked and called, when a backend is wrapped (e.g.
crypt, compress, ...).

To have a general way to perform operations at the end of the backend
lifecycle with proper error handling, we can call Shutdown at cache
clear time.

We add a finalize hook to the cache which will be called when values
drop out of the cache.

Previous discussion: https://forum.rclone.org/t/31336
2022-06-28 12:51:59 +01:00
vyloy
326c43ab3f s3: add IDrive e2 to provider list 2022-06-28 09:12:36 +01:00
mirekphd
32006033e6
docs: note wider impact of --checkers=N on parallelism #6280 2022-06-27 19:33:07 +01:00
buda
517e7d9271
accounting: fix unknown length file transfers count 3 transfers each #6213
This was caused by nested calls to NewTransfer/Done.

This fixes the problem by only incrementing transfers if the remote is
present in the transferMap which means we only increment it once.
2022-06-27 17:56:03 +01:00
albertony
fdd2f8e6d2 Error strings should not be capitalized
Reported by staticcheck 2022.1.2 (v0.3.2)

See: staticcheck.io
2022-06-23 23:26:02 +02:00
Abhiraj
027746ef6e
drive: moved rclone_folder_id to advanced section - fixes #3463 2022-06-23 22:08:09 +02:00
albertony
53f831f40a docs: add missing code section formatting to commands and flags 2022-06-21 23:43:00 +02:00
albertony
9f81b4df4f docs/lsjson: fix code block indentation 2022-06-21 23:43:00 +02:00
albertony
bf54c909c9 docs: improved capitalization 2022-06-21 23:43:00 +02:00
albertony
dbf1234edf docs: skip "Connection" suffix from FTP, SSH/SFTP and HTTP backend names 2022-06-21 23:43:00 +02:00
albertony
70d9d75801 docs: improve serve command descriptions 2022-06-21 23:43:00 +02:00
albertony
bc70a95fca docs: consistent capitalization of WebDAV, DLNA, HTTP 2022-06-21 23:43:00 +02:00
albertony
ee87e919c5 docs/ncdu: note that refresh screen shortcut will fix screen corruption 2022-06-21 23:43:00 +02:00
albertony
4f0eae366f docs/ncdu: fix inconsistency in key map help text
Ctrl+L was listed as ^L, while Ctrl+c was listed as c-C. Changed the latter to ^c.
2022-06-21 23:43:00 +02:00
albertony
de5ccaab8e docs: cross link doc pages for related commands 2022-06-21 23:43:00 +02:00
albertony
4b7dc35cf4 Fix sync docs incorrect merge
This copies the changes from an autogenerated section in the following commit:
f2a15a174f
2022-06-21 23:43:00 +02:00
Nick Craig-Wood
bc705e14d8 vfscache: fix fatal error: sync: unlock of unlocked mutex error
This message is a double panic and was actually caused by an assertion
panic in:

vfs/vfscache/downloaders/downloaders.go

This is triggered by the code added relatively recently to fix a bug
with renaming files:

ec72432cec vfs: fix failed to _ensure cache internal error: downloaders is nil error

So it appears that item.o may be nil at this point.

This patch detects item.o being nil and fetches it again with NewObject.

Fixes #6190 Fixes #6235
2022-06-21 14:28:53 +01:00
Nick Craig-Wood
ea5bb79366 drive: document export for google apps scripts better
This also adds some more mime types from the code.

See: https://forum.rclone.org/t/how-do-i-copy-google-apps-scripts-with-rclone/31373
2022-06-21 14:28:53 +01:00
Nick Craig-Wood
e95dff2fa1 drive: add backend commands exportformats and importformats for debugging 2022-06-21 14:28:53 +01:00
Nick Craig-Wood
99dfe1eeae Add Martin Czygan to contributors 2022-06-21 14:28:53 +01:00
Nick Craig-Wood
ed92bf335d Add J-P Treen to contributors 2022-06-21 14:28:53 +01:00
Nick Craig-Wood
3d55b537c6 Add Caleb to contributors 2022-06-21 14:28:53 +01:00
Nick Craig-Wood
d03fffdf8d Add eNV25 to contributors 2022-06-21 14:28:53 +01:00
Martin Czygan
7a909ebfb0
fs/cache: fix cache unpin 2022-06-20 12:14:58 +01:00
buengese
ac0dc9922e copyurl: add tests for the option to honor the HTTP header filename directive 2022-06-20 12:06:09 +01:00
J-P Treen
8b8802a078 copyurl: Add option to honor the HTTP header filename directive.
Implemented --header-filename for use with copyurl.

For specifically setting preferred download filenames for HTTP requests, RFC 6226
specifies a 'filename' directive, available within 'Content-Disposition'
header. We can handle with 'mime.ParseMediaType'.

See below for details:
https://httpwg.org/specs/rfc6266.html#disposition.parameter.filename
https://httpwg.org/specs/rfc6266.html#advice.generating

Co-authored-by: buengese <buengese@protonmail.com>
2022-06-20 12:06:09 +01:00
Caleb
f2a15a174f
docs: grammatical clarification in sync docs 2022-06-19 15:26:53 +02:00
buengese
21c746a56c fichier: parse api error codes and them accordingly 2022-06-19 15:07:33 +02:00
eNV25
36add0afbf
ncdu: replace termbox with tcell's termbox wrapper
The https://github.com/nsf/termbox-go library is no longer maintained
so this change replaces it with the maintained
github.com/gdamore/tcell library which has a termbox backwards
compatibility layer.

There are a few minor changes from the termbox library:

- Using Clear with fg bg ColorDefault resulted in a white background for some reason.
    - Clear with fg ColorWhite bg ColorBlack was used instead.
- tcell's termbox wrapper doesn't support ColorLightYellow.
    - ColorYellow + 8 was used instead.
2022-06-19 11:22:45 +01:00
Nick Craig-Wood
14e0396fcb test_all: allow internet archive backend more time 2022-06-18 15:13:18 +01:00
Nick Craig-Wood
100acc570a test_all: fix -clean so it works on remotes with paths 2022-06-18 15:10:09 +01:00
Nick Craig-Wood
b9de37af80 test_all: Only run backend tests for Internet Archive as it is too slow 2022-06-17 16:52:30 +01:00
Nick Craig-Wood
f7c36ce0f9 s3: unwrap SDK errors to reveal underlying errors on upload
The SDK doesn't wrap errors in a Go standard way so they can't be
unwrapped and tested for - eg fatal error.

The code looks for a Serialization or RequestError and returns the
unwrapped underlying error if possible.

This fixes the fs/operations integration tests checking for fatal
errors being returned.
2022-06-17 16:52:30 +01:00
Nick Craig-Wood
f829ded456 Add Phil Shackleton to contributors 2022-06-17 16:52:30 +01:00
Nick Craig-Wood
2fac8fdde6 Add Scott Grimes to contributors 2022-06-17 16:52:30 +01:00
Phil Shackleton
8e2d9a4cb9
drive: update Internal OAuth consent screen docs
Updated instructions in `Making your own client_id` section to record process for selecting "Internal" OAuth consent screen.
2022-06-17 16:50:01 +01:00
Scott Grimes
295006f662 opendrive: resolve lag and truncate bugs fixes #5936
Co-authored-by: buengese <buengese@protonmail.com>
2022-06-17 16:48:03 +01:00
Lu Wang
dcc128c70d
docs/onedrive: document creation of client ID for OneDrive Business 2022-06-17 07:33:56 +02:00
Nick Craig-Wood
c85fbebce6 s3: simplify PutObject code to use the Request.SetStreamingBody method
In this commit

e5974ac4b0 s3: use PutObject from the aws SDK to upload single part objects

rclone was made to upload objects to s3 using PUT requests rather than
using signed uploads.

However this change missed the fact that there is a supported way to
do this in the SDK using the SetStreamingBody method on the Request.

This therefore reverts a lot of the previous commit to do with making
an unsigned connection and other complication and uses the SDK
facility.
2022-06-16 23:26:19 +01:00