TrueCloudLab/rclone
16
0
Fork 2
Code Issues 1 Pull requests 1 Projects Releases Packages Wiki Activity Actions
7649 commits 181 branches 114 tags 335 MiB
Commit graph

7649 commits

This branch
This branch
All branches
Author SHA1 Message Date
Nick Craig-Wood
b8591b230d onedrive: implement ListR method which gives --fast-list support 
This implents ListR for onedrive. The API only allows doing this at
the root so it is inefficient to use it not at the root.

Fixes #7317
 2023-10-02 11:12:08 +01:00
Nick Craig-Wood
ecb09badba onedrive: factor API types back into correct file 2023-10-02 10:48:06 +01:00
Nick Craig-Wood
cb43e86d16 b2: reduce default --b2-upload-concurrency to 4 to reduce memory usage 
In v1.63 memory usage in the b2 backend was limited to `--transfers` *
`--b2-chunk-size`

However in v1.64 this was raised to `--transfers` * `--b2-chunk-size`
* `--b2-upload-concurrency`.

The default value for this was accidently set quite high at 16 which
means by default rclone could use up to 6.4GB of memory!

The new default sets a more reasonable (but still high) max memory of 1.6GB.
 2023-10-01 12:30:26 +01:00
Nick Craig-Wood
5c48102ede b2: fix locking window when getting mutipart upload URL 
Before this change, the lock was held while the upload URL was being
fetched from the server.

This meant that any other threads were blocked from getting upload
URLs unecessarily.

It also increased the potential for deadlock.
 2023-10-01 12:30:26 +01:00
Nick Craig-Wood
96438ff259 pacer: fix b2 deadlock by defaulting max connections to unlimited 
Before this change, the maximum number of connections was set to 10.

This means that b2 could deadlock while uploading multipart uploads
due to a lock being held longer than it should have been.
 2023-10-01 12:30:26 +01:00
albertony
c1df3ce08c docs: add utime (time of file upload) to standard system metadata 2023-09-29 13:19:57 +02:00
albertony
19ad39fa1c jottacloud: add support for reading and writing metadata 
Most useful is the addition of the file created timestamp, but also a timestamp for
when the file was uploaded.

Currently supporting a rather minimalistic set of metadata items, see PR #6359 for
some thoughts about possible extensions.
 2023-09-29 13:19:57 +02:00
Nick Craig-Wood
b296f37801 s3: fix slice bounds out of range error when listing 
In this commit:

5f938fb9ed s3: fix "Entry doesn't belong in directory" errors when using directory markers

We checked that the remote has the prefix and then changed the remote
before removing the prefix. This sometimes causes:

    panic: runtime error: slice bounds out of range [56:55]

The fix is to do the modification of the remote after removing the
prefix.

See: https://forum.rclone.org/t/cryptcheck-panic-runtime-error-slice-bounds-out-of-range/41977
 2023-09-25 11:52:23 +01:00
Nick Craig-Wood
23e44c6065 Add rinsuki to contributors 2023-09-25 11:52:23 +01:00
rinsuki
8fd66daab6 drive: add support of SHA-1 and SHA-256 checksum 2023-09-24 17:38:30 +01:00
Nick Craig-Wood
9e80d48b03 s3: add docs on how to add a new provider 2023-09-23 14:36:48 +01:00
Nick Craig-Wood
eb3082a1eb s3: add Linode provider 2023-09-23 14:34:00 +01:00
Nick Craig-Wood
77ea22ac5b s3: Factor providers list out and auto generate textual version 2023-09-23 14:34:00 +01:00
Nick Craig-Wood
9959712a06 docs: fix backend doc generator to not output duplicate config names 
This was always the intention, it was just implemented wrong.

This shortens the s3 docs by 1369 bringing them down to half the size
just about.

Fixes #7325
 2023-09-23 12:54:08 +01:00
Nick Craig-Wood
7586fecbca Add Nikita Shoshin to contributors 2023-09-23 12:54:08 +01:00
Nikita Shoshin
94cdb00eb6 rcserver: set Last-Modified header for files served by --rc-serve 2023-09-23 12:20:29 +01:00
Dimitri Papadopoulos Orfanos
3d473eb54e
docs: fix typos found by codespell in docs and code comments 2023-09-23 12:20:01 +01:00
Nick Craig-Wood
50b4a2398e onedrive: fix the configurator to allow /teams/ID in the config 
See: https://forum.rclone.org/t/sharepoint-to-google/41548/
 2023-09-22 15:54:22 +01:00
Nick Craig-Wood
e6b718c938 build: add btesth target to output beta log in HTML for email pasting 2023-09-21 16:15:48 +01:00
Nick Craig-Wood
9370dbcc47 lsjson: make sure we set the global metadata flag too 2023-09-21 16:15:38 +01:00
Nick Craig-Wood
8c1e9a2905 rc: always report an error as JSON 
Before this change, the rclone rc command wouldn't actually report the
error as a JSON blob which is inconsitent with what the HTTP API does.

This change make sure we always report a JSON error, making a
synthetic one if necessary.

See: https://forum.rclone.org/t/when-using-rclone-rc-commands-somehow-return-errors-as-parsable-json/41855
Co-authored-by: Fawzib Rojas
 2023-09-20 21:57:40 +01:00
Nick Craig-Wood
6072d314e1 b2: fix multipart upload: corrupted on transfer: sizes differ XXX vs 0 
Before this change the b2 backend wasn't writing the metadata to the
object properly after a multipart upload.

The symptom of this was that sometimes it would give the error:

    corrupted on transfer: sizes differ XXX vs 0

This was fixed by returning the metadata in the chunk writer and setting it in Update.

See: https://forum.rclone.org/t/multipart-upload-to-b2-sometimes-failing-with-corrupted-on-transfer-sizes-differ/41829
 2023-09-18 20:41:31 +01:00
Nick Craig-Wood
9277ca1e54 b2: implement --b2-lifecycle to control lifecycle when creating buckets 2023-09-16 17:01:43 +01:00
Nick Craig-Wood
d6722607cb b2: implement "rclone backend lifecycle" to read and set bucket lifecycles 2023-09-16 16:44:28 +01:00
Nick Craig-Wood
4ef30db209 b2: fix listing all buckets when not needed 
Before this change the b2 backend listed all the buckets to turn a
single bucket name into an ID.

However in July 26, 2018 a parameter was added to the list buckets API
to make listing all the buckets unecessary.

This code sets the bucketName parameter so that only the results for
the desired bucket are returned.
 2023-09-16 16:04:50 +01:00
Nick Craig-Wood
55c12c9a2d azureblob: fix "fatal error: concurrent map writes" 
Before this change, the metadata map could be accessed from multiple
goroutines at once, sometimes causing this error.

This fix adds a global mutex for adjusting the metadata map to make
all accesses safe.

See: https://forum.rclone.org/t/azure-blob-storage-with-vfs-cache-concurrent-map-writes-exception/41686
 2023-09-16 11:33:03 +01:00
dependabot[bot]
4349dae784 build(deps): bump docker/setup-qemu-action from 2 to 3 
Bumps [docker/setup-qemu-action](https://github.com/docker/setup-qemu-action) from 2 to 3.
- [Release notes](https://github.com/docker/setup-qemu-action/releases)
- [Commits](https://github.com/docker/setup-qemu-action/compare/v2...v3)

---
updated-dependencies:
- dependency-name: docker/setup-qemu-action
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
 2023-09-15 10:25:14 +01:00
David Sze
3e63f2c249 box: add more logging for polling 2023-09-15 10:24:43 +01:00
David Sze
5118ab9609 box: filter more EventIDs when polling 2023-09-15 10:24:43 +01:00
dependabot[bot]
7ea118aeae build(deps): bump docker/setup-buildx-action from 2 to 3 
Bumps [docker/setup-buildx-action](https://github.com/docker/setup-buildx-action) from 2 to 3.
- [Release notes](https://github.com/docker/setup-buildx-action/releases)
- [Commits](https://github.com/docker/setup-buildx-action/compare/v2...v3)

---
updated-dependencies:
- dependency-name: docker/setup-buildx-action
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
 2023-09-15 09:15:26 +01:00
Kaloyan Raev
af260921c0 storj: update storj.io/uplink to v1.12.0 
The improved upload logic is active by default in uplink v1.12.0, so the
`testuplink.WithConcurrentSegmentUploadsDefaultConfig(ctx)` is not
required anymore.

See https://github.com/rclone/rclone/pull/7198
 2023-09-14 14:01:35 +01:00
Nick Craig-Wood
62db2bb329 docs: add notes on how to update the website between releases 2023-09-13 17:02:29 +01:00
Nick Craig-Wood
8d58adbd54 docs: remove minio sponsor box for the moment 2023-09-13 17:02:29 +01:00
Nick Craig-Wood
b4d251081f docs: update Storj partner link 2023-09-13 17:02:29 +01:00
Nick Craig-Wood
a382bf8f03 Add Herby Gillot to contributors 2023-09-13 17:02:29 +01:00
Nick Craig-Wood
28fc43fb11 Add Pat Patterson to contributors 2023-09-13 17:02:29 +01:00
Herby Gillot
83be1501db docs: add MacPorts install info 
https://ports.macports.org/port/rclone/
 2023-09-13 16:06:15 +01:00
dependabot[bot]
be156133c5 build(deps): bump docker/metadata-action from 4 to 5 
Bumps [docker/metadata-action](https://github.com/docker/metadata-action) from 4 to 5.
- [Release notes](https://github.com/docker/metadata-action/releases)
- [Upgrade guide](https://github.com/docker/metadata-action/blob/master/UPGRADE.md)
- [Commits](https://github.com/docker/metadata-action/compare/v4...v5)

---
updated-dependencies:
- dependency-name: docker/metadata-action
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
 2023-09-13 15:56:38 +01:00
dependabot[bot]
d494db78d9 build(deps): bump docker/login-action from 2 to 3 
Bumps [docker/login-action](https://github.com/docker/login-action) from 2 to 3.
- [Release notes](https://github.com/docker/login-action/releases)
- [Commits](https://github.com/docker/login-action/compare/v2...v3)

---
updated-dependencies:
- dependency-name: docker/login-action
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
 2023-09-13 15:55:58 +01:00
dependabot[bot]
1b9eb74204 build(deps): bump docker/build-push-action from 4 to 5 
Bumps [docker/build-push-action](https://github.com/docker/build-push-action) from 4 to 5.
- [Release notes](https://github.com/docker/build-push-action/releases)
- [Commits](https://github.com/docker/build-push-action/compare/v4...v5)

---
updated-dependencies:
- dependency-name: docker/build-push-action
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
 2023-09-13 15:55:23 +01:00
Manoj Ghosh
cda09704a8 fix overview of oracle object storage as it supports multithreaded 2023-09-13 12:14:07 +01:00
Pat Patterson
71f2883562 operations: ensure concurrency is no greater than the number of chunks - fixes #7299 2023-09-13 12:12:27 +01:00
Nick Craig-Wood
d29d263329 docs: fix minimum Go version and update to 1.18 
Fixes #7288
 2023-09-12 11:24:17 +01:00
Nick Craig-Wood
20126de1aa Start v1.65.0-DEV development 2023-09-12 11:24:17 +01:00
Nick Craig-Wood
77f7bb08af Version v1.64.0 2023-09-11 15:59:44 +01:00
Nick Craig-Wood
a5a61f4874 protondrive: make cached keys rclone style and not show with rclone config redacted 2023-09-11 15:57:08 +01:00
Nick Craig-Wood
e8879f3e77 docs: document release signing and verification 
Fixes #7257
 2023-09-11 12:28:23 +01:00
Nick Craig-Wood
2a6675cffd docs: fix typo in rc docs - fixes #7287 2023-09-11 09:34:22 +01:00
Nick Craig-Wood
fa4d171f62 protondrive: complete docs with all references to Proton Drive 2023-09-10 11:57:39 +01:00
Nick Craig-Wood
d4d530bd8e drive: add --drive-fast-list-bug-fix to control ListR bug workaround 
See: https://forum.rclone.org/t/how-to-list-empty-directories-recursively/40995/12
 2023-09-09 17:46:03 +01:00