TrueCloudLab/rclone
16
0
Fork 2
Code Issues Pull requests 1 Projects Releases Packages Wiki Activity Actions 1
8340 commits 181 branches 114 tags 335 MiB
Commit graph

251 commits

Author SHA1 Message Date
Ivan Andreev
3859fe2f52
cmd/version: print os/version, kernel and bitness (#5204) 
Related to #5121

Note: OpenBSD is stub yet. This will be fixed after upstream PR gets resolved
https://github.com/shirou/gopsutil/pull/993
 2021-04-05 21:53:09 +03:00
Nick Craig-Wood
f6dbb98a1d cmount: update cgofuse to the latest version to bring in macfuse 4 fix 2021-03-25 09:02:17 +00:00
nguyenhuuluan434
d6870473a1 swift: implement copying large objects 2021-03-24 08:56:39 +00:00
Nick Craig-Wood
713f8f357d sftp: fix "file not found" errors for read once servers - fixes #5077 
It introduces a new flag --sftp-disable-concurrent-reads to stop the
problematic behaviour in the SFTP library for read-once servers.

This upgrades the sftp library to v1.13.0 which has the fix.
 2021-03-13 15:38:38 +00:00
Ivan Andreev
75056dc9b2
ftp: update dependency jlaffaye/ftp (#5097) 2021-03-05 15:58:04 +03:00
Robert Thomas
e5aa92c922
dropbox: add polling support - fixes #2949 
This implements polling support for the Dropbox backend. The Dropbox SDK dependency had to be updated due to an auth issue, which was fixed on Jan 12 2021. A secondary internal Dropbox service was created to handle unauthorized SDK requests, as is necessary when using the ListFolderLongpoll function/endpoint. The config variable was renamed to cfg to avoid potential conflicts with the imported config package.
 2021-02-24 09:33:31 +00:00
Rauno Ots
9808a53416 webdav: add support for sharepoint with NTLM authentication (#2921) 
Add new option option "sharepoint-ntlm" for the vendor setting.
Use it when your hosted Sharepoint is not tied to the OneDrive
accounts and uses NTLM authentication.
Also add documentation and integration test.

Fixes: #2171
 2021-02-23 23:27:30 +00:00
Klaus Post
424aaac2e1 encoder/filename: Add SCSU as tables 
Instead of only adding SCSU, add it as an existing table.

Allow direct SCSU and add a, perhaps, reasonable table as well.

Add byte interfaces that doesn't base64 encode the URL as well with `EncodeBytes` and `DecodeBytes`.

Fuzz tested and decode tests added.
 2021-02-12 11:39:39 +00:00
Nick Craig-Wood
993146375e build: update all dependencies 2021-02-03 21:34:38 +00:00
Nick Craig-Wood
bbe791a886 swift: update github.com/ncw/swift to v2.0.0 
The update to v2 of the swift library introduces a context parameter
to each function. This required a lot of mostly mechanical changes
adding context parameters.

See: https://github.com/ncw/swift/issues/159
See: https://github.com/ncw/swift/issues/161
 2021-02-03 20:23:37 +00:00
buengese
d3b7f14b66 bump sgzip to v1.1.0 - fixes #4970 2021-02-01 18:34:42 +00:00
Nick Craig-Wood
189ef5f257 azureblob: fix memory usage by upgrading the SDK and implementing a TransferManager 
In the Azure SDK there was a bug which caused excessive memory use
when doing repeated transfers:

https://github.com/Azure/azure-storage-blob-go/issues/233

This patch updates the SDK to v0.13.0 which allowed us to implement a
custom TransferManager which integrates with rclone's memory pool.

This fixes the excessive memory consumption.

See: https://forum.rclone.org/t/ask-for-settings-recommendation-for-azureblob/21505/
 2021-01-30 17:26:59 +00:00
Yury Stankevich
b569dc11a0 hdfs: support kerberos authentication #42 2021-01-27 18:16:58 +00:00
Yury Stankevich
71edc75ca6 HDFS (Hadoop Distributed File System) implementation - #42 
This includes an HDFS docker image to use with the integration tests.

Co-authored-by: Ivan Andreev <ivandeex@gmail.com>
Co-authored-by: Nick Craig-Wood <nick@craig-wood.com>
 2021-01-07 09:48:51 +00:00
Mitsuo Heijo
9ea990d5a2 azureblob: update azure-storage-blob-go to v0.12.0 
See https://github.com/Azure/azure-storage-blob-go/blob/master/ChangeLog.md#version-0120
 2020-12-28 13:29:38 +00:00
James Lim
2fd543c989 azure: add support for service principals - fixes #3230 
Before: users can only connect to Azure blob containers using the access keys
from the storage account.

After: users can additionally choose connect to Azure blob containers
using service principals. This uses OAuth2 under the hood to exchange
a client ID and client secret for a short-lived access token.

Ref:
- https://github.com/rclone/rclone/issues/3230
- https://docs.microsoft.com/en-us/azure/storage/common/storage-auth-aad-app?tabs=dotnet#well-known-values-for-authentication-with-azure-ad
- https://docs.microsoft.com/en-us/azure/developer/go/azure-sdk-authorization#available-authentication-types-and-methods
- https://gist.github.com/ItalyPaleAle/ec6498bfa81a96f9ca27a2da6f60a770
 2020-12-09 17:52:15 +00:00
Caleb Case
1efb8ea280 backend/tardigrade: Upgrade to uplink v1.4.1 
Uplink v1.4.1 provides two important improvements for rclone:

* Fix for a connection handling issue where an open project could
  potentially become unusable because the underlying connection had
  failed.
* Fix for concurrent use issue in drpc.
 2020-12-06 11:45:47 +00:00
Nick Craig-Wood
23469c9c7c ftp: add --ftp-disable-msld option to ignore MLSD for really old servers 
This is useful for servers which advertise MLSD (eg some versions of
Serv-U) but don't support it properly.

See: https://forum.rclone.org/t/double-folder-names-on-target-destination-paths-ftp/18822
See: https://github.com/jlaffaye/ftp/pull/196
 2020-11-13 11:25:34 +00:00
buengese
bc4282e49e compress: added experimental compression remote - implements #2098, #1356, #675 
This remote implements transparent compression using gzip. Uses JSON as a for storing metadata.

Co-authored-by: id01 <gaviniboom@gmail.com>
 2020-11-13 02:31:59 +01:00
Nick Craig-Wood
1973fc1ecc azureblob: update lib from v0.10.0 to v0.11.0 and fix API breakage 
See: https://github.com/Azure/azure-storage-blob-go/issues/226
 2020-10-29 13:34:39 +00:00
Nick Craig-Wood
7c39a13281 build: update all dependencies 2020-10-29 13:34:39 +00:00
Hekmon
c58023a9ba enhance systemd integration 
* log level identification
* manual activation with flag
* automatic systemd launch detection
 2020-10-06 16:21:30 +01:00
Leo Luan
c665201b85 vfs: support synchronous cache space recovery upon ENOSPC 
This patch provides the support of synchronous cache space recovery
to allow read threads to recover from ENOSPC errors when cache space
can be recovered from cache items that are not in use or safe to be
reset/emptied .

The patch complements the existing cache cleaning process in two ways.

Firstly, the existing cache cleaning process is time-driven that runs
periodically. The cache space can run out while the cache cleaner
thread is still waiting for its next scheduled run. The io threads
encountering ENOSPC return an internal error to the applications
in this case even when cache space can be recovered to avoid this
error. This patch addresses this problem by having the read threads
kick the cache cleaner thread in this condition to recover cache
space preventing unnecessary ENOSPC errors from being seen by the
applications.

Secondly, this patch enhances the cache cleaner to support cache
item reset. Currently the cache purge process removes cache
items that are not in use. This may not be sufficient when the
total size of the working set exceeds the cache directory's
capacity. Like in the current code, this patch starts the purge
process by removing cache files that are not in use. Cache items
whose access times are older than vfs-cache-max-age are removed first.
After that, other not-in-use items are removed in LRU order until
vfs-cache-max-size is reached. If the vfs-cache-max-size (the quota)
is still not reached at this time, this patch adds a cache reset
step to reset/empty cache files that are still in use but not
dirtied.  This enables application processes to continue without
seeing an error even when the working set depletes the cache space
as long as there is not a large write working set hoarding the
entire cache space.

By design this patch does not add ENOSPC error recovery for write
IOs. Rclone does not empty a write cache item until the file data
is written back to the backend upon close. Allowing more cache
space to be consumed by dirty cache items when the cache space is
already running low would increase the risk of exhausting the cache
space in a way that the vfs mount becomes unreadable.
 2020-08-25 21:12:06 +01:00
Kaloyan Raev
3d5a63607e backend/tardigrade: Upgrade to uplink v1.2.0 
Uplink v1.2.0 comes with two improvements related to rclone:
* Fix for resource leak in uploads.
* The socket dialer comes with better congestion control in some
environments. On Linux environments, if a congestion controller named
'ledbat' is installed, it will be used. Consider installing
https://github.com/silviov/TCP-LEDBAT
 2020-08-13 16:32:18 +01:00
Klaus Post
cb7534dcdf
lib: Add file name compression 
Allows to compress short arbitrary strings and returns a string using base64 url encoding.

Generator for tables included and a few samples has been added. Add more to init.go

Tested with fuzzing for crash resistance and symmetry, see fuzz.go
 2020-08-13 16:14:11 +01:00
Nick Craig-Wood
78fdc5805b vendor: Update github.com/billziss-gh/cgofuse to v1.4.0 #4393 2020-07-24 15:12:05 +01:00
Nick Craig-Wood
db56b1bfec serve/ftp: use refactored goftp.io/server library for binary shrink 
This uses the refactored goftp library which doesn't include the minio
driver. This reduces the binary size by 1.5MB

See: https://gitea.com/goftp/server/pulls/120
 2020-07-21 16:23:55 +01:00
Nick Craig-Wood
990a33b393 build: go mod tidy 2020-07-21 16:19:24 +01:00
Nick Craig-Wood
664c658da6 ftp: Update github.com/jlaffaye/ftp to fix interop with pure-ftpd 
See: https://github.com/jlaffaye/ftp/pull/190
 2020-07-21 16:17:37 +01:00
Nick Craig-Wood
f524a4c1cc vendor: drop unused github.com/djherbis/times 2020-07-06 14:29:42 +01:00
Nick Craig-Wood
c7eae60944 vendor: update goftp.io/server to v0.3.5-master to fix auth proxy #4394 
See upstream issue: https://gitea.com/goftp/server/issues/117
 2020-06-30 09:16:43 +01:00
Nick Craig-Wood
2aed3bf9ab vendor: roll back bazil.org/fuse to the last version which supports macOS #4393 
Roll back the bazil.org/fuse update to give us some time to explore
alternatives for macOS.

See upstream issue: https://github.com/bazil/fuse/issues/224
 2020-06-29 14:45:02 +01:00
Nick Craig-Wood
ec4e0e4d58 vendor: revert goftp.io/server from v0.3.4 to v0.3.3 to fix auth proxy #4394 
See upstream issue: https://gitea.com/goftp/server/issues/117
 2020-06-29 14:45:02 +01:00
Nick Craig-Wood
696d012c05 vendor: update all dependencies 2020-06-29 14:44:57 +01:00
Caleb Case
e780cda1d4 backend/tardigrade: Upgrade to uplink v1.1.1 
This fixes issue #4370 by restoring the correct error response.
 2020-06-20 16:44:06 +01:00
Heiko Bornholdt
c4ce260b49 vendor: update jlaffaye/ftp 2020-06-16 09:13:50 +01:00
Caleb Case
a4f1f3d4e8 backend/tardigrade: Upgrade uplink to v1.0.7 
This fixes a regression in the rclone tests from the v1.0.6 upgrade of
uplink. The failure was due to an improperly converted error resulting
in the wrong type of error.
 2020-06-05 10:51:33 +01:00
Nick Craig-Wood
b59999dd59 build: update go.mod to go1.14 to enable -mod=vendor build 
When the main module contains a top-level vendor directory and its
go.mod file specifies go 1.14 or higher, the go command now defaults
to -mod=vendor for operations that accept that flag.
 2020-06-01 16:13:23 +01:00
Caleb Case
e7bd392a69 backend/tardigrade: Upgrade to uplink v1.0.6 
This fixes an important bug with listing that affects users with more
than 500 objects in a listing operation.
 2020-05-29 18:00:08 +01:00
Caleb Case
03b629064a Tardigrade Backend: Dependencies 2020-05-12 15:56:50 +00:00
Fred
62cfe3f384 vendor: add github.com/coreos/go-semver 2020-05-06 17:33:22 +00:00
Nick Craig-Wood
b03cad3cf6 vendor: update google.golang.org/api/drive to pull in shortcuts definition 2020-05-02 18:28:38 +01:00
Matan Rosenberg
3b4c24af4e vendor: update github.com/spf13/cobra to v1.0.0 2020-04-29 16:19:35 +01:00
Nick Craig-Wood
84369286df vendor: add github.com/aalpar/deheap 2020-03-16 15:50:04 +00:00
Robert-André Mauchin
e2e400e63c Use proper import path go.etcd.io/bbolt 
Signed-off-by: Robert-André Mauchin <zebob.m@gmail.com>
 2020-03-03 12:40:52 +00:00
Gary Kim
3fd38cbe8d vendor: add github.com/prometheus/client_golang/prometheus 
Signed-off-by: Gary Kim <gary@garykim.dev>
 2020-03-01 09:58:34 +00:00
Nick Craig-Wood
abb9f89f65 vendor: update all dependencies 2020-02-26 21:26:46 +00:00
Nick Craig-Wood
c38d7be373 vendor: add github.com/hanwen/go-fuse/v2@master for mount2 2020-02-11 14:28:13 +00:00
Nick Craig-Wood
18d26e2ddb Revert "vendor: update x/crypto/ssh - to fix Windows password length issues fixes #3798" 
This turned out to introduce a regression, not being able to press Enter.

See: #3888 and https://github.com/golang/go/issues/36609

This reverts commit 251cfc100e.
 2020-01-20 12:46:52 +00:00
Nick Craig-Wood
36157d8ae5 vendor: update t3rm1n4l/go-mega - fixes mega: couldn't login: crypto/aes: invalid key size 0 
Fixes #3740
 2020-01-17 21:42:32 +00:00
Nick Craig-Wood
251cfc100e vendor: update x/crypto/ssh - to fix Windows password length issues fixes #3798 2020-01-17 21:33:47 +00:00
Kuang-che Wu
3801b8109e vendor: update termbox-go to fix ncdu command on FreeBSD 
see 58d4fcbce2
 2020-01-12 14:20:12 +00:00
Nick Craig-Wood
92662baceb vendor: update github.com/t3rm1n4l/go-mega to fix mega "illegal base64 data at input byte 22" 
Thanks to Ajaja for figuring this out.

See: https://forum.rclone.org/t/problem-to-login-with-mega/12276
 2020-01-11 16:47:06 +00:00
Nick Craig-Wood
21b17b14a9 vendor: update bazil.org/fuse to fix FreeBSD 12.1 - fixes #3697 2020-01-05 16:35:30 +00:00
Nick Craig-Wood
479c803fd9 vendor: update all dependencies 2019-11-14 21:51:34 +00:00
Nick Craig-Wood
3dcf1e61cf cache: follow move of upstream library github.com/coreos/bbolt github.com/etcd-io/bbolt 2019-11-14 21:51:34 +00:00
Xiaoxing Ye
6d58d9a86f vendor: change goftp/server url 
Closing #3674
 2019-10-29 17:41:56 +00:00
Nick Craig-Wood
1833167d10 vendor: run go mod tidy / go mod vendor with go1.13 2019-10-08 19:59:47 +01:00
Nick Craig-Wood
45e440d356 vendor: remove github.com/Azure/go-ansiterm 2019-10-08 19:59:47 +01:00
Nick Craig-Wood
c78d1dd18b vendor: add github.com/mattn/go-colorable 2019-10-08 19:59:47 +01:00
David
643192b347 vendor: add pkcs8 helpers for decrypting encrypted private keys 2019-09-20 17:15:16 +01:00
Nick Craig-Wood
ef7b001626 build: update to use go1.13 for the build 2019-09-04 10:23:48 +01:00
Nick Craig-Wood
af192d2507 vendor: update all dependencies 2019-08-26 18:00:17 +01:00
Cenk Alti
566aa0fca7 vendor: add github.com/putdotio/go-putio for putio client 2019-08-23 12:11:36 +01:00
Nick Craig-Wood
a72ef7ca0e vendor: update github.com/yunify/qingstor-sdk-go to v3 2019-08-14 16:15:34 +01:00
Nick Craig-Wood
57d5de6fba build: fix up package paths after repo move 
git grep -l github.com/ncw/rclone | xargs -d'\n' perl -i~ -lpe 's|github.com/ncw/rclone|github.com/rclone/rclone|g'
goimports -w `find . -name \*.go`
 2019-07-28 18:47:38 +01:00
justinalin
a3449bda30 vendor: add github.com/sirupsen/logrus 2019-07-28 12:05:50 +01:00
jaKa
3df9dbf887 vendor: updated github.com/koofr/go-koofrclient for set mtime support. 2019-07-24 21:11:58 +01:00
Nick Craig-Wood
5df5a3b78e vendor: tidy go.mod and go.sum - fixes #3317 2019-07-02 09:47:00 +01:00
Laura
2d0e9885bd vendor: add jzelinskie/whirlpool 2019-06-30 18:35:01 +01:00
Nick Craig-Wood
d61ba7ef78 vendor: update all dependencies 2019-06-27 13:52:32 +01:00
Gary Kim
2655bea86f vendor: add github.com/atotto/clipboard 2019-06-26 16:49:53 +01:00
Gary Kim
08bf8faa2f vendor: update github.com/jlaffaye/ftp 2019-06-26 16:42:12 +01:00
Gary Kim
66b3795eb8 vendor: update github.com/jlaffaye/ftp 2019-06-09 16:06:39 +01:00
Nick Craig-Wood
b7800e96d7 vendor: update golang.org/x/net/webdav - fixes #3002 
This fixes duplicacy working with rclone serve webdav
 2019-06-07 11:54:57 +01:00
Nick Craig-Wood
da3b685cd8 vendor: update github.com/pkg/sftp to fix sftp client issues 
See: https://forum.rclone.org/t/failed-to-copy-sftp-folder-not-found-c-ftpsites-ssh-fx-failure/9778
See: https://github.com/pkg/sftp/issues/288
 2019-05-24 15:03:23 +01:00
Nick Craig-Wood
1fefa6adfd vendor: add github.com/mitchellh/go-homedir 2019-05-11 23:39:04 +01:00
Nick Craig-Wood
8f89b03d7b vendor: update github.com/t3rm1n4l/go-mega and dependencies 
This is to fix a crash reported in #3140
 2019-05-01 16:09:50 +01:00
Nick Craig-Wood
613a9bb86b vendor: update all dependencies 2019-04-15 20:12:56 +01:00
Nick Craig-Wood
b0380aad95 vendor: update github.com/ncw/swift to help with #3041 2019-03-25 13:41:34 +00:00
Nick Craig-Wood
595fea757d vendor: update github.com/ncw/swift to bring in Expires changes 2019-03-18 13:30:59 +00:00
Nick Craig-Wood
1318be3b0a vendor: update github.com/goftp/server to fix hang while reading a file from the server 
See: https://forum.rclone.org/t/minor-issue-with-linux-ftp-client-and-rclone-ftp-access-denied/8959
 2019-03-17 09:30:57 +00:00
jaKa
1d14e30383 vendor: add github.com/koofr/go-koofrclient 
* added koofr client SDK dep for koofr backend
 2019-03-06 13:41:43 +00:00
Nick Craig-Wood
5991315990 vendor: update github.com/spf13/cobra to master 2019-02-13 23:02:22 +00:00
Nick Craig-Wood
43bc381e90 vendor: update all dependencies 2019-02-10 20:55:16 +00:00
Nick Craig-Wood
fd370fcad2 vendor: update github.com/t3rm1n4l/go-mega to add new error codes 2019-02-05 17:22:28 +00:00
Nick Craig-Wood
7d5d6c041f vendor: update github.com/t3rm1n4l/go-mega to fix v2 account login 
Fixes #2771
 2019-02-04 17:33:15 +00:00
kayrus
38c0018906 Bump github.com/ncw/swift to v1.0.44 2019-01-29 14:43:10 +00:00
Nick Craig-Wood
8d16a5693c vendor: update github.com/goftp/server - fixes #2845 2019-01-12 17:09:11 +00:00
nicolov
5edfd31a6d vendor: add github.com/anacrolix/dms 2019-01-09 19:14:14 +00:00
Nick Craig-Wood
039e2a9649 vendor: pull in github.com/ncw/swift latest to fix reauth on big files 2018-12-28 09:23:57 +00:00
Nick Craig-Wood
1c01d0b84a vendor: update dropbox SDK to fix failing integration tests #2829 2018-12-26 15:17:03 +00:00
Jay
f7b08a6982 vendor: add github.com/shurcooL/vfsgen 2018-12-26 15:07:21 +00:00
Nick Craig-Wood
63b51c6742 vendor: add golang.org/x/sync as a dependency 2018-12-13 10:45:52 +00:00
Mathieu Carbou
4138d5aa75 Issue #1788: Pointing to Dropbox's v5.0.0 tag 2018-12-05 22:03:30 +00:00
Nick Craig-Wood
89625e54cf vendor: update dependencies to latest 2018-11-26 14:10:33 +00:00
Nick Craig-Wood
c514cb752d vendor: update to latest versions of everything 2018-10-25 17:06:33 +01:00
Nick Craig-Wood
467fe30a5e vendor: update to latest versions of everything 2018-09-21 18:23:37 +01:00
Nick Craig-Wood
a14f0d46d7 vendor: add github.com/goftp/server 2018-09-17 08:50:34 +01:00
sandeepkru
233507bfe0 vendor: Update AzureSDK version to latest one, fixes failing integration tests 2018-09-12 08:14:38 +01:00
Nick Craig-Wood
da1682a30e vendor: switch to using go1.11 modules 2018-08-28 16:08:48 +01:00