Commit graph

3750 commits

Author SHA1 Message Date
Nick Craig-Wood
0baafb158f mount: allow files of unkown size to be read properly
Before this change, files of unknown size (eg Google Docs) would
appear in file listings with 0 size and would only allow 0 bytes to be
read.

This change sets the direct_io flag in the FUSE return which bypasses
the cache for these files.  This means that they can be read properly.

This is compatible with some, but not all applications.
2019-09-14 13:22:33 +01:00
Nick Craig-Wood
ba121eddf0 vfs: make objects of unknown size readable through the VFS
These objects (eg Google Docs) appear with 0 length in the VFS.

Before this change, these only read 0 bytes.

After this change, even though the size appears to be 0, the objects
can be read to the end.  If the objects are read to the end then the
size on the handle will be updated.
2019-09-14 13:09:07 +01:00
Nick Craig-Wood
2e80e035c9 fstest/mockobject: add UnknownSize() method to make Size() return -1 2019-09-14 13:07:01 +01:00
Nick Craig-Wood
ea9b6087cf fstest/mockfs: allow fs.Objects to be added to the root 2019-09-14 13:05:36 +01:00
Nick Craig-Wood
6959c997e2 config: remove error: can't use --size-only and --ignore-size together.
As sometimes they are useful together, for example when the remote
changes the sizes of the uploaded files.

See: https://forum.rclone.org/t/files-upload-will-be-auto-compress-how-do-i-sync-a-file-to-remote/10578
2019-09-14 10:13:44 +01:00
Nick Craig-Wood
25786cafd3 s3: fix SetModTime on GLACIER/ARCHIVE objects and implement set/get tier
- Read the storage class for each object
- Implement SetTier/GetTier
- Check the storage class on the **object** before using SetModTime

This updates the fix in 1a2fb52 so that SetModTime works when you are
using objects which have been migrated to GLACIER but you aren't using
GLACIER as a storage class.

Fixes #3522
2019-09-14 09:18:55 +01:00
Nick Craig-Wood
23dc313fa5 azureblob: add missing type assertions for GetTier/SetTier 2019-09-14 09:18:55 +01:00
Nick Craig-Wood
1a16849df0 http: fix race introduced in 7982aaf151 2019-09-14 08:48:13 +01:00
Nick Craig-Wood
3b68340eac http: add --http-no-head to stop rclone doing HEAD in listings #3523 2019-09-14 00:17:39 +01:00
Nick Craig-Wood
7982aaf151 http: HEAD directory entries in parallel to speedup #3523 2019-09-14 00:16:44 +01:00
Nick Craig-Wood
7b29ed8ec1 Add Lars Lehtonen to contributors 2019-09-13 23:50:54 +01:00
Lars Lehtonen
c93e0ff8ee rest: fix missing error check 2019-09-13 23:50:39 +01:00
Nick Craig-Wood
3b91fb6a2f Add David Baumgold to contributors 2019-09-13 18:39:36 +01:00
David Baumgold
7d8c15c030 docs: GitHub has a capital H 2019-09-13 18:39:23 +01:00
Nick Craig-Wood
bfbddab46b fs/accounting: Fix "file already closed" on transfer retries
This was caused by the recent reworking of the accounting interface.
The Transfer object was recycling the Accounting object without
resetting the stream.

See: https://forum.rclone.org/t/error-file-already-closed/11469/
See: https://forum.rclone.org/t/rclone-b2-sync-post-error-method-not-supported/11718/
2019-09-13 18:35:02 +01:00
Nick Craig-Wood
e09a4ff019 cmd: Make --progress work in git bash on Windows - fixes #3531
This detects the presence of a VT100 terminal by using the TERM
environment variable and switches to using VT100 codes directly under
windows if it is found.

This makes --progress work correctly with git bash.
2019-09-13 15:24:47 +01:00
Nick Craig-Wood
48e23d8c85 build: fix circleci build to use billziss/xgo-cgofuse container
Use built rclone to do upload
2019-09-12 21:59:50 +01:00
Aleksandar Jankovic
934440a9df accounting: fix total duration calculation
Fixes: #3498
2019-09-12 10:29:40 +01:00
Nick Craig-Wood
29b4f211ab gcs: add context to SDK calls #3257 2019-09-09 23:27:07 +01:00
Nick Craig-Wood
bd863f8868 drive: add context to SDK calls #3257 2019-09-09 23:27:07 +01:00
Nick Craig-Wood
66c23723e3 Add context to all http.NewRequest #3257
When we drop support for go1.12 we can use http.NewRequestWithContext
2019-09-09 23:27:07 +01:00
Nick Craig-Wood
58a531a203 rest: add context propagation to rest library #3257
This fixes up the calling and propagates the contexts for the backends
which use lib/rest.
2019-09-09 23:27:07 +01:00
Ivan Andreev
ba1daea072 mailru: backend for mail.ru 2019-09-09 21:56:16 +01:00
Ivan Andreev
bdcd0b4c64 Add mailru hash (mrhash) 2019-09-09 21:34:15 +01:00
Nick Craig-Wood
94eb9a4014 Start v1.49.2-DEV development 2019-09-08 17:36:21 +01:00
Nick Craig-Wood
e028c006fc test_all: write index.json and add branch, commit and Go version to report 2019-09-08 11:35:56 +01:00
Nick Craig-Wood
3f3f038b73 build: make sure we add version info to test_all build 2019-09-08 11:35:36 +01:00
calisro
2298834e83 docs: spell out google photos download limitations 2019-09-07 11:47:53 +01:00
Nick Craig-Wood
07dfb3aa11 bin: convert python scripts to python3 2019-09-06 22:55:28 +01:00
Danil Semelenov
1382dba3c8 cmd: make autocomplete compatible with bash's posix mode #3489 2019-09-06 13:11:08 +01:00
Nick Craig-Wood
f1347139fa config: check config names more carefully and report errors - fixes #3506
Before this change it was possible to make a remote with an invalid
name in the config file, either manually or with `rclone config
create` (but not with `rclone config`).

When this remote was used, because it was invalid, rclone would
presume this remote name was a local directory for a very suprising
user experience!

This change checks remote names more carefully and returns errors
- when the user tries to use an invalid remote name on the command line
- when an invalid remote name is used in `rclone config create/update/password`
- when the user tries to enter an invalid remote name in `rclone config`

This does not prevent the user entering a remote name with invalid
characters in the config manually, but such a remote will fail
immediately when it is used on the command line.
2019-09-06 12:07:09 +01:00
Nick Craig-Wood
27a730ef8f docs: Update changelog and RELEASE.md from v1.49.1 release 2019-09-06 12:07:09 +01:00
Ivan Andreev
d0c6e5cf5a vfs: skip TestCaseSensitivity on case insensitive backends 2019-09-06 10:44:59 +01:00
Nick Craig-Wood
cf9b973fe4 accounting: fix locking in Transfer to avoid deadlock with --progress
Before this change, using -P occasionally deadlocked on the Transfer
mutex when Transfer.Done() was called with a non nil error and the
StatsInfo mutex since they mutually call each other.

This was fixed by making sure that the Transfer mutex is always
released before calling any StatsInfo methods.

This improves on: 6f87267b34

Fixes #3505
2019-09-06 10:00:44 +01:00
Nick Craig-Wood
ffa1dac10b build: apply gofmt from go1.13 to change case of number literals 2019-09-05 13:59:06 +01:00
Nick Craig-Wood
7b0966880e Add Ivan Andreev to contributors 2019-09-04 21:32:16 +01:00
Ivan Andreev
1c4e33d4ad vfs: add flag --vfs-case-insensitive for windows/macOS mounts
rclone mount when run on Windows & macOS will now default to `--vfs-case-insensitive`.
This means that
2019-09-04 21:30:48 +01:00
Nick Craig-Wood
530ba66d35 operations: fix -u/--update with google photos / files of unknown size
Before this change if -u/--update was in effect we compared the size
of the files to see if the transfer should go ahead.  This was
comparing -1 with an actual size so the transfer always proceeded.

After this change we use the existing `sizeDiffers` function which
does the correct comparison with -1 for files of unknown length.

See: https://forum.rclone.org/t/sync-with-google-photos-to-local-drive-will-result-in-recoping/11605
2019-09-04 17:31:17 +01:00
Danil Semelenov
b3db38ae31 Disable __rclone_custom_func if posix mode is on
A workaround for #3489. Code in `__rclone_custom_func` relies on process substitutions `<(...)` to preserve changes of variables within `while` bodies, which is not supported in the posix mode.
2019-09-04 14:48:10 +01:00
Danil Semelenov
c0d1869204 Fix 'compopt: command not found' on autocomplete on macOS
As reported in #3489.
2019-09-04 14:47:26 +01:00
Nick Craig-Wood
89b6d89077 build: drop support for go1.9 2019-09-04 10:23:48 +01:00
Nick Craig-Wood
ef7b001626 build: update to use go1.13 for the build 2019-09-04 10:23:48 +01:00
Nick Craig-Wood
f97a3e853e Add Alfonso Montero to contributors 2019-09-03 17:26:13 +01:00
Denis
b71ac141cc copyurl: add --auto-filename flag for using file name from url in destination path (#3451) 2019-09-03 17:25:19 +01:00
Nick Craig-Wood
5932acfee3 rc: fix docs for config/create /update /password 2019-09-03 08:34:15 +01:00
Alfonso Montero
e2ce687f93 README.md: Add Docker pulls badge 2019-09-02 15:14:45 +01:00
Nick Craig-Wood
a3fb460c6b docs: add info on how to build and use the docker images 2019-09-02 14:30:11 +01:00
Nick Craig-Wood
8d296d0e1d Start v1.49.1-DEV development 2019-09-02 13:10:47 +01:00
Nick Craig-Wood
20a57aaccb gcs: fix need for elevated permissions on SetModTime - fixes #3493
Before this change we used PATCH on the object to update the metadata.

Apparently this requires the "full_control" scope which Google were
unhappy with in their oauth review.

This changes it to update the metadata by copying the object ontop of
itself (which is the way s3 works).  This can be done with normal
permissions.
2019-09-02 09:26:33 +01:00
Nick Craig-Wood
50a4ed8fc4 operations: fix accounting for server side copies
See: https://forum.rclone.org/t/b2-server-side-copy-doesnt-show-cumulative-progress/11154
2019-09-02 09:26:33 +01:00