Before this change `rclone rc --loopback` would give the error "bad
JSON".
This was because the output of the `rc/list` command was not serialzed
through JSON.
This serializes it through JSON and fixes that (and probably other)
command.
rc: WebGUI should check for new update only when rc-web-gui-update is specified or not already downloaded.
rc: change permission to 0755 instead of 755 to prevent unexpected behaviour.
Before this change, using -P occasionally deadlocked on the progress
mutex and the stats mutex since they call each other.
This is fixed by shortening the locking window in the progress routine
so as not to include the stats calculation.
This was factored from fstest as we were including the testing
enviroment into the main binary because of it.
This was causing opening the browser to fail because of 8243ff8bc8.
This adds experimental support for web gui integration so that rclone can fetch and run a web based GUI using the --rc-web-ui and related flags.
It downloads and caches a webui zip file which it then unpacks and opens in the browser.
Introduce stats groups that will isolate accounting for logically
different transferring operations. That way multiple accounting
operations can be done in parallel without interfering with each other
stats.
Using groups is optional. There is dedicated global stats that will be
used by default if no group is specified. This is operating mode for CLI
usage which is just fire and forget operation.
For running rclone as rc http server each request will create it's own
group. Also there is an option to specify your own group.
This is done to make clear ownership over accounting object and prepare
for removing global stats object.
Stats elapsed time calculation has been altered to account for actual
transfer time instead of stats creation time.
This was started by Fionera, finished off by Laura with fixes and more
docs from Nick.
Co-authored-by: Fionera <fionera@fionera.de>
Co-authored-by: Nick Craig-Wood <nick@craig-wood.com>
In f544234 we removed the global flags from each command as it was
making each page very big and causing 1000s of lines of duplication in
the man page.
This change adds a new flags page with all the global flags on and
links each command page to it.
Fixes#3273
- Change rclone/fs interfaces to accept context.Context
- Update interface implementations to use context.Context
- Change top level usage to propagate context to lover level functions
Context propagation is needed for stopping transfers and passing other
request-scoped values.
Before this change using --user and --pass was impossible on the rc
from a browser as the browser needed to make the OPTIONS request first
before sending Authorization: headers, but the OPTIONS request
required an Authorization: header.
After this change we allow OPTIONS requests to go through without
checking the Authorization: header.
Before this change when using "rclone config create" it wasn't
possible to add passwords in one go, it was necessary to call "rclone
config password" to add the passwords afterwards as "rclone config
create" didn't obscure passwords.
After this change "rclone config create" and "rclone config update"
will obscure passwords as necessary as will the corresponding API
calls config/create and config/update.
This makes "rclone config password" and its API config/password
obsolete, however they will be left for backwards compatibility.
Mostly trying to get logging to happen through rclone's log methods.
Added request logging, and a trace parameter that will dump the
entire request/response for debugging when dealing with poorly
written clients.
Also added a flag to specify the device's "Friendly Name" explicitly,
and made an attempt at allowing mime types in addition to video.
Again, mostly just copying what I see in other implementations. This
does seem to have done the trick so that I can now pause, fast forward,
rewind, etc., on my Samsung F series.
Brings in icons for devices to display. Based on what some
other open implementations have done, it's worth having a simple
stub implmentation of ConnectionManagerService. Advertise
X_MS_MediaReceiverRegistrar as well, which sounds like it
is necessary for certain MSFT devices (like the X-Box.)
For various reasons, it seems to make sense to move away from generating
the XML with objects. Namespace support is minimal in go, the objects we
have are in an upstream project, and some subtitlties seem likely to
cause problems with poorly written clients.
This removes the empty <iconList></iconList>, but is otherwise the
same output.
Before this change, rclone didn't report errors for commands which
didn't return an error directly. For example `rclone ls` could
encounter an error and rclone would log nothing, even though the exit
code was non zero.
After this change we always log a message if we are exiting with a
non-zero exit code.
Tests have been randomly failing with messages like
listen tcp 127.0.0.1:51778: bind: address already in use
Rework all the test servers so they choose a random free port on
startup and use that for the tests to avoid.
The previous behavior of the remotes completion was that only
alphanumeric characters were allowed in a remote name. This limitation
has been lifted somewhat by #2985, which also allowed an underscore.
With the new implementation introduced in this commit, the completion of
the remote name has been simplified: If there is no colon (":") in the
current word, then complete remote name. Otherwise, complete the path
inside the specified remote. This allows correct completion of all
remote names that are allowed by the config (including - and _).
Actually it matches much more than that, even remote names that are not
allowed by the config, but in such a case there already would be a wrong
identifier in the configuration file.
With this simpler string comparison, we can get rid of the regular
expression, which makes the completion multiple times faster. For a
sample benchmark, try the following:
# Old way
$ time bash -c 'for _ in {1..1000000}; do
[[ remote:path =~ ^[[:alnum:]]*$ ]]; done'
real 0m15,637s
user 0m15,613s
sys 0m0,024s
# New way
$ time bash -c 'for _ in {1..1000000}; do
[[ remote:path != *:* ]]; done'
real 0m1,324s
user 0m1,304s
sys 0m0,020s
The UPnP MediaServer spec says that the ConnectionManager service is
required, and adding it was enough to get dlna support working on my
other TV (LG webOS 2.2.1).
The SCPD URL was being set after marshalling the XML, and thus coming
out blank. Now works on my Samsung TV, and likely fixes some issues
reported by others in #2648.
This brings it up to par with lsjson.
This commit also reworks the framework to use ListJSON internally
which removes duplicated code and makes testing easier.
Before this change, rclone used the `__custom_func` hook to control
the completions of remote files. However this clashes with other
cobra users, the most notable example being kubectl.
Upgrading cobra to master allows us to use a namespaced function
`__rclone_custom_func` which fixes the problem.
Fixes#1529
* drive: don't run teamdrive config if auto confirm set
* onedrive: don't run extra config if auto confirm set
* make Confirm results customisable by config
Fixes#1010
The --no-traverse flag was not implemented when the new sync routines
(using the march package) was implemented.
This re-implements --no-traverse in march by trying to find a match
for each object with NewObject rather than from a directory listing.
delete files by pressing 'd' in the ncdu listing
GUI Improvements:
Boxes now have a border around them
Boxes can ask questions and allow the selection of options. The
selected option will be given to the UI.boxMenuHandler function.
Fixes#2571
Remove files/directories from the in memory structs of the cloud
directory. Size and Count will be recalculated and populated upwards
to the parent directories.
Instead of showing all flags/backends all the time, you can type
rclone help flags
rclone help flags <regexp>
rclone help backends
rclone help backend <name>
Before this fix rclone didn't wait for the stats to be finished before
exiting, so the final new line was never printed.
After this change rclone will wait for the stats routine to cease
before exiting.
Before this change x/net/webdav would open each file to find out its
Content-Type.
Now we override the FileInfo and provide that directly from rclone.
An --etag-hash has also been implemented to override the ETag with the
hash passed in.
Fixes#2273
OSX FUSE only supports 32 bit number of blocks which means that block
counts have been wrapping. This causes f_bavail to be 0 which in turn
causes problems with programs like borg backup.
Fixes#2356
By default the timeout is 60s which isn't long enough for long
transactions. The symptoms are rclone just quitting for no reason.
Supplying the --daemon-timeout flag fixes this causing the kernel to
wait longer for rclone.
Before this change, boolean flags such as `--b2-hard-delete` were
failing to be recognised unless they had a parameter.
This bug was introduced as part of the config re-organisation:
f3f48d7d49
This unifies the 3 methods of reading config
* command line
* environment variable
* config file
And allows them all to be configured in all places. This is done by
making the []fs.Option in the backend registration be the master
source of what the backend options are.
The backend changes are:
* Use the new configmap.Mapper parameter
* Use configstruct to parse it into an Options struct
* Add all config to []fs.Option including defaults and help
* Remove all uses of pflag
* Remove all uses of config.FileGet
Before this copyto would parse windows paths incorrectly.
This change moves the parsing code into fspath and makes sure
fspath.Split calls fspath.Parse which does the parsing correctly for
This also renames fspath.RemoteParse to fspath.Parse for consistency
--one-way argument will check that all files on source matches the files on detination,
but not the other way. For example files present on destination but not on source will not
trigger an error.
Fixes: #1526
When running `rclone mount`, there were 2 signal handlers for `os.Interrupt`.
Those handlers would run concurrently and in some cases cause either unmount or `atexit.Run()` being skipped.
In addition `atexit.Run()` will get called in `resolveExitCode` to ensure cleanup on errors.
* Add Mkdir, Rmdir, Purge, Delete, SetModTime, Copy, Move, DirMove
* Update file size after upload
* Add Open seek
* Set private permission for new folder and uploaded file
* Add docs
* Update List function
* Fix UserSessionInfo struct
* Fix socket leaks
* Don’t close resp.Body in Open method
* Get hash when listing files
Before this change rclone would inefficiently and confusingly read all
the files in the source directory when copy or moving a single file.
This caused confusion for the users to see log messages about files
which weren't part of the sync.
After the change the copy and move commands use the new infrastructure
made for the copyto and moveto command for single file copy and move.
Before this change we would unconditionally set the OSXFUSE options
noappledouble and noapplexattr.
However the noapplexattr options caused problems with copies in the
Finder.
Now the default for noapplexattr is false so we don't add the option
by default and the user can override the defaults using the
--noappledouble and --noapplexattr flags.
Before this change rclone would set the volume name from the
remote:path normally. However this has `:` and `/` in which make it
difficult to use in macOS.
Now rclone will remove the special characters and replace them with
spaces. It also allows the volume name to be set with the --volname
flag.
By default bazil fuse will return ENOTSUPP for these. However if we
return ENOSYS then OSXFUSE (at least) will never call them again
saving round trips though fuse.
List or ListR of an non existent directory must return
ErrorDirNotFound for non bucket based remotes. For bucket based
remotes it may return ErrorDirNotFound or it may return no error and
no entries.
* Implement about for:
* local, crypt, cache, drive, swift, hubic, onedrive, pcloud, dropbox
* Implement `--json` and `---full` flag for `rclone about`
* change About interface to return a Usage structure
* Remove operations.About as it is too thin an interface
* Implement Integration test
Relates to #1138 and #1564
This stops the cache cleaner running unnecessarily and saves
resources.
This also helps with issue #2227 which was caused by a second mount
deleting objects in the first mounts cache.
The written out list of tests was replaced with a nested test for
mount and cmount. The tests for each VFS cache mode were also replaced
with nested tests which makes the output and the code much cleaner.
This means the bandwidth stats will be correct and the bandwidth
throttling will work correctly. This was forgotten as a previous
iteration of the code was using the higher level operations.Rcat which
took care of this.
Previously this was adding it in to all file opens which was causing
inefficiencies under Windows where it stats the file using
open/fstat/close.
This change will make stat operations run much quicker under Windows
as they won't have to open the underlying file
This problem was introduced in61b6159a05336bd7ba105766de2d2ff171f7fb81
where we added O_CREATE to all file opens and creates.
This implements a remote control protocol activated with the --rc flag
and a new command `rclone rc` to use that interface.
Still to do
* docs - need finishing
* tests
This flag allows the attribute caching in the kernel to be controlled.
The default is 0s - no caching - which is recommended for filesystems
which can change outside the control of the kernel.
Previously this was at the default meaning it was 60s for mount and 1s
for cmount. This showed strange effects when files changed on the
remote not via the kernel. For instance Caddy would serve corrupted
files for a while when serving from an rclone mount when a file
changed on the remote.
Before this change Open("name", os.O_RDONLY|os.O_TRUNC) would have
truncated the file. This is what Linux does, but is counterintuitive.
POSIX states this is undefined, so return an error in this case
instead. This preserves the invariant O_RDONLY => file is not
changed.
Change absolute binary paths in scripts to /usr/bin/env or make them
relative.
This allows the scripts to be used on linux distributions
like NixOS, where binaries are not located in /usr/ or /bin/.
The purpose of this is to make it easier to maintain and eventually to
allow the rclone backends to be re-used in other projects without
having to use the rclone configuration system.
The new code layout is documented in CONTRIBUTING.
The filesystem does a certain amount of things asynchronously waiting
for the file to be released after writing it means everything should
be in a consistent state.
On Windows the fuse.O_* flags do not have the same values as the
os.O_* flags so translate between the two representations. They are
mostly the same which is why this hasn't caused a problem before.
* Tests for VFS layer
* Small fixes found during testing
* Fix Close, Flush and Release behaviour for ReadFileHandle and WriteFileHandle
* Fix nil object bugs on File
Previously config sub commands were manually parsed rather than using
cobra.
Make config command have the following sub commands:
* create Create a new remote with name, type and options.
* delete Delete an existing remote <name>.
* dump Dump the config file as JSON.
* edit Enter an interactive configuration session.
* file Show path of configuration file in use.
* providers List in JSON format all the providers and options.
* show Print (decrypted) config file, or the config for a single remote.
* update Update options in an existing remote.
The following changes were made to existing commands
* listproviders was renamed to providers
* listoptions was removed in favour of providing the output in providers
* jsonconfig was renamed to create
* an optional parameter was added to the show command
* Fixup bitrot (rclone and Azure library)
* Implement Copy
* Add modtime to metadata under mtime key as RFC3339Nano
* Make multipart upload work
* Make it pass the integration tests
* Fix uploading of zero length blobs
* Rename to azureblob as it seems likely we will do azurefile
* Add docs
Add new package qingstor to support QingStor API.
Add new unit test for its and tested through; But I commented
on some tests case because of some of the features of QingStor.
Add new docs for it.
In certain circumstances if an upload failed then the mount could hang
indefinitely. This was fixed by closing the read pipe after the Put
completed. This will cause the write side to return a pipe closed
error fixing the hang.
Fixes#1498
* Fix remaining problems
* Refactor to make testing easier and add a test suite
* Make path parsing more robust.
* Add single file operations
* Add MimeType reading for objects
* Add documentation
* Note go1.7+ is required to build
This is a fix left over from the v2 conversion. Dropbox ignores the
client modification on an incoming file if it was identical to the
existing file. This change deletes the existing file first before
re-uploading the new one.
* add support to hashing module
* add dbhashsum to list the hashes
* add support to dropbox module
This means objects up and downloaded to/from Dropbox will have their
hashes checked.
Note after this change local objects are calculating MD5, SHA1 and
DBHASH which is excessive and needs to be fixed.
Normally mount/cmount use `-o ro` to get the kernel to mark the fs as
read only. However this is ignored by WinFsp, so in addition if
`--read-only` is in effect then return EROFS ("Read only File System")
from all methods which attempt to modify something.
