382 lines
11 KiB
Markdown
382 lines
11 KiB
Markdown
---
|
|
title: "Remote Control"
|
|
description: "Remote controlling rclone"
|
|
date: "2018-03-05"
|
|
---
|
|
|
|
# Remote controlling rclone #
|
|
|
|
If rclone is run with the `--rc` flag then it starts an http server
|
|
which can be used to remote control rclone.
|
|
|
|
**NB** this is experimental and everything here is subject to change!
|
|
|
|
## Supported parameters
|
|
|
|
#### --rc ####
|
|
Flag to start the http server listen on remote requests
|
|
|
|
#### --rc-addr=IP ####
|
|
IPaddress:Port or :Port to bind server to. (default "localhost:5572")
|
|
|
|
#### --rc-cert=KEY ####
|
|
SSL PEM key (concatenation of certificate and CA certificate)
|
|
|
|
#### --rc-client-ca=PATH ####
|
|
Client certificate authority to verify clients with
|
|
|
|
#### --rc-htpasswd=PATH ####
|
|
htpasswd file - if not provided no authentication is done
|
|
|
|
#### --rc-key=PATH ####
|
|
SSL PEM Private key
|
|
|
|
#### --rc-max-header-bytes=VALUE ####
|
|
Maximum size of request header (default 4096)
|
|
|
|
#### --rc-user=VALUE ####
|
|
User name for authentication.
|
|
|
|
#### --rc-pass=VALUE ####
|
|
Password for authentication.
|
|
|
|
#### --rc-realm=VALUE ####
|
|
Realm for authentication (default "rclone")
|
|
|
|
#### --rc-server-read-timeout=DURATION ####
|
|
Timeout for server reading data (default 1h0m0s)
|
|
|
|
#### --rc-server-write-timeout=DURATION ####
|
|
Timeout for server writing data (default 1h0m0s)
|
|
|
|
## Accessing the remote control via the rclone rc command
|
|
|
|
Rclone itself implements the remote control protocol in its `rclone
|
|
rc` command.
|
|
|
|
You can use it like this
|
|
|
|
```
|
|
$ rclone rc rc/noop param1=one param2=two
|
|
{
|
|
"param1": "one",
|
|
"param2": "two"
|
|
}
|
|
```
|
|
|
|
Run `rclone rc` on its own to see the help for the installed remote
|
|
control commands.
|
|
|
|
## Supported commands
|
|
<!--- autogenerated start - run make rcdocs - don't edit here -->
|
|
### cache/expire: Purge a remote from cache
|
|
|
|
Purge a remote from the cache backend. Supports either a directory or a file.
|
|
Params:
|
|
- remote = path to remote (required)
|
|
- withData = true/false to delete cached data (chunks) as well (optional)
|
|
|
|
Eg
|
|
|
|
rclone rc cache/expire remote=path/to/sub/folder/
|
|
rclone rc cache/expire remote=/ withData=true
|
|
|
|
### cache/stats: Get cache stats
|
|
|
|
Show statistics for the cache remote.
|
|
|
|
### core/bwlimit: Set the bandwidth limit.
|
|
|
|
This sets the bandwidth limit to that passed in.
|
|
|
|
Eg
|
|
|
|
rclone rc core/bwlimit rate=1M
|
|
rclone rc core/bwlimit rate=off
|
|
|
|
The format of the parameter is exactly the same as passed to --bwlimit
|
|
except only one bandwidth may be specified.
|
|
|
|
### core/gc: Runs a garbage collection.
|
|
|
|
This tells the go runtime to do a garbage collection run. It isn't
|
|
necessary to call this normally, but it can be useful for debugging
|
|
memory problems.
|
|
|
|
### core/memstats: Returns the memory statistics
|
|
|
|
This returns the memory statistics of the running program. What the values mean
|
|
are explained in the go docs: https://golang.org/pkg/runtime/#MemStats
|
|
|
|
The most interesting values for most people are:
|
|
|
|
* HeapAlloc: This is the amount of memory rclone is actually using
|
|
* HeapSys: This is the amount of memory rclone has obtained from the OS
|
|
* Sys: this is the total amount of memory requested from the OS
|
|
* It is virtual memory so may include unused memory
|
|
|
|
### core/pid: Return PID of current process
|
|
|
|
This returns PID of current process.
|
|
Useful for stopping rclone process.
|
|
|
|
### core/stats: Returns stats about current transfers.
|
|
|
|
This returns all available stats
|
|
|
|
rclone rc core/stats
|
|
|
|
Returns the following values:
|
|
|
|
```
|
|
{
|
|
"speed": average speed in bytes/sec since start of the process,
|
|
"bytes": total transferred bytes since the start of the process,
|
|
"errors": number of errors,
|
|
"checks": number of checked files,
|
|
"transfers": number of transferred files,
|
|
"deletes" : number of deleted files,
|
|
"elapsedTime": time in seconds since the start of the process,
|
|
"lastError": last occurred error,
|
|
"transferring": an array of currently active file transfers:
|
|
[
|
|
{
|
|
"bytes": total transferred bytes for this file,
|
|
"eta": estimated time in seconds until file transfer completion
|
|
"name": name of the file,
|
|
"percentage": progress of the file transfer in percent,
|
|
"speed": speed in bytes/sec,
|
|
"speedAvg": speed in bytes/sec as an exponentially weighted moving average,
|
|
"size": size of the file in bytes
|
|
}
|
|
],
|
|
"checking": an array of names of currently active file checks
|
|
[]
|
|
}
|
|
```
|
|
Values for "transferring", "checking" and "lastError" are only assigned if data is available.
|
|
The value for "eta" is null if an eta cannot be determined.
|
|
|
|
### rc/error: This returns an error
|
|
|
|
This returns an error with the input as part of its error string.
|
|
Useful for testing error handling.
|
|
|
|
### rc/list: List all the registered remote control commands
|
|
|
|
This lists all the registered remote control commands as a JSON map in
|
|
the commands response.
|
|
|
|
### rc/noop: Echo the input to the output parameters
|
|
|
|
This echoes the input parameters to the output parameters for testing
|
|
purposes. It can be used to check that rclone is still alive and to
|
|
check that parameter passing is working properly.
|
|
|
|
### vfs/forget: Forget files or directories in the directory cache.
|
|
|
|
This forgets the paths in the directory cache causing them to be
|
|
re-read from the remote when needed.
|
|
|
|
If no paths are passed in then it will forget all the paths in the
|
|
directory cache.
|
|
|
|
rclone rc vfs/forget
|
|
|
|
Otherwise pass files or dirs in as file=path or dir=path. Any
|
|
parameter key starting with file will forget that file and any
|
|
starting with dir will forget that dir, eg
|
|
|
|
rclone rc vfs/forget file=hello file2=goodbye dir=home/junk
|
|
|
|
### vfs/refresh: Refresh the directory cache.
|
|
|
|
This reads the directories for the specified paths and freshens the
|
|
directory cache.
|
|
|
|
If no paths are passed in then it will refresh the root directory.
|
|
|
|
rclone rc vfs/refresh
|
|
|
|
Otherwise pass directories in as dir=path. Any parameter key
|
|
starting with dir will refresh that directory, eg
|
|
|
|
rclone rc vfs/refresh dir=home/junk dir2=data/misc
|
|
|
|
If the parameter recursive=true is given the whole directory tree
|
|
will get refreshed. This refresh will use --fast-list if enabled.
|
|
|
|
<!--- autogenerated stop -->
|
|
|
|
## Accessing the remote control via HTTP
|
|
|
|
Rclone implements a simple HTTP based protocol.
|
|
|
|
Each endpoint takes an JSON object and returns a JSON object or an
|
|
error. The JSON objects are essentially a map of string names to
|
|
values.
|
|
|
|
All calls must made using POST.
|
|
|
|
The input objects can be supplied using URL parameters, POST
|
|
parameters or by supplying "Content-Type: application/json" and a JSON
|
|
blob in the body. There are examples of these below using `curl`.
|
|
|
|
The response will be a JSON blob in the body of the response. This is
|
|
formatted to be reasonably human readable.
|
|
|
|
If an error occurs then there will be an HTTP error status (usually
|
|
400) and the body of the response will contain a JSON encoded error
|
|
object.
|
|
|
|
The sever implements basic CORS support and allows all origins for that.
|
|
The response to a preflight OPTIONS request will echo the requested "Access-Control-Request-Headers" back.
|
|
|
|
### Using POST with URL parameters only
|
|
|
|
```
|
|
curl -X POST 'http://localhost:5572/rc/noop/?potato=1&sausage=2'
|
|
```
|
|
|
|
Response
|
|
|
|
```
|
|
{
|
|
"potato": "1",
|
|
"sausage": "2"
|
|
}
|
|
```
|
|
|
|
Here is what an error response looks like:
|
|
|
|
```
|
|
curl -X POST 'http://localhost:5572/rc/error/?potato=1&sausage=2'
|
|
```
|
|
|
|
```
|
|
{
|
|
"error": "arbitrary error on input map[potato:1 sausage:2]",
|
|
"input": {
|
|
"potato": "1",
|
|
"sausage": "2"
|
|
}
|
|
}
|
|
```
|
|
|
|
Note that curl doesn't return errors to the shell unless you use the `-f` option
|
|
|
|
```
|
|
$ curl -f -X POST 'http://localhost:5572/rc/error/?potato=1&sausage=2'
|
|
curl: (22) The requested URL returned error: 400 Bad Request
|
|
$ echo $?
|
|
22
|
|
```
|
|
|
|
### Using POST with a form
|
|
|
|
```
|
|
curl --data "potato=1" --data "sausage=2" http://localhost:5572/rc/noop/
|
|
```
|
|
|
|
Response
|
|
|
|
```
|
|
{
|
|
"potato": "1",
|
|
"sausage": "2"
|
|
}
|
|
```
|
|
|
|
Note that you can combine these with URL parameters too with the POST
|
|
parameters taking precedence.
|
|
|
|
```
|
|
curl --data "potato=1" --data "sausage=2" "http://localhost:5572/rc/noop/?rutabaga=3&sausage=4"
|
|
```
|
|
|
|
Response
|
|
|
|
```
|
|
{
|
|
"potato": "1",
|
|
"rutabaga": "3",
|
|
"sausage": "4"
|
|
}
|
|
|
|
```
|
|
|
|
### Using POST with a JSON blob
|
|
|
|
```
|
|
curl -H "Content-Type: application/json" -X POST -d '{"potato":2,"sausage":1}' http://localhost:5572/rc/noop/
|
|
```
|
|
|
|
response
|
|
|
|
```
|
|
{
|
|
"password": "xyz",
|
|
"username": "xyz"
|
|
}
|
|
```
|
|
|
|
This can be combined with URL parameters too if required. The JSON
|
|
blob takes precedence.
|
|
|
|
```
|
|
curl -H "Content-Type: application/json" -X POST -d '{"potato":2,"sausage":1}' 'http://localhost:5572/rc/noop/?rutabaga=3&potato=4'
|
|
```
|
|
|
|
```
|
|
{
|
|
"potato": 2,
|
|
"rutabaga": "3",
|
|
"sausage": 1
|
|
}
|
|
```
|
|
|
|
## Debugging rclone with pprof ##
|
|
|
|
If you use the `--rc` flag this will also enable the use of the go
|
|
profiling tools on the same port.
|
|
|
|
To use these, first [install go](https://golang.org/doc/install).
|
|
|
|
Then (for example) to profile rclone's memory use you can run:
|
|
|
|
go tool pprof -web http://localhost:5572/debug/pprof/heap
|
|
|
|
This should open a page in your browser showing what is using what
|
|
memory.
|
|
|
|
You can also use the `-text` flag to produce a textual summary
|
|
|
|
```
|
|
$ go tool pprof -text http://localhost:5572/debug/pprof/heap
|
|
Showing nodes accounting for 1537.03kB, 100% of 1537.03kB total
|
|
flat flat% sum% cum cum%
|
|
1024.03kB 66.62% 66.62% 1024.03kB 66.62% github.com/ncw/rclone/vendor/golang.org/x/net/http2/hpack.addDecoderNode
|
|
513kB 33.38% 100% 513kB 33.38% net/http.newBufioWriterSize
|
|
0 0% 100% 1024.03kB 66.62% github.com/ncw/rclone/cmd/all.init
|
|
0 0% 100% 1024.03kB 66.62% github.com/ncw/rclone/cmd/serve.init
|
|
0 0% 100% 1024.03kB 66.62% github.com/ncw/rclone/cmd/serve/restic.init
|
|
0 0% 100% 1024.03kB 66.62% github.com/ncw/rclone/vendor/golang.org/x/net/http2.init
|
|
0 0% 100% 1024.03kB 66.62% github.com/ncw/rclone/vendor/golang.org/x/net/http2/hpack.init
|
|
0 0% 100% 1024.03kB 66.62% github.com/ncw/rclone/vendor/golang.org/x/net/http2/hpack.init.0
|
|
0 0% 100% 1024.03kB 66.62% main.init
|
|
0 0% 100% 513kB 33.38% net/http.(*conn).readRequest
|
|
0 0% 100% 513kB 33.38% net/http.(*conn).serve
|
|
0 0% 100% 1024.03kB 66.62% runtime.main
|
|
```
|
|
|
|
Possible profiles to look at:
|
|
|
|
* Memory: `go tool pprof http://localhost:5572/debug/pprof/heap`
|
|
* 30-second CPU profile: `go tool pprof http://localhost:5572/debug/pprof/profile`
|
|
* 5-second execution trace: `wget http://localhost:5572/debug/pprof/trace?seconds=5`
|
|
|
|
See the [net/http/pprof docs](https://golang.org/pkg/net/http/pprof/)
|
|
for more info on how to use the profiling and for a general overview
|
|
see [the Go team's blog post on profiling go programs](https://blog.golang.org/profiling-go-programs).
|
|
|
|
The profiling hook is [zero overhead unless it is used](https://stackoverflow.com/q/26545159/164234).
|