--- title: "Remote Control" description: "Remote controlling rclone" date: "2018-03-05" --- # Remote controlling rclone # If rclone is run with the `--rc` flag then it starts an http server which can be used to remote control rclone. **NB** this is experimental and everything here is subject to change! ## Supported parameters #### --rc #### Flag to start the http server listen on remote requests #### --rc-addr=IP #### IPaddress:Port or :Port to bind server to. (default "localhost:5572") #### --rc-cert=KEY #### SSL PEM key (concatenation of certificate and CA certificate) #### --rc-client-ca=PATH #### Client certificate authority to verify clients with #### --rc-htpasswd=PATH #### htpasswd file - if not provided no authentication is done #### --rc-key=PATH #### SSL PEM Private key #### --rc-max-header-bytes=VALUE #### Maximum size of request header (default 4096) #### --rc-user=VALUE #### User name for authentication. #### --rc-pass=VALUE #### Password for authentication. #### --rc-realm=VALUE #### Realm for authentication (default "rclone") #### --rc-server-read-timeout=DURATION #### Timeout for server reading data (default 1h0m0s) #### --rc-server-write-timeout=DURATION #### Timeout for server writing data (default 1h0m0s) ## Accessing the remote control via the rclone rc command Rclone itself implements the remote control protocol in its `rclone rc` command. You can use it like this ``` $ rclone rc rc/noop param1=one param2=two { "param1": "one", "param2": "two" } ``` Run `rclone rc` on its own to see the help for the installed remote control commands. `rclone rc` also supports a `--json` flag which can be used to send more complicated input parameters. ``` $ rclone rc --json '{ "p1": [1,"2",null,4], "p2": { "a":1, "b":2 } }' rc/noop { "p1": [ 1, "2", null, 4 ], "p2": { "a": 1, "b": 2 } } ``` ## Special parameters The rc interface supports some special parameters which apply to **all** commands. These start with `_` to show they are different. ### Running asynchronous jobs with _async = true If `_async` has a true value when supplied to an rc call then it will return immediately with a job id and the task will be run in the background. The `job/status` call can be used to get information of the background job. The job can be queried for up to 1 minute after it has finished. It is recommended that potentially long running jobs, eg `sync/sync`, `sync/copy`, `sync/move`, `operations/purge` are run with the `_async` flag to avoid any potential problems with the HTTP request and response timing out. Starting a job with the `_async` flag: ``` $ rclone rc --json '{ "p1": [1,"2",null,4], "p2": { "a":1, "b":2 }, "_async": true }' rc/noop { "jobid": 2 } ``` Query the status to see if the job has finished. For more information on the meaning of these return parameters see the `job/status` call. ``` $ rclone rc --json '{ "jobid":2 }' job/status { "duration": 0.000124163, "endTime": "2018-10-27T11:38:07.911245881+01:00", "error": "", "finished": true, "id": 2, "output": { "_async": true, "p1": [ 1, "2", null, 4 ], "p2": { "a": 1, "b": 2 } }, "startTime": "2018-10-27T11:38:07.911121728+01:00", "success": true } ``` `job/list` can be used to show the running or recently completed jobs ``` $ rclone rc job/list { "jobids": [ 2 ] } ``` ## Supported commands ### cache/expire: Purge a remote from cache Purge a remote from the cache backend. Supports either a directory or a file. Params: - remote = path to remote (required) - withData = true/false to delete cached data (chunks) as well (optional) Eg rclone rc cache/expire remote=path/to/sub/folder/ rclone rc cache/expire remote=/ withData=true ### cache/fetch: Fetch file chunks Ensure the specified file chunks are cached on disk. The chunks= parameter specifies the file chunks to check. It takes a comma separated list of array slice indices. The slice indices are similar to Python slices: start[:end] start is the 0 based chunk number from the beginning of the file to fetch inclusive. end is 0 based chunk number from the beginning of the file to fetch exclisive. Both values can be negative, in which case they count from the back of the file. The value "-5:" represents the last 5 chunks of a file. Some valid examples are: ":5,-5:" -> the first and last five chunks "0,-2" -> the first and the second last chunk "0:10" -> the first ten chunks Any parameter with a key that starts with "file" can be used to specify files to fetch, eg rclone rc cache/fetch chunks=0 file=hello file2=home/goodbye File names will automatically be encrypted when the a crypt remote is used on top of the cache. ### cache/stats: Get cache stats Show statistics for the cache remote. ### core/bwlimit: Set the bandwidth limit. This sets the bandwidth limit to that passed in. Eg rclone rc core/bwlimit rate=1M rclone rc core/bwlimit rate=off The format of the parameter is exactly the same as passed to --bwlimit except only one bandwidth may be specified. ### core/gc: Runs a garbage collection. This tells the go runtime to do a garbage collection run. It isn't necessary to call this normally, but it can be useful for debugging memory problems. ### core/memstats: Returns the memory statistics This returns the memory statistics of the running program. What the values mean are explained in the go docs: https://golang.org/pkg/runtime/#MemStats The most interesting values for most people are: * HeapAlloc: This is the amount of memory rclone is actually using * HeapSys: This is the amount of memory rclone has obtained from the OS * Sys: this is the total amount of memory requested from the OS * It is virtual memory so may include unused memory ### core/pid: Return PID of current process This returns PID of current process. Useful for stopping rclone process. ### core/stats: Returns stats about current transfers. This returns all available stats rclone rc core/stats Returns the following values: ``` { "speed": average speed in bytes/sec since start of the process, "bytes": total transferred bytes since the start of the process, "errors": number of errors, "fatalError": whether there has been at least one FatalError, "retryError": whether there has been at least one non-NoRetryError, "checks": number of checked files, "transfers": number of transferred files, "deletes" : number of deleted files, "elapsedTime": time in seconds since the start of the process, "lastError": last occurred error, "transferring": an array of currently active file transfers: [ { "bytes": total transferred bytes for this file, "eta": estimated time in seconds until file transfer completion "name": name of the file, "percentage": progress of the file transfer in percent, "speed": speed in bytes/sec, "speedAvg": speed in bytes/sec as an exponentially weighted moving average, "size": size of the file in bytes } ], "checking": an array of names of currently active file checks [] } ``` Values for "transferring", "checking" and "lastError" are only assigned if data is available. The value for "eta" is null if an eta cannot be determined. ### rc/error: This returns an error This returns an error with the input as part of its error string. Useful for testing error handling. ### rc/list: List all the registered remote control commands This lists all the registered remote control commands as a JSON map in the commands response. ### rc/noop: Echo the input to the output parameters This echoes the input parameters to the output parameters for testing purposes. It can be used to check that rclone is still alive and to check that parameter passing is working properly. ### vfs/forget: Forget files or directories in the directory cache. This forgets the paths in the directory cache causing them to be re-read from the remote when needed. If no paths are passed in then it will forget all the paths in the directory cache. rclone rc vfs/forget Otherwise pass files or dirs in as file=path or dir=path. Any parameter key starting with file will forget that file and any starting with dir will forget that dir, eg rclone rc vfs/forget file=hello file2=goodbye dir=home/junk ### vfs/poll-interval: Get the status or update the value of the poll-interval option. Without any parameter given this returns the current status of the poll-interval setting. When the interval=duration parameter is set, the poll-interval value is updated and the polling function is notified. Setting interval=0 disables poll-interval. rclone rc vfs/poll-interval interval=5m The timeout=duration parameter can be used to specify a time to wait for the current poll function to apply the new value. If timeout is less or equal 0, which is the default, wait indefinitely. The new poll-interval value will only be active when the timeout is not reached. If poll-interval is updated or disabled temporarily, some changes might not get picked up by the polling function, depending on the used remote. ### vfs/refresh: Refresh the directory cache. This reads the directories for the specified paths and freshens the directory cache. If no paths are passed in then it will refresh the root directory. rclone rc vfs/refresh Otherwise pass directories in as dir=path. Any parameter key starting with dir will refresh that directory, eg rclone rc vfs/refresh dir=home/junk dir2=data/misc If the parameter recursive=true is given the whole directory tree will get refreshed. This refresh will use --fast-list if enabled. ## Accessing the remote control via HTTP Rclone implements a simple HTTP based protocol. Each endpoint takes an JSON object and returns a JSON object or an error. The JSON objects are essentially a map of string names to values. All calls must made using POST. The input objects can be supplied using URL parameters, POST parameters or by supplying "Content-Type: application/json" and a JSON blob in the body. There are examples of these below using `curl`. The response will be a JSON blob in the body of the response. This is formatted to be reasonably human readable. ### Error returns If an error occurs then there will be an HTTP error status (eg 500) and the body of the response will contain a JSON encoded error object, eg ``` { "error": "Expecting string value for key \"remote\" (was float64)", "input": { "fs": "/tmp", "remote": 3 }, "status": 400 "path": "operations/rmdir", } ``` The keys in the error response are - error - error string - input - the input parameters to the call - status - the HTTP status code - path - the path of the call ### CORS The sever implements basic CORS support and allows all origins for that. The response to a preflight OPTIONS request will echo the requested "Access-Control-Request-Headers" back. ### Using POST with URL parameters only ``` curl -X POST 'http://localhost:5572/rc/noop/?potato=1&sausage=2' ``` Response ``` { "potato": "1", "sausage": "2" } ``` Here is what an error response looks like: ``` curl -X POST 'http://localhost:5572/rc/error/?potato=1&sausage=2' ``` ``` { "error": "arbitrary error on input map[potato:1 sausage:2]", "input": { "potato": "1", "sausage": "2" } } ``` Note that curl doesn't return errors to the shell unless you use the `-f` option ``` $ curl -f -X POST 'http://localhost:5572/rc/error/?potato=1&sausage=2' curl: (22) The requested URL returned error: 400 Bad Request $ echo $? 22 ``` ### Using POST with a form ``` curl --data "potato=1" --data "sausage=2" http://localhost:5572/rc/noop/ ``` Response ``` { "potato": "1", "sausage": "2" } ``` Note that you can combine these with URL parameters too with the POST parameters taking precedence. ``` curl --data "potato=1" --data "sausage=2" "http://localhost:5572/rc/noop/?rutabaga=3&sausage=4" ``` Response ``` { "potato": "1", "rutabaga": "3", "sausage": "4" } ``` ### Using POST with a JSON blob ``` curl -H "Content-Type: application/json" -X POST -d '{"potato":2,"sausage":1}' http://localhost:5572/rc/noop/ ``` response ``` { "password": "xyz", "username": "xyz" } ``` This can be combined with URL parameters too if required. The JSON blob takes precedence. ``` curl -H "Content-Type: application/json" -X POST -d '{"potato":2,"sausage":1}' 'http://localhost:5572/rc/noop/?rutabaga=3&potato=4' ``` ``` { "potato": 2, "rutabaga": "3", "sausage": 1 } ``` ## Debugging rclone with pprof ## If you use the `--rc` flag this will also enable the use of the go profiling tools on the same port. To use these, first [install go](https://golang.org/doc/install). Then (for example) to profile rclone's memory use you can run: go tool pprof -web http://localhost:5572/debug/pprof/heap This should open a page in your browser showing what is using what memory. You can also use the `-text` flag to produce a textual summary ``` $ go tool pprof -text http://localhost:5572/debug/pprof/heap Showing nodes accounting for 1537.03kB, 100% of 1537.03kB total flat flat% sum% cum cum% 1024.03kB 66.62% 66.62% 1024.03kB 66.62% github.com/ncw/rclone/vendor/golang.org/x/net/http2/hpack.addDecoderNode 513kB 33.38% 100% 513kB 33.38% net/http.newBufioWriterSize 0 0% 100% 1024.03kB 66.62% github.com/ncw/rclone/cmd/all.init 0 0% 100% 1024.03kB 66.62% github.com/ncw/rclone/cmd/serve.init 0 0% 100% 1024.03kB 66.62% github.com/ncw/rclone/cmd/serve/restic.init 0 0% 100% 1024.03kB 66.62% github.com/ncw/rclone/vendor/golang.org/x/net/http2.init 0 0% 100% 1024.03kB 66.62% github.com/ncw/rclone/vendor/golang.org/x/net/http2/hpack.init 0 0% 100% 1024.03kB 66.62% github.com/ncw/rclone/vendor/golang.org/x/net/http2/hpack.init.0 0 0% 100% 1024.03kB 66.62% main.init 0 0% 100% 513kB 33.38% net/http.(*conn).readRequest 0 0% 100% 513kB 33.38% net/http.(*conn).serve 0 0% 100% 1024.03kB 66.62% runtime.main ``` Possible profiles to look at: * Memory: `go tool pprof http://localhost:5572/debug/pprof/heap` * 30-second CPU profile: `go tool pprof http://localhost:5572/debug/pprof/profile` * 5-second execution trace: `wget http://localhost:5572/debug/pprof/trace?seconds=5` See the [net/http/pprof docs](https://golang.org/pkg/net/http/pprof/) for more info on how to use the profiling and for a general overview see [the Go team's blog post on profiling go programs](https://blog.golang.org/profiling-go-programs). The profiling hook is [zero overhead unless it is used](https://stackoverflow.com/q/26545159/164234).