Nick Craig-Wood
5065c422b4
lib/random: unify random string generation into random.String
...
This was factored from fstest as we were including the testing
enviroment into the main binary because of it.
This was causing opening the browser to fail because of 8243ff8bc8
.
2019-08-06 12:44:08 +01:00
Nick Craig-Wood
57d5de6fba
build: fix up package paths after repo move
...
git grep -l github.com/ncw/rclone | xargs -d'\n' perl -i~ -lpe 's|github.com/ncw/rclone|github.com/rclone/rclone|g'
goimports -w `find . -name \*.go`
2019-07-28 18:47:38 +01:00
Aleksandar Jankovic
f78cd1e043
Add context propagation to rclone
...
- Change rclone/fs interfaces to accept context.Context
- Update interface implementations to use context.Context
- Change top level usage to propagate context to lover level functions
Context propagation is needed for stopping transfers and passing other
request-scoped values.
2019-06-19 11:59:46 +01:00
Nick Craig-Wood
8ee6034b23
Look for Fs in the cache rather than calling NewFs directly
...
This will save operations when rclone is used in remote control mode
or with the same remote multiple times in the command line.
2019-05-27 16:14:55 +01:00
Nick Craig-Wood
7c4fe3eb75
local: define OpenWriterAt interface and test and implement it #2252
...
This will enable multipart downloads in future commits
2019-05-11 23:35:19 +01:00
Nick Craig-Wood
aba15f11d8
cache: note unsupported optional methods
2019-04-16 13:34:06 +01:00
Nick Craig-Wood
47099466c0
cache: Use walk.ListR for listing the temporary Fs.
2019-03-16 17:41:12 +00:00
Fabian Möller
a0d4c04687
backend: fix misspellings
2019-02-07 19:51:03 +01:00
Nick Craig-Wood
d966cef14c
build: fix problems found with unconvert
2019-02-02 11:45:07 +00:00
Nick Craig-Wood
97752ca8fb
build: fix problems found with ineffasign linter
2019-02-02 11:45:07 +00:00
Nick Craig-Wood
8d5d332daf
build: fix problems found with golint
2019-02-02 11:45:07 +00:00
Nick Craig-Wood
26b5f55cba
Update after goimports change
2018-12-04 10:11:57 +00:00
Nick Craig-Wood
0b80d1481a
cache: make tests not start an rc but use the internal framework
2018-11-04 15:11:51 +00:00
Nick Craig-Wood
0bfa9811f7
rc: factor server code into rcserver and implement serving objects
...
If a GET or HEAD request is receivied with a URL parameter of fs then
it will be served from that remote.
2018-11-03 11:32:00 +00:00
Florian Gamboeck
696e7b2833
backend/cache: Print correct info about Cache Writes
2018-10-27 00:27:47 +01:00
Nick Craig-Wood
ade252f13b
build: fixup code formatting after goimports change
2018-10-13 22:47:12 +01:00
Fabian Möller
1a40bceb1d
backend: unify NewFs path handling for wrapping remotes
...
Use the same function to join the root paths for the wrapping remotes
alias, cache and crypt.
The new function fspath.JoinRootPath is equivalent to path.Join, but if
the first non empty element starts with "//", this is preserved to allow
Windows network path to be used in these remotes.
2018-10-10 17:50:27 +01:00
Nick Craig-Wood
a9273c5da5
docs: move documentation for options from docs/content into backends
...
In the following commit, the documentation will be autogenerated.
2018-10-06 11:47:46 +01:00
dcpu
c0084f43dd
cache: Remove entries that no longer exist in the source
...
list directory with 25k files
before(1.43.1)
5m24s
after
3m21s
2018-10-06 11:23:33 +01:00
Fabian Möller
1eec59e091
fs: update ChangeNotifier interface
...
This introduces a channel to the ChangeNotify function, which can be
used to update the poll-interval and cleanly exit the polling function.
2018-09-25 14:01:13 +02:00
Fabian Möller
3c44ef788a
cache: add plex_insecure option to skip certificate validation
...
Fixes #2215
2018-09-10 21:19:25 +01:00
Fabian Möller
f6ee0795ac
cache: preserve leading / in wrapped remote path
...
When combining the remote value and the root path, preserve the absence
or presence of the / at the beginning of the wrapped remote path.
e.g. a remote "cloud:" and root path "dir" becomes "cloud:dir" instead
of "cloud:/dir".
Fixes #2553
2018-09-10 20:35:50 +01:00
Anagh Kumar Baranwal
5a6750e1cd
cache: documentation fix for cache-chunk-total-size - Fixes #2519
...
Signed-off-by: Anagh Kumar Baranwal <anaghk.dos@gmail.com>
2018-09-04 16:16:35 +01:00
Fabian Möller
deda093637
cache: fix error return value of cache/fetch rc method
2018-09-03 17:32:11 +02:00
dcpu
a4c4019032
cache: improve performance by not sending info requests for cached chunks
2018-09-03 15:41:06 +01:00
Fabian Möller
a0b3fd3a33
cache: fix worker scale down
...
Ensure that calling scaleWorkers will create/destroy the right amount of
workers.
2018-09-03 12:29:35 +02:00
Fabian Möller
cdbe3691b7
cache: add cache/fetch rc function
2018-09-03 12:29:35 +02:00
Dr. Tobias Quathamer
76450c01f3
cache: remove accidentally committed files
...
* Delete cache_upload_test.go.orig
* Delete cache_upload_test.go.rej
2018-09-02 15:51:22 +01:00
Nick Craig-Wood
6156bc5806
cache: fix nil pointer deref - fixes #2448
2018-08-07 21:33:13 +01:00
Fabian Möller
eb6aafbd14
cache: implement fs.ObjectUnWrapper
2018-08-07 09:28:55 +01:00
Nick Craig-Wood
ffd11662ba
cache: fix nil pointer deref when using lsjson on cached directory
...
This stops embedding the fs.Directory into the cache.Directory because
it can be nil and implements an ID method which checks the nil status.
See: https://forum.rclone.org/t/runtime-error-with-cache-remote-and-lsjson/6305
2018-08-05 09:42:31 +01:00
Nick Craig-Wood
f3f48d7d49
Implement new backend config system
...
This unifies the 3 methods of reading config
* command line
* environment variable
* config file
And allows them all to be configured in all places. This is done by
making the []fs.Option in the backend registration be the master
source of what the backend options are.
The backend changes are:
* Use the new configmap.Mapper parameter
* Use configstruct to parse it into an Options struct
* Add all config to []fs.Option including defaults and help
* Remove all uses of pflag
* Remove all uses of config.FileGet
2018-07-16 21:20:47 +01:00
remusb
aaa8591661
cache: add non cached dirs on notifications - #2155
2018-06-13 23:57:26 +03:00
remusb
4df1794932
cache: fix panic when running without plex configs
2018-06-13 15:06:14 +03:00
remusb
339fbf0df5
cache: reconnect plex websocket on failures
2018-06-12 22:58:15 +03:00
remusb
13ccb39819
cache: allow root to be expired from rc - #2237
2018-06-12 22:19:03 +03:00
remusb
f9a1a7e700
cache: fix root folder caching
2018-06-10 21:54:20 +03:00
remusb
4e8e5fed7d
cache: clean remaining empty folders from temp upload path
2018-06-09 14:52:31 +03:00
remusb
c0f772bc14
cache: update internal tests and small fixes
2018-06-08 23:34:38 +03:00
remusb
8a84975993
cache: cache lists using batch writes
2018-06-04 21:04:45 +03:00
John Clayton
71b4f1ccab
cache: use secure websockets for HTTPS Plex addresses
2018-05-22 11:47:57 +01:00
Nick Craig-Wood
e5ff375948
Use config.FileGet instead of fs.ConfigFileGet
2018-05-22 09:43:24 +01:00
Nick Craig-Wood
1320e84bc2
build: remove unused code spotted by the deadcode linter
2018-05-05 17:32:41 +01:00
Nick Craig-Wood
cb5bd47e61
build: fix errors spotted by ineffassign linter
...
These were mostly caused by shadowing err and a good fraction of them
will have caused errors not to be propagated properly.
2018-05-05 17:32:41 +01:00
Nick Craig-Wood
9ab2521ef2
rc: autogenerate and tidy the docs and commands
...
* Rename rc/pid -> core/pid
* Sort the output of `rc list`
* Make a script to autogenerate the docs
* Tidy docs
2018-04-23 20:57:17 +01:00
Nick Craig-Wood
1ac6dacf0f
about: complete other providers and re-work internals
...
* Implement about for:
* local, crypt, cache, drive, swift, hubic, onedrive, pcloud, dropbox
* Implement `--json` and `---full` flag for `rclone about`
* change About interface to return a Usage structure
* Remove operations.About as it is too thin an interface
* Implement Integration test
Relates to #1138 and #1564
2018-04-17 21:53:27 +01:00
a-roussos
94e277d759
about: add new command 'about' to get quota info from a remote
...
Implemented for drive only.
Relates to #1138 and #1564 .
2018-04-17 21:50:14 +01:00
Nick Craig-Wood
0fa700b3cf
Make integration tests use go1.7+ nested tests #2154
...
* Removed generated code and code generator
* Updated docs on how to write integration tests
* Tidied up the actual integration tests
2018-04-08 15:04:14 +01:00
Nick Craig-Wood
be54fd8f70
Remove builds conditional on go1.7 since that is now guaranteed #2154
...
Old fallback code was deleted and the go1.7 style code inlined where
appropriate.
2018-04-07 11:42:55 +01:00
Nick Craig-Wood
e5be471ce0
Use io.SeekStart/End/Current constants now for go1.7+ #2154
2018-04-07 11:42:36 +01:00
Nick Craig-Wood
80588a5a6b
Replace "golang.org/x/net/context" with "context" for go1.7+ #2154
2018-04-07 11:42:08 +01:00
remusb
6e11a25df5
cache: flush the memory cache after close
2018-04-04 23:25:53 +03:00
remusb
1dea99ab20
cache: purge file data on notification
2018-04-03 23:24:45 +03:00
remusb
e42cee5e02
cache: always forget parent dir for notifications - for #2117
2018-03-31 12:44:09 +03:00
Stefan
a8267d1628
link: allow creating public link to files and folders - closes #1562
2018-03-29 09:10:19 +02:00
remusb
b52e34ef5e
cache: add info log on notification - for #2150
2018-03-23 22:41:01 +02:00
remusb
0ed0d9a7bc
cache: integrate with Plex websocket
2018-03-22 21:21:03 +02:00
seuffert
d9c13bff83
add rc cache/stats
2018-03-22 21:16:16 +02:00
remusb
4924ac2f17
cache: reduce log level for plex api - for #2102
2018-03-17 11:57:36 +02:00
remusb
078d705dbe
cache: notify vfs and support crypt in rpc - #2111
2018-03-15 11:39:16 +02:00
remusb
677971643c
cache: add support for rc
2018-03-14 22:58:20 +02:00
Remus Bunduc
d76da1f5fd
cache: fix dir cache issue - #2117
2018-03-14 11:08:30 +02:00
Nick Craig-Wood
a7dbf32c53
cache: Implement --cache-db-wait-time flag
...
This can be used to make the cache wait for other running cache
backends to finish rather than erroring after 1 second.
2018-03-08 21:14:09 +00:00
Remus Bunduc
70f07fd3ac
fs: add ChangeNotify and backend support for it ( #2094 )
...
* fs: rename DirChangeNotify to ChangeNotify
* cache: switch to ChangeNotify
* ChangeNotify: keep order of notifications
2018-03-08 22:03:34 +02:00
Nick Craig-Wood
fe8eeec5b5
cache: improve efficiency with RangeOption and RangeSeek #1825
...
* All remotes now support RangeOption so remove SeekOption
* Correct off by one error as RangeOption arguments are inclusive.
* Use RangeSeek in preference to Seek if available
2018-03-02 17:10:56 +00:00
Fabian Möller
aeefa34f62
fstests: add TestInternal ( #2085 )
...
TestInternal allows to perform a custom test on the backend using the
optional InternalTester interface.
2018-02-25 10:58:06 +01:00
remusb
9a34fd984c
cache: fix dirmove with temp fs enabled
2018-02-14 23:47:45 +02:00
remusb
54724a1362
cache: notify vfs when using temp fs - fixes #2051
2018-02-11 22:30:58 +02:00
remusb
b33e3f779c
cache: add support for polling
2018-02-10 22:01:05 +02:00
remusb
b3d8b7e22e
cache: use atexit for cleanup
2018-01-30 22:35:53 +02:00
Fabian Möller
29d428040c
cache: clean root path ( #2023 )
...
Trim "/" from the root path to fix "slice bounds out of range" panic
in cache.go:1272.
Fixes #1945
2018-01-30 14:35:40 +01:00
remusb
40af98b0b3
cache: offline uploading
2018-01-30 00:05:04 +02:00
Nick Craig-Wood
fe52502f19
fs: Adjust RangeOption.Decode to return -1 for read to end
...
A Range request can never request 0 bytes however this change was made
to make a clearer signal that the limit means read to the end.
Add test and more documentation and fixup uses
2018-01-27 14:31:29 +00:00
Nick Craig-Wood
88e0770f2d
cache: Implement RangeOption #1825
2018-01-22 19:44:55 +00:00
Nick Craig-Wood
8653944a6d
Make RangeOption manadatory for Open - #1825
...
Add an integration test to make sure all backends implement
RangeOption correctly.
2018-01-21 17:09:12 +00:00
Nick Craig-Wood
71722b5b95
config: factor Obscure and Reveal into its own package
2018-01-18 20:19:55 +00:00
Nick Craig-Wood
11da2a6c9b
Break the fs package up into smaller parts.
...
The purpose of this is to make it easier to maintain and eventually to
allow the rclone backends to be re-used in other projects without
having to use the rclone configuration system.
The new code layout is documented in CONTRIBUTING.
2018-01-15 17:51:14 +00:00
Nick Craig-Wood
b8b620f5c2
Move all backends into backend directory
2018-01-12 17:07:38 +00:00