Nick Craig-Wood
fa4a25a73b
fs: only test one level of cache
...
Can't test multiple caches at once as can only have 1 DB open at once
2017-12-04 15:50:59 +00:00
Ernest Borowski
934df67aef
filter: warn the user if he use --include and --exclude together fixes #1764
...
Signed-off-by: Ernest Borowski <er.borowski@gmail.com>
2017-12-04 14:20:01 +00:00
Nick Craig-Wood
006b296c34
Tidy up Makefile to get rid of vendor directory avoidance workarounds
2017-12-03 13:03:20 +00:00
Nick Craig-Wood
38b85e94ea
vfs: rename --cache-* options to --vfs-cache-* to save confusion
...
..as the backend cache options are all called --cache-* too. Adjust
docs to point out what the vfs cache does vs the backend cache.
2017-12-03 12:14:15 +00:00
Nick Craig-Wood
4b185355df
fs: rcat - use in memory object and Copy for more reliable transfers
2017-12-03 12:14:15 +00:00
Nick Craig-Wood
7d15c33e42
fs: make Copy and Move return the destination object if possible
2017-12-03 12:14:15 +00:00
Nick Craig-Wood
11332a19a0
fs: make an in memory object for short transfers
2017-12-03 12:14:15 +00:00
Nick Craig-Wood
a1f8318b29
Add Laurence to contributors
2017-12-03 10:24:53 +00:00
Laurence
e767c9ac9f
Fix typo in dbhashsum description
2017-12-03 10:24:33 +00:00
Nick Craig-Wood
56cfb810a8
Add Tim Cooijmans to contributors
2017-12-03 10:22:42 +00:00
Tim Cooijmans
835ca15ec8
drive: add service account support. Fixes #839 .
2017-12-03 10:21:41 +00:00
remusb
4af4bbb539
cache: add support for PutStream - fixes #1836
2017-11-30 21:16:45 +02:00
remusb
47450ba326
cache: handle errors when bolt tries to start
2017-11-30 12:27:59 +02:00
Nick Craig-Wood
639e812789
fs: integration tests: add SUMMARY heading for log scraping
2017-11-29 15:55:37 +00:00
Nick Craig-Wood
1c6cad2252
fs: integration tests: add 30 minute timeout per test
2017-11-29 13:51:17 +00:00
Nick Craig-Wood
6d3df6f172
cmount: make tests more reliable on Windows
2017-11-28 20:39:24 +00:00
Nick Craig-Wood
c16ac697a9
vfs: keep track of directories in the cache also #1860
...
This makes managing empty directories more reliable.
2017-11-28 20:39:23 +00:00
Nick Craig-Wood
0978957a2e
vfs: make sure all 96 combinations of flags for Open work
2017-11-28 20:39:23 +00:00
Nick Craig-Wood
d1b19f975d
vfs: remove items from cache when deleted #1860
...
Also fixes Error message when items have been deleted from the cache
(eg when Moved) when the cache reaper comes to delete them.
2017-11-28 16:13:58 +00:00
ishuah
aab8051f50
move: add --delete-empty-src-dirs flag - fixes #1854
2017-11-28 11:38:19 +03:00
Nick Craig-Wood
1248beb0b2
cachestats: Fix nil pointer if not a cache remote - fixes #1855
...
Also don't retry or show stats
2017-11-24 10:22:23 +00:00
Nick Craig-Wood
6448c445f5
acd: Fix download of large files failing - Fixes #1501
...
Previously it was necessary to work around large files failing to
download with `--acd-templink-threshold`. This change makes that flag
obsolete and all files should download. Templinks may be useful under
some circumstances though the flag isn't being removed.
It does this by filtering `Authorization:` headers out in the
transport if the authorization is supplied in the URL. This prevents
the "Only one auth mechanism allowed; only the X-Amz-Algorithm query
parameter, Signature query string parameter or the Authorization
header should be specified" error from AWS.
2017-11-24 09:14:25 +00:00
Nick Craig-Wood
fdb01437d8
fs: Allow the http Transport to have an optional filter request function
2017-11-24 09:07:56 +00:00
Nick Craig-Wood
729e1305b7
oauthutil: Allow the http.Client to be passed in
2017-11-24 09:07:03 +00:00
Nick Craig-Wood
02ffd43572
fs: Save the config before asking for a token - fixes #1220
...
Before this if the client_id/client_secret was edited it would
disappear when asking for the new token.
This means the post config is done after the user has confirmed the
config is OK which can't be helped.
2017-11-23 14:01:32 +00:00
Nick Craig-Wood
e53892f53b
fs,drive,dropbox: Make and use new RepeatableReader variants to lower memory use
...
RepeatableReaderSized has a pre-allocated buffer which should help
with memory usage - before it grew the buffer. Since we know the size
of the chunks, pre-allocating it should be much more efficient.
RepeatableReaderBuffer uses the buffer passed in.
RepeatableLimit* are convenience funcitions for wrapping a reader in
an io.LimitReader and then a RepeatableReader with the same buffer
size.
2017-11-23 13:53:46 +00:00
ishuah
6c62fced60
move: fixed root source directories getting deleted after move - fixes #1849
2017-11-23 12:01:35 +03:00
Nick Craig-Wood
c64ad851af
Add David Minor to contributors
2017-11-23 08:57:34 +00:00
David Minor
4c116af1d0
s3: add support for ECS task IAM roles
...
ECS container IAM metadata is in a different place than EC2 IAM metadata.
Use defaults' RemoteCredProvider function to query the standard locations
for the credentials.
Give the ECS role precedence over the role available from the underlying
EC2 instance.
2017-11-23 08:56:56 +00:00
Nick Craig-Wood
8357a82eee
dropbox: change default chunk size to 48MB now we are buffering them in memory
2017-11-22 17:15:37 +00:00
Nick Craig-Wood
483f4b8ad9
dropbox: multiparts uploads retry retry every error after the first chunk is done
2017-11-22 17:15:37 +00:00
Nick Craig-Wood
6f61da5c75
dropbox: buffer the chunks when uploading large files so they can be retried
...
We use fs.RepeatableReader to buffer the chunks which plays nice with
the accounting. The default chunk size is 128M which may be too
large.
Fixes #1806
2017-11-22 17:15:37 +00:00
Nick Craig-Wood
159fce0106
fs: fix --cache-dir to have some effect
2017-11-22 17:05:02 +00:00
remusb
569c1a2ec1
cache: catch signal interrupt for bolt handle cleanup
2017-11-22 18:32:36 +02:00
remusb
2497ca5134
cache: add extra logging in Move and Copy
2017-11-22 00:38:25 +02:00
Nick Craig-Wood
cbe5d7ce64
fs: Remove X-Auth-Token: from headers when dumping for swift
2017-11-21 17:32:07 +00:00
Nick Craig-Wood
1a65a4e769
fs: Add --dump flag, introduce --dump requests, responses and remove --dump-auth, --dump-filters
...
Now --dump-flag is written as --dump flag. This is a comma separated list which can contain
* headers - HTTP headers as before
* bodies - HTTP bodies as before
* requests - HTTP request bodies
* responses - HTTP response bodies
* auth - HTTP auth
* filters - Filter rexeps
Leave --dump-headers and --dump-bodies for the time being but remove
the other --dump-* flags as they aren't used very often.
2017-11-21 17:32:07 +00:00
Nick Craig-Wood
bcf1ece43b
Update MAINTAINERS with our new maintainer Remus Bunduc
2017-11-21 17:32:07 +00:00
ishuah
b4aa920a3d
stats: show the amount of data transferred in kb/mb - fixes #1167
2017-11-21 12:40:02 +03:00
Remus Bunduc
41a97e39c8
cache: fix option help text
2017-11-21 11:25:28 +02:00
Nick Craig-Wood
abbcb2f5e0
cache: disable another unreliable test #1844
2017-11-20 21:25:38 +00:00
Nick Craig-Wood
cb6de4a2cf
cache: disable unreliable test #1844
2017-11-20 19:55:00 +00:00
Nick Craig-Wood
dc1c679c65
mount: support truncate properly
2017-11-20 19:42:35 +00:00
Nick Craig-Wood
3fb4fe31d2
vfs: make sure write only handles never truncate files they shouldn't
2017-11-20 19:42:25 +00:00
Nick Craig-Wood
76b151984c
vfs: cache the size of the object in the read handle
2017-11-20 17:57:13 +00:00
Nick Craig-Wood
f0ed384786
cache: fix default setting for warmup_age
2017-11-20 14:39:12 +00:00
Nick Craig-Wood
f80f7a0509
cache: use fs.CacheDir to make the default directory for the cache
...
NB this changes the default dir for the cache
2017-11-20 14:38:28 +00:00
Nick Craig-Wood
af50f31f7d
mounttest: wait for Release after every Read to stop using in use files under Windows
2017-11-20 12:46:24 +00:00
Nick Craig-Wood
8e2213fbbd
local: add error message for cross file system moves
2017-11-20 12:46:24 +00:00
Nick Craig-Wood
085c690798
build: add in 64bit path for WinFSP headers
2017-11-20 12:46:24 +00:00
