restic/internal/backend
Michael Pratt 9fa4f5eb6b gs: disable resumable uploads
By default, the GCS Go packages have an internal "chunk size" of 8MB,
used for blob uploads.

Media().Do() will buffer a full 8MB from the io.Reader (or less if EOF
is reached) then write that full 8MB to the network all at once.

This behavior does not play nicely with --limit-upload, which only
limits the Reader passed to Media. While the long-term average upload
rate will be correctly limited, the actual network bandwidth will be
very spikey.

e.g., if an 8MB/s connection is limited to 1MB/s, Media().Do() will
spend 8s reading from the rate-limited reader (performing no network
requests), then 1s writing to the network at 8MB/s.

This is bad for network connections hurt by full-speed uploads,
particularly when writing 8MB will take several seconds.

Disable resumable uploads entirely by setting the chunk size to zero.
This causes the io.Reader to be passed further down the request stack,
where there is less (but still some) buffering.

My connection is around 1.5MB/s up, with nominal ~15ms ping times to
8.8.8.8.

Without this change, --limit-upload 1024 results in several seconds of
~200ms ping times (uploading), followed by several seconds of ~15ms ping
times (reading from rate-limited reader). A bandwidth monitor reports
this as several seconds of ~1.5MB/s followed by several seconds of
0.0MB/s.

With this change, --limit-upload 1024 results in ~20ms ping times and
the bandwidth monitor reports a constant ~1MB/s.

I've elected to make this change unconditional of --limit-upload because
the resumable uploads shouldn't be providing much benefit anyways, as
restic already uploads mostly small blobs and already has a retry
mechanism.

--limit-download is not affected by this problem, as Get().Download()
returns the real http.Response.Body without any internal buffering.

Updates #1216
2017-10-17 21:12:04 -07:00
..
azure Remove Deleter interface 2017-10-14 16:04:29 +02:00
b2 Remove Deleter interface 2017-10-14 16:04:29 +02:00
gs gs: disable resumable uploads 2017-10-17 21:12:04 -07:00
local backend: Add Delete() to restic.Backend interface 2017-10-14 15:56:25 +02:00
location Merge pull request #1149 from restic/azure-support 2017-08-09 21:30:35 +02:00
mem Move restic package to internal/restic 2017-07-24 17:43:32 +02:00
rest Remove Deleter interface 2017-10-14 16:04:29 +02:00
s3 Remove Deleter interface 2017-10-14 16:04:29 +02:00
sftp sftp: Fix Delete() 2017-10-14 16:08:15 +02:00
swift Remove Deleter interface 2017-10-14 16:04:29 +02:00
test Ensure TestDelete runs last 2017-10-14 16:04:29 +02:00
testdata Moves files 2017-07-23 14:19:13 +02:00
backend_error.go Retry failed backend requests 2017-10-14 15:56:25 +02:00
backend_retry.go Retry failed backend requests 2017-10-14 15:56:25 +02:00
doc.go Moves files 2017-07-23 14:19:13 +02:00
http_transport.go Add REST backend option to use CA root certificate 2017-10-04 22:14:10 +02:00
layout.go Move restic package to internal/restic 2017-07-24 17:43:32 +02:00
layout_default.go Move restic package to internal/restic 2017-07-24 17:43:32 +02:00
layout_rest.go Move restic package to internal/restic 2017-07-24 17:43:32 +02:00
layout_s3legacy.go Move restic package to internal/restic 2017-07-24 17:43:32 +02:00
layout_test.go Remove all dot-imports 2017-10-02 15:06:39 +02:00
paths.go Moves files 2017-07-23 14:19:13 +02:00
semaphore.go Run goimports 2017-07-23 14:21:03 +02:00
utils.go Move restic package to internal/restic 2017-07-24 17:43:32 +02:00
utils_test.go Remove all dot-imports 2017-10-02 15:06:39 +02:00