KurlesHS/restic
1
0
Fork 0
forked from TrueCloudLab/restic
Code Pull requests Activity
restic/CHANGELOG.md
Michael Pratt 9fa4f5eb6b gs: disable resumable uploads 
By default, the GCS Go packages have an internal "chunk size" of 8MB,
used for blob uploads.

Media().Do() will buffer a full 8MB from the io.Reader (or less if EOF
is reached) then write that full 8MB to the network all at once.

This behavior does not play nicely with --limit-upload, which only
limits the Reader passed to Media. While the long-term average upload
rate will be correctly limited, the actual network bandwidth will be
very spikey.

e.g., if an 8MB/s connection is limited to 1MB/s, Media().Do() will
spend 8s reading from the rate-limited reader (performing no network
requests), then 1s writing to the network at 8MB/s.

This is bad for network connections hurt by full-speed uploads,
particularly when writing 8MB will take several seconds.

Disable resumable uploads entirely by setting the chunk size to zero.
This causes the io.Reader to be passed further down the request stack,
where there is less (but still some) buffering.

My connection is around 1.5MB/s up, with nominal ~15ms ping times to
8.8.8.8.

Without this change, --limit-upload 1024 results in several seconds of
~200ms ping times (uploading), followed by several seconds of ~15ms ping
times (reading from rate-limited reader). A bandwidth monitor reports
this as several seconds of ~1.5MB/s followed by several seconds of
0.0MB/s.

With this change, --limit-upload 1024 results in ~20ms ping times and
the bandwidth monitor reports a constant ~1MB/s.

I've elected to make this change unconditional of --limit-upload because
the resumable uploads shouldn't be providing much benefit anyways, as
restic already uploads mostly small blobs and already has a retry
mechanism.

--limit-download is not affected by this problem, as Get().Download()
returns the real http.Response.Body without any internal buffering.

Updates #1216
2017-10-17 21:12:04 -07:00

16 KiB
Raw Blame History

This file describes changes relevant to all users that are made in each released version of restic from the perspective of the user.

Important Changes in 0.X.Y

Small changes

Important Changes in 0.7.3

Important Changes in 0.7.2

Small changes

Important Changes in 0.7.1

Small changes

Important Changes in 0.7.0

Small changes

Important Changes in 0.6.1

This is mostly a bugfix release and only contains small changes:

  • We've fixed a bug where rebuild-index would corrupt the index when used with the s3 backend together with the default layout. This is not the default setting.

  • Backends based on HTTP now allow several idle connections in parallel. This is especially important for the REST backend, which (when used with a local server) may create a lot connections and exhaust available ports quickly. https://github.com/restic/restic/issues/985 https://github.com/restic/restic/pull/986

  • Regular status report: We've removed the status report that was printed every 10 seconds when restic is run non-interactively. You can still force reporting the current status by sending a USR1 signal to the process. https://github.com/restic/restic/pull/974

  • The build.go now strips the temporary directory used for compilation from the binary. This is the first step in enabling reproducible builds. https://github.com/restic/restic/pull/981

Important Changes in 0.6.0

Consistent forget policy

The forget command was corrected to be more consistent in which snapshots are to be forgotten. It is possible that the new code removes more snapshots than before, so please review what would be deleted by using the --dry-run option.

https://github.com/restic/restic/pull/957 https://github.com/restic/restic/issues/953

Unified repository layout

Up to now the s3 backend used a special repository layout. We've decided to unify the repository layout and implemented the default layout also for the s3 backend. For creating a new repository on s3 with the default layout, use restic -o s3.layout=default init. For further commands the option is not necessary any more, restic will automatically detect the correct layout to use. A future version will switch to the default layout for new repositories.

https://github.com/restic/restic/pull/966 https://github.com/restic/restic/issues/965

Memory and time improvements for the s3 backend

We've updated the library used for accessing s3, switched to using a lower level API and added caching for some requests. This lead to a decrease in memory usage and a great speedup. In addition, we added benchmark functions for all backends, so we can track improvements over time. The Continuous Integration test service we're using (Travis) now runs the s3 backend tests not only against a Minio server, but also against the Amazon s3 live service, so we should be notified of any regressions much sooner.

https://github.com/restic/restic/pull/962 https://github.com/restic/restic/pull/960 https://github.com/restic/restic/pull/946 https://github.com/restic/restic/pull/938 https://github.com/restic/restic/pull/883