Commit graph

3270 commits

Author SHA1 Message Date
Alexander Neumann
d01d07fc0a Merge pull request #1400 from armhold/deadlock2
log unexpected errs from b2 ListCurrentObject()
2017-10-30 18:39:33 +01:00
Alexander Neumann
526aaca6f5 Merge pull request #1389 from wojas/fix-modtime-tests-on-apfs
Fix modtime tests on APFS: allow 1us difference
2017-10-29 14:20:40 +01:00
George Armhold
2f8147af59 log unexpected errs from b2 ListCurrentObject()
gh-1385
2017-10-29 08:53:39 -04:00
Konrad Wojas
f3016a9096 Darwin test fix: allow 1μs timestamp difference
On Darwin, allow a 1μs difference in restored timestamps, because
macOS <10.13 cannot restore with nanosecond precision and the current
version of Go (1.9.2) does not yet support the new syscall required
for this. (#1087 #1389)
2017-10-29 20:48:07 +08:00
Alexander Neumann
f854a41ba9 Merge pull request #1399 from armhold/deadlock2
prevent deadlock in List() for B2 when b2.connections=1
2017-10-29 09:26:46 +01:00
Alexander Neumann
ca3cadef5e Merge pull request #1398 from armhold/sftp-readdir
sftp ReadDir: add path to return error messages (gh-1323)
2017-10-29 09:26:42 +01:00
George Armhold
3304b0fcf0 prevent deadlock in List() for B2 when b2.connections=1
This is a fix for the following situation (gh-1188):

List() grabs a semaphore token upon entry, starts a goroutine, and
does not release the token until the routine exits (via a defer).

The goroutine iterates over the results from ListCurrentObjects(),
sending them one at a time to a channel, where they are ultimately
processed by be.Load().

Since be.Load() also needs a token, this will result in deadlock if
b2.connections=1.

This fix changes List() so that the token is only held during the call
to ListCurrentObjects().
2017-10-28 18:46:47 -04:00
George Armhold
d8938e259a sftp ReadDir: add path to return error messages (gh-1323)
fix missing "Close" string in debug log fmt
2017-10-28 14:16:27 -04:00
Alexander Neumann
53a554c89d Merge pull request #1395 from restic/small-crypto-improvements
Small improvements
2017-10-28 12:57:18 +02:00
Alexander Neumann
e71db01230 Improve crypto test 2017-10-28 12:09:33 +02:00
Alexander Neumann
178e946fc7 Rename KDFParams -> Params 2017-10-28 10:28:29 +02:00
Alexander Neumann
f3bff12939 Merge pull request #1394 from zcalusic/master
Use lowercase in messages from check/tag commands, too
2017-10-28 09:56:29 +02:00
Alexander Neumann
7a99418dc5 Merge pull request #1393 from armhold/lint-errcheck
detect errors from fs.Walk() in local backend List()
2017-10-28 09:56:11 +02:00
Alexander Neumann
c71ba466ea Merge pull request #1391 from armhold/b2-listmax
pass in defaultListMaxItems to b2Backend constructor
2017-10-28 09:54:57 +02:00
Alexander Neumann
8ce5d35543 Merge pull request #1390 from armhold/lint
small cleanup
2017-10-28 09:54:35 +02:00
Zlatko Čalušić
134f834c60 Use lowercase in messages from check/tag commands, too 2017-10-27 21:06:34 +02:00
George Armhold
8a37c07295 send errors from fs.Walk() to debug log
clarify non-err returns from Walk where err is already proved to be nil
2017-10-27 08:41:17 -04:00
George Armhold
bd0ada7842 go fmt 2017-10-26 16:37:11 -04:00
George Armhold
eea96f652d go fmt 2017-10-26 16:22:10 -04:00
George Armhold
38c3061df7 pass in defaultListMaxItems to b2Backend constructor
gh-1385
2017-10-26 14:22:16 -04:00
George Armhold
f5fa602482 detect and return error from file Close() in Node.createFileAt()
gh-1385
2017-10-26 13:53:31 -04:00
Alexander Neumann
e44ac55f63 Merge pull request #1387 from wojas/snapshots-filter-last
Add --last flag to snapshots command
2017-10-26 19:08:49 +02:00
Alexander Neumann
f1cfb73a8b Merge pull request #1384 from wojas/reduce-ticker-refresh
Reduce ticker refresh rate with RESTIC_PROGRESS_FPS env
2017-10-26 19:02:59 +02:00
Konrad Wojas
5b96885c6d Control progress rate with RESTIC_PROGRESS_FPS env
Add a RESTIC_PROGRESS_FPS environment variable to limit the interval
at which the progress indicator updates (allowed values: 1-60).

The default rate of 60 FPS can cause high terminal CPU load on some
systems, like iTerm2 on macOS with font anti-aliasing enabled.

Usage:

    RESTIC_PROGRESS_FPS=1 restic ...
    RESTIC_PROGRESS_FPS=60 restic ...
2017-10-26 14:46:56 +08:00
Konrad Wojas
c5da90a5b7 Add --last flag to snapshots command
Add --last flag to snapshots command to only show the last entry for any
(hostname, paths) combination.

This makes it easier to check when various paths were last backed up.
2017-10-26 14:02:29 +08:00
George Armhold
bcdebfb84e small cleanup:
- be explicit when discarding returned errors from .Close(), etc.
- remove named return values from funcs when naked return not used
- fix some "err" shadowing when redeclaration not needed
2017-10-25 12:03:55 -04:00
Alexander Neumann
359b273649 Merge pull request #1381 from tbm/broken-link
Remove Markdown syntax from documentation
2017-10-24 20:38:40 +02:00
Martin Michlmayr
2e2c8dc620 Remove Markdown syntax from documentation
Convert a link from Markdown to RST syntax.  This link must
have been missed during the recent conversion.
2017-10-24 14:40:33 +01:00
Alexander Neumann
8d37b723ca Upadte vendored dependencies 2017-10-22 10:07:36 +02:00
Alexander Neumann
315b7f282f Remove explicit version constraints 2017-10-22 10:07:23 +02:00
Alexander Neumann
a3f8e9dfa7 Add entry to CHANGELOG 2017-10-21 12:22:50 +02:00
Alexander Neumann
982810f7cc Merge pull request #1368 from TobyLL/master
Ignore comments (lines starting with #) in the --files-from file
2017-10-21 12:21:11 +02:00
Alexander Neumann
90b96d19cd Merge pull request #1365 from felix9/fix_1068
Fix failure to detect some legacy s3 repos
2017-10-21 12:19:58 +02:00
Alexander Neumann
6a52bb6f54 Merge pull request #1374 from restic/update-blazer
Update github.com/kurin/blazer
2017-10-21 11:26:49 +02:00
Alexander Neumann
cacaa4393f Merge pull request #1373 from restic/check-gopkg-lock
CI: Check the vendor dir and Gopkg.lock
2017-10-21 11:14:30 +02:00
Alexander Neumann
d63ab4e9a4 Merge pull request #1358 from prattmic/chunk_size
gs: add option to set chunk size
2017-10-21 11:13:48 +02:00
Alexander Neumann
ca6daec8dd CI: Check the vendor dir and Gopkg.lock 2017-10-21 10:50:38 +02:00
Alexander Neumann
c87f2420a6 Update github.com/kurin/blazer 2017-10-21 10:30:39 +02:00
Alexander Neumann
f5bbbc52f4 Update issue template 2017-10-21 10:17:58 +02:00
TobyLL
9e3dde8ec7 Fix whitespace 2017-10-19 15:52:06 +01:00
TobyLL
9dba182e51 Ignore comments (lines starting with #) in the --files-from file 2017-10-19 15:48:22 +01:00
Felix Lee
944fc857eb Fix failure to detect some legacy s3 repos
Sometimes s3 listobjects for a directory includes an entry for that
directory. The restic s3 backend doesn't expect that and returns
an error.

Symptom is:
  ReadDir: invalid key name restic/key/, removing prefix
     restic/key/ yielded empty string

I'm not sure when s3 does that; I'm unable to reproduce it myself.

But in any case, it seems correct to ignore that when it happens.

Fixes #1068
2017-10-18 13:45:31 -07:00
Alexander Neumann
7507a658ac Merge pull request #1362 from restic/fix-backend-retry
Fix Save() backend retry
2017-10-18 07:45:56 +02:00
Michael Pratt
9fa4f5eb6b gs: disable resumable uploads
By default, the GCS Go packages have an internal "chunk size" of 8MB,
used for blob uploads.

Media().Do() will buffer a full 8MB from the io.Reader (or less if EOF
is reached) then write that full 8MB to the network all at once.

This behavior does not play nicely with --limit-upload, which only
limits the Reader passed to Media. While the long-term average upload
rate will be correctly limited, the actual network bandwidth will be
very spikey.

e.g., if an 8MB/s connection is limited to 1MB/s, Media().Do() will
spend 8s reading from the rate-limited reader (performing no network
requests), then 1s writing to the network at 8MB/s.

This is bad for network connections hurt by full-speed uploads,
particularly when writing 8MB will take several seconds.

Disable resumable uploads entirely by setting the chunk size to zero.
This causes the io.Reader to be passed further down the request stack,
where there is less (but still some) buffering.

My connection is around 1.5MB/s up, with nominal ~15ms ping times to
8.8.8.8.

Without this change, --limit-upload 1024 results in several seconds of
~200ms ping times (uploading), followed by several seconds of ~15ms ping
times (reading from rate-limited reader). A bandwidth monitor reports
this as several seconds of ~1.5MB/s followed by several seconds of
0.0MB/s.

With this change, --limit-upload 1024 results in ~20ms ping times and
the bandwidth monitor reports a constant ~1MB/s.

I've elected to make this change unconditional of --limit-upload because
the resumable uploads shouldn't be providing much benefit anyways, as
restic already uploads mostly small blobs and already has a retry
mechanism.

--limit-download is not affected by this problem, as Get().Download()
returns the real http.Response.Body without any internal buffering.

Updates #1216
2017-10-17 21:12:04 -07:00
Alexander Neumann
ce4d71d626 backend: Add partial read failure to error backend 2017-10-17 22:11:38 +02:00
Alexander Neumann
8e2ef3f38b cache: Store first, then store in the backend
Store the file in the backend, then rewind the reader and store it
in the cache afterwards.
2017-10-17 22:01:20 +02:00
Alexander Neumann
8dc952775e backend: Correctly retry Save() calls
Make sure the given reader is an io.Seeker and rewind it properly each
time.
2017-10-17 21:46:38 +02:00
Alexander Neumann
99b6163e27 Add 'dump' to manual 2017-10-16 20:24:47 +02:00
Alexander Neumann
beaf55f1fc dump: Allow absolute paths 2017-10-16 20:22:01 +02:00
Alexander Neumann
980bb9059f Add entry to changelog 2017-10-16 20:15:19 +02:00