Michael Eischer
257740b0cc
b2: Successful delete if file does not exist
...
When deleting a file, B2 sometimes returns a "500 Service Unavailable"
error but nevertheless correctly deletes the file. Due to retries in
the B2 library blazer, we sometimes also see a "400 File not present"
error. The retries of restic for the delete request then fail with
"404 File with such name does not exist.".
As we have to rely on request retries in a distributed system to handle
temporary errors, also consider a delete request to be successful if the
file is reported as not existing. This should be safe as B2 claims to
provide a strongly consistent bucket listing and thus a missing file
shouldn't mysteriously show up again later on.
2021-10-09 23:51:12 +02:00
MichaelEischer
46d08d9404
Merge pull request #3535 from jtagcat/writingverbose
...
Change: selfupdate: 'writing restic to' as verbose
2021-10-09 22:41:51 +02:00
jtagcat
a7853057ab
Change: selfupdate: 'writing restic to' as verbose
...
Running restic self-update --quiet no longer
prints "writing restic to /usr/local/bin/restic".
The only output printed with -q is failures or
"successfully updated restic to version 0.12.1"
https://github.com/restic/restic/pull/3535
fix test fail: changelog title can't end with `.`
shorten changelog title
2021-10-09 23:24:18 +03:00
MichaelEischer
eb282532dc
Merge pull request #3534 from jtagcat/patch-1
...
docs/example: useradd: restic now system account
2021-10-09 20:42:43 +02:00
MichaelEischer
f2a3b3b4a1
Merge pull request #3537 from greatroar/dump-lru
...
Check cap instead of len in bloblru
2021-10-09 20:00:27 +02:00
MichaelEischer
58e8b34633
Merge pull request #3532 from garrmcnu/s3-credentials-config
...
s3: Add warning if key ID or secret is empty
2021-10-09 19:32:32 +02:00
MichaelEischer
a02cea6e83
Merge pull request #3539 from jnoxon/fix-ec2-metadata-v2
...
get iam roles working with ec2 metadata v2
2021-10-07 23:00:27 +02:00
Garry McNulty
708d7a2574
s3: Add warning if key ID or secret is empty
...
Also add debug message if no credential types are available.
Closes #2388
2021-10-06 23:13:40 +01:00
Jeff Noxon
6f4b5ab8d1
get iam roles working with ec2 metadata v2
2021-10-03 20:06:55 -05:00
greatroar
634a9c162d
Check cap instead of len in bloblru
...
restic dump uses bloblru.Cache to keep buffers alive, but also reuses
evicted buffers. That means large buffers may be used to store small
blobs, causing the cache to think it's using less memory than it
actually does.
2021-10-03 09:34:17 +02:00
jtagcat
632ca2ef52
docs/example: useradd: restic now system account
...
and: use /sbin/nologin
and: use long flags in the useradd command
and: download v0.12.1 instead of v0.9.6
2021-10-03 00:51:06 +03:00
MichaelEischer
24088f8307
Merge pull request #3528 from greatroar/tree-insert
...
internal/restic: Don't allocate in Tree.Insert
2021-09-27 22:22:03 +02:00
greatroar
c892c0bab9
internal/restic: Don't allocate in Tree.Insert
...
name old time/op new time/op delta
BuildTree-8 34.6µs ± 4% 7.0µs ± 3% -79.68% (p=0.000 n=18+19)
name old alloc/op new alloc/op delta
BuildTree-8 34.0kB ± 0% 0.9kB ± 0% -97.37% (p=0.000 n=20+20)
name old allocs/op new allocs/op delta
BuildTree-8 108 ± 0% 1 ± 0% -99.07% (p=0.000 n=20+20)
2021-09-26 18:08:48 +02:00
MichaelEischer
78dac2fd48
Merge pull request #3523 from greatroar/swift-v2
...
Upgrade ncw/swift to v2
2021-09-24 21:41:36 +02:00
Michael Eischer
5ea8bba1a1
swift: restore context err check for list()
2021-09-24 21:19:46 +02:00
MichaelEischer
a5e103a212
Merge pull request #3522 from greatroar/dump-lru
...
Use LRU cache in restic dump
2021-09-24 20:33:58 +02:00
greatroar
e7ec0453b1
Upgrade ncw/swift to v2
2021-09-24 19:08:37 +02:00
Uli Martens
1ebcb1d097
Add changelog entry for PR #3508
2021-09-24 15:45:09 +02:00
greatroar
fe04d024c7
Use LRU cache in restic dump
2021-09-24 15:45:08 +02:00
Uli Martens
718966a81a
Move Blobcache into dedicated internal package
2021-09-24 15:45:00 +02:00
greatroar
4f33eca634
Remove unused Writer arg to internal/dump.writeDump
2021-09-24 15:40:42 +02:00
MichaelEischer
cc110c42e6
Merge pull request #2657 from mansam/add-skip-tls-verification-flag
...
Add --insecure-tls flag to disable SSL cert verification
2021-09-22 21:40:01 +02:00
Sam Lucidi
897d8e662c
Add --insecure-tls flag to disable SSL cert verification
...
Signed-off-by: Sam Lucidi <slucidi@redhat.com>
2021-09-21 10:52:40 -04:00
MichaelEischer
1827b16ade
Merge pull request #3519 from greatroar/maphash
...
Replace siphash by hash/maphash
2021-09-19 19:46:03 +02:00
greatroar
8b758c78a3
Require Go 1.14 to build
2021-09-19 16:18:19 +02:00
greatroar
8d2996eaaa
Replace siphash by hash/maphash
...
In Go 1.17.1, maphash has become quite a bit faster than siphash, so we
can drop one third-party dependency. maphash is just an interface to the
standard Go map's hash function, which we already trust for other use
cases.
Benchmark results on linux/amd64, -benchtime=3s:
name old time/op new time/op delta
IndexHasUnknown-8 50.6ns ±10% 41.0ns ±19% -18.92% (p=0.000 n=9+10)
IndexHasKnown-8 52.6ns ±12% 41.5ns ±12% -21.13% (p=0.000 n=9+10)
IndexMapHash-8 3.64µs ± 1% 2.00µs ± 0% -45.09% (p=0.000 n=10+9)
IndexAlloc-8 700ms ± 1% 601ms ± 6% -14.18% (p=0.000 n=8+10)
IndexAllocParallel-8 205ms ± 5% 192ms ± 8% -6.18% (p=0.043 n=10+10)
MasterIndexAlloc-8 319ms ± 1% 279ms ± 5% -12.58% (p=0.000 n=10+10)
MasterIndexLookupSingleIndex-8 156ns ± 8% 147ns ± 6% -5.46% (p=0.023 n=10+10)
MasterIndexLookupMultipleIndex-8 150ns ± 7% 142ns ± 8% -5.69% (p=0.007 n=10+10)
MasterIndexLookupSingleIndexUnknown-8 74.4ns ± 6% 72.0ns ± 9% ~ (p=0.175 n=10+9)
MasterIndexLookupMultipleIndexUnknown-8 67.4ns ± 9% 65.5ns ± 7% ~ (p=0.340 n=9+9)
MasterIndexLookupParallel/known,indices=25-8 461ns ± 2% 445ns ± 2% -3.49% (p=0.000 n=10+10)
MasterIndexLookupParallel/unknown,indices=25-8 408ns ±11% 378ns ± 5% -7.22% (p=0.035 n=10+9)
MasterIndexLookupParallel/known,indices=50-8 479ns ± 1% 437ns ± 4% -8.82% (p=0.000 n=10+10)
MasterIndexLookupParallel/unknown,indices=50-8 406ns ± 8% 343ns ±15% -15.44% (p=0.001 n=10+10)
MasterIndexLookupParallel/known,indices=100-8 480ns ± 1% 455ns ± 5% -5.15% (p=0.000 n=8+10)
MasterIndexLookupParallel/unknown,indices=100-8 391ns ±18% 382ns ± 8% ~ (p=0.315 n=10+10)
MasterIndexLookupBlobSize-8 71.0ns ± 8% 57.2ns ±11% -19.36% (p=0.000 n=9+10)
PackerManager-8 254ms ± 1% 254ms ± 1% ~ (p=0.285 n=15+15)
name old speed new speed delta
IndexMapHash-8 1.12GB/s ± 1% 2.05GB/s ± 0% +82.13% (p=0.000 n=10+9)
PackerManager-8 208MB/s ± 1% 207MB/s ± 1% ~ (p=0.281 n=15+15)
name old alloc/op new alloc/op delta
IndexMapHash-8 0.00B 0.00B ~ (all equal)
IndexAlloc-8 400MB ± 0% 400MB ± 0% ~ (p=1.000 n=9+10)
IndexAllocParallel-8 401MB ± 0% 401MB ± 0% +0.00% (p=0.000 n=10+10)
MasterIndexAlloc-8 258MB ± 0% 262MB ± 0% +1.42% (p=0.000 n=9+10)
PackerManager-8 73.1kB ± 0% 73.1kB ± 0% ~ (p=0.382 n=13+13)
name old allocs/op new allocs/op delta
IndexMapHash-8 0.00 0.00 ~ (all equal)
IndexAlloc-8 907k ± 0% 907k ± 0% -0.00% (p=0.000 n=10+10)
IndexAllocParallel-8 907k ± 0% 907k ± 0% +0.00% (p=0.009 n=10+10)
MasterIndexAlloc-8 327k ± 0% 317k ± 0% -3.06% (p=0.000 n=10+10)
PackerManager-8 744 ± 0% 744 ± 0% ~ (all equal)
2021-09-19 16:05:18 +02:00
MichaelEischer
58efe21eca
Merge pull request #3264 from amozoss/upstream-master
...
Refactor backup progress
2021-09-19 14:54:42 +02:00
MichaelEischer
71fcf48533
Merge pull request #2594 from greatroar/concurrent-restore-verify
...
Concurrent restore --verify
2021-09-19 14:52:31 +02:00
Michael Eischer
921e328b56
restore: Fix linting error
2021-09-19 14:41:07 +02:00
Michael Eischer
e62d4f622f
Reword changelog
2021-09-19 14:41:07 +02:00
Michael Eischer
2cdc0719af
restorer: Sanitize verify errors
2021-09-19 14:01:26 +02:00
greatroar
bdcdfaf6b4
restore --verify: buffer reuse consistency and comment
2021-09-19 13:11:27 +02:00
greatroar
2b94742ca5
Replace no-op closures in restorer by nil check
2021-09-19 13:11:16 +02:00
greatroar
d357744104
Handle canceled context in restore --verify properly
2021-09-19 13:11:05 +02:00
greatroar
d4225ec803
Simplify buffer growing in Restorer.verifyFile
...
Suggested-by: Igor Fedorenko <igor@ifedorenko.com>
2021-09-19 13:10:55 +02:00
greatroar
de8521ae56
Report number of successes from Restorer.VerifyFiles
...
Previously, the number of attempts was reported.
2021-09-19 13:10:44 +02:00
greatroar
bb066cf7d3
Concurrent Restorer.VerifyFiles
...
Time to verify a 2GB snapshot down from 9.726s to 4.645s (-52%).
2021-09-19 13:10:41 +02:00
greatroar
556424d61b
Reuse buffer in Restorer.VerifyFiles
...
Time to verify a 2GB snapshot down from 11.568s to 9.726s (-16%).
2021-09-19 12:55:31 +02:00
greatroar
92ae951ffa
Report timing from restic restore --verify
2021-09-19 12:53:09 +02:00
greatroar
973fa921cb
Test and document Restorer.VerifyFiles
2021-09-19 12:52:11 +02:00
MichaelEischer
ef5672a902
Merge pull request #3509 from ajaspers/patch-1
...
Update instructions for reproducing build
2021-09-12 21:40:37 +02:00
ajaspers
c0eddc9969
Update instructions for reproducing build
...
Dependencies are fetched at build time and stored in the GOPATH. These paths end up being in the final binary.
Bump restic version to latest and go version to the 1.16.6, which was used to build restic 0.12.1.
2021-09-12 09:02:57 -07:00
Michael Eischer
fbb0e6499a
ui: consolidate backup ui in ui/backup package
2021-09-12 16:20:15 +02:00
Michael Eischer
503d4c3e2f
Add changelog
2021-09-12 16:00:49 +02:00
Michael Eischer
cccb0d4064
ui: assert that ProgressPrinter interface is implemented
2021-09-12 15:26:40 +02:00
Michael Eischer
a144c986f2
backup: Reenable JSON status updates with redirected output
...
After the refactoring status updates were no longer printed in quiet
mode or when the output is not an interactive terminal. However, the
JSON output is often piped to e.g. another program. Thus, don't set the
update frequency to 0 in that case. The status updates are still
disabled for backup --quiet.
This also reduces the status update frequency to 60fps compared to a
potentially much higher value before the refactoring.
2021-09-12 15:26:40 +02:00
Michael Eischer
d62bfed65d
ui: move SetDryRun to ProgressReporter
2021-09-12 15:25:58 +02:00
Michael Eischer
77b129ec74
ui: remove unused Summary field
2021-09-12 15:25:58 +02:00
Michael Eischer
3024239e40
ui/json: restore summary output
2021-09-12 15:25:58 +02:00
Michael Eischer
5ccf583b8a
ui: restore a few comments
2021-09-12 15:25:58 +02:00
