KurlesHS/restic
1
0
Fork 0
forked from TrueCloudLab/restic
Code Pull requests Activity
5645 commits 10 branches 49 tags 79 MiB
Commit graph

1105 commits

Author SHA1 Message Date
MichaelEischer
f2a3b3b4a1
Merge pull request #3537 from greatroar/dump-lru 
Check cap instead of len in bloblru
 2021-10-09 20:00:27 +02:00
Garry McNulty
708d7a2574 s3: Add warning if key ID or secret is empty 
Also add debug message if no credential types are available.

Closes #2388
 2021-10-06 23:13:40 +01:00
greatroar
634a9c162d Check cap instead of len in bloblru 
restic dump uses bloblru.Cache to keep buffers alive, but also reuses
evicted buffers. That means large buffers may be used to store small
blobs, causing the cache to think it's using less memory than it
actually does.
 2021-10-03 09:34:17 +02:00
greatroar
c892c0bab9 internal/restic: Don't allocate in Tree.Insert 
name         old time/op    new time/op    delta
BuildTree-8    34.6µs ± 4%     7.0µs ± 3%  -79.68%  (p=0.000 n=18+19)

name         old alloc/op   new alloc/op   delta
BuildTree-8    34.0kB ± 0%     0.9kB ± 0%  -97.37%  (p=0.000 n=20+20)

name         old allocs/op  new allocs/op  delta
BuildTree-8       108 ± 0%         1 ± 0%  -99.07%  (p=0.000 n=20+20)
 2021-09-26 18:08:48 +02:00
MichaelEischer
78dac2fd48
Merge pull request #3523 from greatroar/swift-v2 
Upgrade ncw/swift to v2
 2021-09-24 21:41:36 +02:00
Michael Eischer
5ea8bba1a1 swift: restore context err check for list() 2021-09-24 21:19:46 +02:00
MichaelEischer
a5e103a212
Merge pull request #3522 from greatroar/dump-lru 
Use LRU cache in restic dump
 2021-09-24 20:33:58 +02:00
greatroar
e7ec0453b1 Upgrade ncw/swift to v2 2021-09-24 19:08:37 +02:00
greatroar
fe04d024c7 Use LRU cache in restic dump 2021-09-24 15:45:08 +02:00
Uli Martens
718966a81a Move Blobcache into dedicated internal package 2021-09-24 15:45:00 +02:00
greatroar
4f33eca634 Remove unused Writer arg to internal/dump.writeDump 2021-09-24 15:40:42 +02:00
Sam Lucidi
897d8e662c Add --insecure-tls flag to disable SSL cert verification 
Signed-off-by: Sam Lucidi <slucidi@redhat.com>
 2021-09-21 10:52:40 -04:00
MichaelEischer
1827b16ade
Merge pull request #3519 from greatroar/maphash 
Replace siphash by hash/maphash
 2021-09-19 19:46:03 +02:00
greatroar
8d2996eaaa Replace siphash by hash/maphash 
In Go 1.17.1, maphash has become quite a bit faster than siphash, so we
can drop one third-party dependency. maphash is just an interface to the
standard Go map's hash function, which we already trust for other use
cases.

Benchmark results on linux/amd64, -benchtime=3s:

name                                             old time/op    new time/op    delta
IndexHasUnknown-8                                  50.6ns ±10%    41.0ns ±19%  -18.92%  (p=0.000 n=9+10)
IndexHasKnown-8                                    52.6ns ±12%    41.5ns ±12%  -21.13%  (p=0.000 n=9+10)
IndexMapHash-8                                     3.64µs ± 1%    2.00µs ± 0%  -45.09%  (p=0.000 n=10+9)
IndexAlloc-8                                        700ms ± 1%     601ms ± 6%  -14.18%  (p=0.000 n=8+10)
IndexAllocParallel-8                                205ms ± 5%     192ms ± 8%   -6.18%  (p=0.043 n=10+10)
MasterIndexAlloc-8                                  319ms ± 1%     279ms ± 5%  -12.58%  (p=0.000 n=10+10)
MasterIndexLookupSingleIndex-8                      156ns ± 8%     147ns ± 6%   -5.46%  (p=0.023 n=10+10)
MasterIndexLookupMultipleIndex-8                    150ns ± 7%     142ns ± 8%   -5.69%  (p=0.007 n=10+10)
MasterIndexLookupSingleIndexUnknown-8              74.4ns ± 6%    72.0ns ± 9%     ~     (p=0.175 n=10+9)
MasterIndexLookupMultipleIndexUnknown-8            67.4ns ± 9%    65.5ns ± 7%     ~     (p=0.340 n=9+9)
MasterIndexLookupParallel/known,indices=25-8        461ns ± 2%     445ns ± 2%   -3.49%  (p=0.000 n=10+10)
MasterIndexLookupParallel/unknown,indices=25-8      408ns ±11%     378ns ± 5%   -7.22%  (p=0.035 n=10+9)
MasterIndexLookupParallel/known,indices=50-8        479ns ± 1%     437ns ± 4%   -8.82%  (p=0.000 n=10+10)
MasterIndexLookupParallel/unknown,indices=50-8      406ns ± 8%     343ns ±15%  -15.44%  (p=0.001 n=10+10)
MasterIndexLookupParallel/known,indices=100-8       480ns ± 1%     455ns ± 5%   -5.15%  (p=0.000 n=8+10)
MasterIndexLookupParallel/unknown,indices=100-8     391ns ±18%     382ns ± 8%     ~     (p=0.315 n=10+10)
MasterIndexLookupBlobSize-8                        71.0ns ± 8%    57.2ns ±11%  -19.36%  (p=0.000 n=9+10)
PackerManager-8                                     254ms ± 1%     254ms ± 1%     ~     (p=0.285 n=15+15)

name                                             old speed      new speed      delta
IndexMapHash-8                                   1.12GB/s ± 1%  2.05GB/s ± 0%  +82.13%  (p=0.000 n=10+9)
PackerManager-8                                   208MB/s ± 1%   207MB/s ± 1%     ~     (p=0.281 n=15+15)

name                                             old alloc/op   new alloc/op   delta
IndexMapHash-8                                      0.00B          0.00B          ~     (all equal)
IndexAlloc-8                                        400MB ± 0%     400MB ± 0%     ~     (p=1.000 n=9+10)
IndexAllocParallel-8                                401MB ± 0%     401MB ± 0%   +0.00%  (p=0.000 n=10+10)
MasterIndexAlloc-8                                  258MB ± 0%     262MB ± 0%   +1.42%  (p=0.000 n=9+10)
PackerManager-8                                    73.1kB ± 0%    73.1kB ± 0%     ~     (p=0.382 n=13+13)

name                                             old allocs/op  new allocs/op  delta
IndexMapHash-8                                       0.00           0.00          ~     (all equal)
IndexAlloc-8                                         907k ± 0%      907k ± 0%   -0.00%  (p=0.000 n=10+10)
IndexAllocParallel-8                                 907k ± 0%      907k ± 0%   +0.00%  (p=0.009 n=10+10)
MasterIndexAlloc-8                                   327k ± 0%      317k ± 0%   -3.06%  (p=0.000 n=10+10)
PackerManager-8                                       744 ± 0%       744 ± 0%     ~     (all equal)
 2021-09-19 16:05:18 +02:00
MichaelEischer
58efe21eca
Merge pull request #3264 from amozoss/upstream-master 
Refactor backup progress
 2021-09-19 14:54:42 +02:00
Michael Eischer
921e328b56 restore: Fix linting error 2021-09-19 14:41:07 +02:00
Michael Eischer
2cdc0719af restorer: Sanitize verify errors 2021-09-19 14:01:26 +02:00
greatroar
bdcdfaf6b4 restore --verify: buffer reuse consistency and comment 2021-09-19 13:11:27 +02:00
greatroar
2b94742ca5 Replace no-op closures in restorer by nil check 2021-09-19 13:11:16 +02:00
greatroar
d357744104 Handle canceled context in restore --verify properly 2021-09-19 13:11:05 +02:00
greatroar
d4225ec803 Simplify buffer growing in Restorer.verifyFile 
Suggested-by: Igor Fedorenko <igor@ifedorenko.com>
 2021-09-19 13:10:55 +02:00
greatroar
de8521ae56 Report number of successes from Restorer.VerifyFiles 
Previously, the number of attempts was reported.
 2021-09-19 13:10:44 +02:00
greatroar
bb066cf7d3 Concurrent Restorer.VerifyFiles 
Time to verify a 2GB snapshot down from 9.726s to 4.645s (-52%).
 2021-09-19 13:10:41 +02:00
greatroar
556424d61b Reuse buffer in Restorer.VerifyFiles 
Time to verify a 2GB snapshot down from 11.568s to 9.726s (-16%).
 2021-09-19 12:55:31 +02:00
greatroar
973fa921cb Test and document Restorer.VerifyFiles 2021-09-19 12:52:11 +02:00
Michael Eischer
fbb0e6499a ui: consolidate backup ui in ui/backup package 2021-09-12 16:20:15 +02:00
Michael Eischer
cccb0d4064 ui: assert that ProgressPrinter interface is implemented 2021-09-12 15:26:40 +02:00
Michael Eischer
d62bfed65d ui: move SetDryRun to ProgressReporter 2021-09-12 15:25:58 +02:00
Michael Eischer
77b129ec74 ui: remove unused Summary field 2021-09-12 15:25:58 +02:00
Michael Eischer
3024239e40 ui/json: restore summary output 2021-09-12 15:25:58 +02:00
Michael Eischer
5ccf583b8a ui: restore a few comments 2021-09-12 15:25:58 +02:00
Michael Eischer
80cbaf6d38 ui: Simplify ReportTotal methods 2021-09-12 15:25:58 +02:00
Dan Willoughby
448419990c Refactor backup progress 
Move the shared logic into the progress

Allows logic to be shared with forth coming restore status
 2021-09-12 15:25:58 +02:00
MichaelEischer
fa5ca8af81
Merge pull request #2856 from aawsome/remove-readahead 
Simplify cache logic
 2021-09-04 20:25:49 +02:00
MichaelEischer
b45d88e124
Merge pull request #3496 from juergenhoetzel/imporove-snapshot-filter-warning-message 
Improve snapshot filter warning message
 2021-09-03 23:42:27 +02:00
MichaelEischer
a29777f467
Merge pull request #3501 from greatroar/printprogress 
Streamline progress printing in cmd/restic
 2021-09-03 23:34:36 +02:00
Alexander Weiss
81876d5c1b Simplify cache logic 2021-09-03 21:01:00 +02:00
greatroar
5aaa3e93c1 internal/ui/termstatus: Optimize and publish Truncate 
name               old time/op  new time/op  delta
TruncateASCII-8     347ns ± 1%    69ns ± 1%  -80.02%  (p=0.000 n=9+10)
TruncateUnicode-8   447ns ± 3%   348ns ± 1%  -22.04%  (p=0.000 n=10+10)
 2021-09-03 11:48:22 +02:00
MichaelEischer
ec2e3b260e
Merge pull request #3499 from greatroar/wrappedconn-pointer 
rclone: Return one fewer value from run
 2021-08-31 21:48:36 +02:00
greatroar
950b818274 rclone: Return one fewer value from run 2021-08-26 18:12:08 +02:00
Juergen Hoetzel
defe19fdf6 Quote snapshot prefix in error string 2021-08-25 16:11:28 +02:00
Michael Eischer
68370feeee backends: Remove TestSaveFilenames test 
Filenames are expected to match the sha256 sum of the file content. This
rule is now enforced by the rest server thus making this test useless.
 2021-08-15 18:24:16 +02:00
Michael Eischer
574c83e47f rest: Fix test to use paths which are the sha256 sum of the data 2021-08-15 18:19:43 +02:00
Michael Eischer
e6a5801155 rest: Fix test backend url 
The rest config normally uses prepareURL to sanitize URLs and ensures
that the URL ends with a slash. However, the test used an URL without a
trailing slash, which after the rest server changes causes test
failures.
 2021-08-15 18:16:17 +02:00
Michael Eischer
f4c5dec05d backend: test that a wrong hash fails an upload 2021-08-04 22:17:46 +02:00
Michael Eischer
7c1903e1ee panic if hash returns an error 
Add a sanity check that the interface contract is honoured.
 2021-08-04 22:17:46 +02:00
Michael Eischer
51b7e3119b mem: calculate md5 content hash for uploads 
The mem backend is primarily used for testing. This ensures that the
upload hash calculation gets appropriate test coverage.
 2021-08-04 22:17:46 +02:00
Michael Eischer
a009b39e4c gs/swift: calculate md5 content hash for upload 2021-08-04 22:17:46 +02:00
Michael Eischer
1d3e99f475 azure: check upload using md5 content hash 
For files below 256MB this uses the md5 hash calculated while assembling
the pack file. For larger files the hash for each 100MB part is
calculated on the fly. That hash is also reused as temporary filename.
As restic only uploads encrypted data which includes among others a
random initialization vector, the file hash shouldn't be susceptible to
md5 collision attacks (even though the algorithm is broken).
 2021-08-04 22:17:46 +02:00
Michael Eischer
9aa2eff384 Add plumbing to calculate backend specific file hash for upload 
This enables the backends to request the calculation of a
backend-specific hash. For the currently supported backends this will
always be MD5. The hash calculation happens as early as possible, for
pack files this is during assembly of the pack file. That way the hash
would even capture corruptions of the temporary pack file on disk.
 2021-08-04 22:17:46 +02:00