Commit graph

93 commits

Author SHA1 Message Date
Michael Eischer
34fe1362da restorer: move zeroPrefixLen to restic package 2022-09-24 21:39:39 +02:00
Michael Eischer
a5ebd5de4b restorer: Fix race condition in partialFile.WriteAt
The restorer can issue multiple calls to WriteAt in parallel. This can
result in unexpected orderings of the Truncate and WriteAt calls and
sometimes too short restored files.
2022-09-24 21:39:39 +02:00
Michael Eischer
5b6a77058a Enable sparseness only conditionally
We can either preallocate storage for a file or sparsify it. This
detects a pack file as sparse if it contains an all zero block or
consists of only one block. As the file sparsification is just an
approximation, hide it behind a `--sparse` parameter.
2022-09-24 21:20:00 +02:00
greatroar
5d4568d393 Write sparse files in restorer
This writes files by using (*os.File).Truncate, which resolves to the
truncate system call on Unix.

Compared to the naive loop,

	for _, b := range p {
		if b != 0 {
			return false
		}
	}

the optimized allZero is about 10× faster:

name       old time/op    new time/op     delta
AllZero-8    1.09ms ± 1%     0.09ms ± 1%    -92.10%  (p=0.000 n=10+10)

name       old speed      new speed       delta
AllZero-8  3.84GB/s ± 1%  48.59GB/s ± 1%  +1166.51%  (p=0.000 n=10+10)
2022-09-24 21:18:48 +02:00
Michael Eischer
f414db987d gofmt all files
Apparently the rules for comment formatting have changed with go 1.19.
2022-08-19 19:12:26 +02:00
MichaelEischer
443cc49afd
Merge pull request #3830 from MichaelEischer/cleanup-repo
Extract Load/SaveTree/JSONUnpacked from repository
2022-07-23 10:46:13 +02:00
Michael Eischer
c44b21d366 restorer: extract hardlinks index from restic package 2022-07-17 13:45:42 +02:00
Michael Eischer
8c11fc3ec9 crypto: move crypto buffer helpers 2022-07-17 13:42:23 +02:00
Michael Eischer
89d3ce852b repository: extract Load/StoreJSONUnpacked
A Load/Store method for each data type is much clearer. As a result the
repository no longer needs a method to load / store json.
2022-07-17 13:22:00 +02:00
Michael Eischer
fbcbd5318c repository: extract LoadTree/SaveTree
The repository has no real idea what a Tree is. So these methods never
belonged there.
2022-07-17 13:11:28 +02:00
Michael Eischer
6f53ecc1ae adapt workers based on whether an operation is CPU or IO-bound
Use runtime.GOMAXPROCS(0) as worker count for CPU-bound tasks,
repo.Connections() for IO-bound task and a combination if a task can be
both. Streaming packs is treated as IO-bound as adding more worker
cannot provide a speedup.

Typical IO-bound tasks are download / uploading / deleting files.
Decoding / Encoding / Verifying are usually CPU-bound. Several tasks are
a combination of both, e.g. for combined download and decode functions.
In the latter case add both limits together. As the backends have their
own concurrency limits restic still won't download more than
repo.Connections() files in parallel, but the additional workers can
decode already downloaded data in parallel.
2022-07-03 12:19:26 +02:00
Michael Eischer
120ccc8754 repository: Rework blob saving to use an async pack uploader
Previously, SaveAndEncrypt would assemble blobs into packs and either
return immediately if the pack is not yet full or upload the pack file
otherwise. The upload will block the current goroutine until it
finishes.

Now, the upload is done using separate goroutines. This requires changes
to the error handling. As uploads are no longer tied to a SaveAndEncrypt
call, failed uploads are signaled using an errgroup.

To count the uploaded amount of data, the pack header overhead is no
longer returned by `packer.Finalize` but rather by
`packer.HeaderOverhead`. This helper method is necessary to continue
returning the pack header overhead directly to the responsible call to
`repository.SaveBlob`. Without the method this would not be possible,
as packs are finalized asynchronously.
2022-07-02 22:42:34 +02:00
Alexander Neumann
99634c0936 Return real size from SaveBlob 2022-07-02 18:55:12 +02:00
Michael Eischer
6fb408d90e repository: implement pack compression 2022-04-30 11:34:10 +02:00
Alexander Neumann
db8a958991
Merge pull request #3683 from MichaelEischer/fix-golangci-lint-warnings
Fix golangci lint warnings
2022-03-29 11:45:10 +02:00
Michael Eischer
c60540b196 add go:build headers everywhere 2022-03-28 22:23:47 +02:00
Michael Eischer
243698680a crypto: Use helpers for size calculations 2022-03-28 22:09:49 +02:00
Michael Eischer
27524979e8 restorer: Remove dead code 2022-02-13 11:43:09 +01:00
Michael Eischer
f40abd92fa restorer: convert to use StreamPack 2022-02-12 20:18:25 +01:00
Michael Eischer
921e328b56 restore: Fix linting error 2021-09-19 14:41:07 +02:00
Michael Eischer
2cdc0719af restorer: Sanitize verify errors 2021-09-19 14:01:26 +02:00
greatroar
bdcdfaf6b4 restore --verify: buffer reuse consistency and comment 2021-09-19 13:11:27 +02:00
greatroar
2b94742ca5 Replace no-op closures in restorer by nil check 2021-09-19 13:11:16 +02:00
greatroar
d357744104 Handle canceled context in restore --verify properly 2021-09-19 13:11:05 +02:00
greatroar
d4225ec803 Simplify buffer growing in Restorer.verifyFile
Suggested-by: Igor Fedorenko <igor@ifedorenko.com>
2021-09-19 13:10:55 +02:00
greatroar
de8521ae56 Report number of successes from Restorer.VerifyFiles
Previously, the number of attempts was reported.
2021-09-19 13:10:44 +02:00
greatroar
bb066cf7d3 Concurrent Restorer.VerifyFiles
Time to verify a 2GB snapshot down from 9.726s to 4.645s (-52%).
2021-09-19 13:10:41 +02:00
greatroar
556424d61b Reuse buffer in Restorer.VerifyFiles
Time to verify a 2GB snapshot down from 11.568s to 9.726s (-16%).
2021-09-19 12:55:31 +02:00
greatroar
973fa921cb Test and document Restorer.VerifyFiles 2021-09-19 12:52:11 +02:00
Alexander Weiss
81876d5c1b Simplify cache logic 2021-09-03 21:01:00 +02:00
Michael Eischer
bd316d3893 restore: Test partial pack downloads in filerestorer 2021-06-29 21:11:30 +02:00
Michael Eischer
e8bbb05328 restore: Correctly handle partial pack download errors
Failed pack/blob downloads should be retried. For blobs that fail
decryption assume that the pack file is really damaged and try to
restore the remaining blobs.
2021-06-29 20:54:16 +02:00
MichaelEischer
cc254dfefe
Merge pull request #3362 from greatroar/darwin-preallocate
Use FcntlFstore to preallocate on Mac
2021-04-10 22:47:56 +02:00
greatroar
23531be272 Use FcntlFstore to preallocate on Mac 2021-04-10 16:54:07 +02:00
Michael Eischer
f70aca6f6f restorer: Skip preallocate test if not supported by the filesystem 2021-03-04 20:33:46 +01:00
Michael Eischer
47faf69230 restorer: Check dropped error 2021-01-31 18:06:28 +01:00
Alexander Weiss
5e22ae10f1 Add error handling for fileRestorer 2021-01-31 14:22:57 +01:00
Alexander Weiss
573221aa40 Add failing test for fileRestorer 2021-01-31 13:40:42 +01:00
Alexander Neumann
04ca69cc78 Address issues reported by golint 2021-01-30 20:45:57 +01:00
Alexander Neumann
0858fbf6aa Add more error handling 2021-01-30 20:19:47 +01:00
Michael Eischer
69d5b4c36b restorer: lower-case variable name 2021-01-03 13:55:59 +01:00
Alexander Weiss
34a33565c8 Fix loadBlob in filerestorer 2021-01-01 08:06:04 +01:00
Alexander Weiss
7409225fa8 Add filerestorer test where only parts of pack are used 2021-01-01 07:24:46 +01:00
Alexander Weiss
07b3f65a6f filesrestorer: Re-use buffer 2021-01-01 07:24:46 +01:00
Alexander Weiss
3e0acf1395 restore: Don't save (part of) pack in memory 2021-01-01 07:24:46 +01:00
Alexander Neumann
36c5d39c2c Fix issues reported by semgrep 2020-12-11 09:41:59 +01:00
Alexander Weiss
aa7a5f19c2 Use BlobHandle in index methods 2020-11-22 20:41:12 +01:00
Alexander Weiss
92bd448691 Make BlobHandle substruct of Blob 2020-11-22 20:41:10 +01:00
Alexander Neumann
56883817d8
Merge pull request #2990 from MichaelEischer/fix-goreport-warnings
Fix some goreport warnings
2020-10-12 20:44:56 +02:00
Alexander Neumann
c193cea119 Pass Context to NewRestorer() 2020-10-10 15:24:26 +02:00