KurlesHS/restic
1
0
Fork 0
forked from TrueCloudLab/restic
Code Pull requests Activity
6257 commits 10 branches 49 tags 79 MiB
Commit graph

6257 commits

This branch
This branch
All branches
Author SHA1 Message Date
Michael Eischer
c147422ba5 repository: special case SaveBlob for all zero chunks 
Sparse files contain large regions containing only zero bytes. Checking
that a blob only contains zeros is possible with over 100GB/s for modern
x86 CPUs. Calculating sha256 hashes is only possible with 500MB/s (or
2GB/s using hardware acceleration). Thus we can speed up the hash
calculation for all zero blobs (which always have length
chunker.MinSize) by checking for zero bytes and then using the
precomputed hash.

The all zeros check is only performed for blobs with the minimal chunk
size, and thus should add no overhead most of the time. For chunks which
are not all zero but have the minimal chunks size, the overhead will be
below 2% based on the above performance numbers.

This allows reading sparse sections of files as fast as the kernel can
return data to us. On my system using BTRFS this resulted in about
4GB/s.
 2022-09-24 21:39:39 +02:00
Michael Eischer
34fe1362da restorer: move zeroPrefixLen to restic package 2022-09-24 21:39:39 +02:00
Michael Eischer
a5ebd5de4b restorer: Fix race condition in partialFile.WriteAt 
The restorer can issue multiple calls to WriteAt in parallel. This can
result in unexpected orderings of the Truncate and WriteAt calls and
sometimes too short restored files.
 2022-09-24 21:39:39 +02:00
Michael Eischer
5b6a77058a Enable sparseness only conditionally 
We can either preallocate storage for a file or sparsify it. This
detects a pack file as sparse if it contains an all zero block or
consists of only one block. As the file sparsification is just an
approximation, hide it behind a `--sparse` parameter.
 2022-09-24 21:20:00 +02:00
greatroar
3047bf611c Changelog entry for sparse file restoring 2022-09-24 21:18:48 +02:00
greatroar
5d4568d393 Write sparse files in restorer 
This writes files by using (*os.File).Truncate, which resolves to the
truncate system call on Unix.

Compared to the naive loop,

	for _, b := range p {
		if b != 0 {
			return false
		}
	}

the optimized allZero is about 10× faster:

name       old time/op    new time/op     delta
AllZero-8    1.09ms ± 1%     0.09ms ± 1%    -92.10%  (p=0.000 n=10+10)

name       old speed      new speed       delta
AllZero-8  3.84GB/s ± 1%  48.59GB/s ± 1%  +1166.51%  (p=0.000 n=10+10)
 2022-09-24 21:18:48 +02:00
Michael Eischer
b48766d7b8
Merge pull request #3928 from restic/rawtaz-doc-b2-s3 
doc: Clarify S3 recommendation for B2 slightly
 2022-09-13 20:59:50 +02:00
rawtaz
20f1913ef7
doc: Clarify S3 recommendation for B2 slightly 
This gives slightly more background to the recommendation for using restic's S3 backend with Backblaze B2.
 2022-09-12 17:48:59 +02:00
rawtaz
d79e61ce5d
Merge pull request #3925 from hgraeber/add-powershell-completion 
Add powershell completion
 2022-09-11 01:04:57 +02:00
Herbert Graeber
988b386e8b Add powershell completion 
- Add code for powersehll complition available in cobra
- Add documentation for powershell completion
- Add changelog for pr3925
 2022-09-11 00:44:12 +02:00
rawtaz
14d09a6081
Merge pull request #3912 from MichaelEischer/cleanup-snapshot-filter-options 
Clean up snapshot filter options
 2022-09-11 00:18:42 +02:00
Michael Eischer
381da0443a tweak snapshot filter descriptions 2022-09-10 23:50:20 +02:00
Michael Eischer
8b9778d537
Merge pull request #3900 from MichaelEischer/b2-init-timeout 
Add timeout for the initial connection to B2
 2022-09-10 23:28:59 +02:00
Michael Eischer
17c27400f8
Merge pull request #3921 from MichaelEischer/filter-cleanup-error-handling 
filter: deduplicate error handling for pattern validation
 2022-09-10 23:24:50 +02:00
Michael Eischer
f76643bd2e
Merge pull request #3894 from MichaelEischer/filter-mount-exit-code 
Mount should return exit code 0 after pressing Ctrl-C
 2022-09-10 23:22:01 +02:00
Michael Eischer
be9ccc186e
Merge pull request #3875 from MichaelEischer/fix-fuse-context-cancel 
mount: Fix input/output errors for canceled syscalls
 2022-09-10 23:20:29 +02:00
Michael Eischer
2363e5c083
Merge pull request #3913 from MichaelEischer/better-migrate-error-message 
migrate: Report why an migration cannot be applied
 2022-09-09 23:37:25 +02:00
Michael Eischer
8e0ca80547 filter: deduplicate error handling for pattern validation 2022-09-09 23:12:41 +02:00
Michael Eischer
d6309961c5 deduplicate the snapshot filter cli option setup 2022-09-04 10:27:33 +02:00
Michael Eischer
8b4dd70013 migrate: Report why an migration cannot be applied 
Just returning that `Migration upgrade cannot be applied: check failed`
is not too useful when running `migrate upgrade_repo_v2`.
 2022-09-03 11:49:31 +02:00
Michael Eischer
7689d6c679 normalize help text for host, tag and path options 2022-09-03 00:06:38 +02:00
Michael Eischer
6c69f08a7b
Merge pull request #3905 from DRON-666/haspaths-linear 
Reduce quadratic time complexity of `Snapshot.HasPaths`
 2022-08-30 20:35:56 +02:00
Michael Eischer
3e70bac56e
Merge pull request #3898 from MichaelEischer/fix-copy-hang 
don't hang when `copy` uses a single connection
 2022-08-30 20:23:39 +02:00
DRON-666
2a630c51c1 Add changelog 2022-08-30 20:22:07 +02:00
DRON-666
d0f1060df7 Fix quadratic time complexity of Snapshot.HasPaths 2022-08-30 04:38:17 +03:00
Michael Eischer
f481ad64c8
Merge pull request #3904 from lbausch/add-newline 
Add newline to keep prompt intact
 2022-08-29 21:43:18 +02:00
Lorenz Bausch
7ddd803e46
Add newline to keep prompt intact 2022-08-29 17:37:49 +02:00
Michael Eischer
e5b2c4d571 b2: sniff the error that caused init retry loops 2022-08-28 17:46:03 +02:00
Michael Eischer
dc2db2de5e b2: cancel connection setup after a minute 
If the connection to B2 fails, the library enters an endless loop.
 2022-08-28 14:56:17 +02:00
Michael Eischer
7682149c9d repository: cleanup copy connection count check 2022-08-28 11:40:56 +02:00
Michael Eischer
b03277ead5 repository: don't hang when copying using a single connection 2022-08-28 11:40:31 +02:00
Michael Eischer
1b233c4e2e
Merge pull request #2661 from creativeprojects/issue-1734 
"Everything is ok" message after retrying
 2022-08-28 11:04:59 +02:00
Fred
4042db5169 Add changelog 2022-08-27 22:36:19 +02:00
Fred
be6baaec12 Add success callback to the backend 2022-08-27 22:27:15 +02:00
Fred
baf58fbaa8 Add unit tests 2022-08-27 22:21:06 +02:00
Fred
d629333efe Add function to notify of success after retrying 2022-08-27 22:21:06 +02:00
Alexander Neumann
c169e37139
Merge pull request #3895 from MichaelEischer/refactor-cat-key 
cat: Simplify implementation of 'cat key'
 2022-08-27 18:40:46 +02:00
Michael Eischer
1b4af0c6e5 cat: Simplify implementation of 'cat key' 2022-08-26 23:21:51 +02:00
Michael Eischer
3174641ca4 add changelog for mount exit code filtering 2022-08-26 23:17:04 +02:00
Michael Eischer
5478ab22c5 mount: return exit code 0 after receiving a SIGINT 2022-08-26 23:07:07 +02:00
Michael Eischer
d768c1c3e4 Allow cleanup handlers to filter the exit code 2022-08-26 23:04:59 +02:00
Michael Eischer
908f7441fe
Merge pull request #3885 from MichaelEischer/delete-fixes 
Improve reliability of upload retries and B2 file deletions
 2022-08-26 22:30:50 +02:00
Michael Eischer
4c90d91d4d backend: Test that failed uploads are not removed for backends with atomic replace 2022-08-26 21:20:52 +02:00
Michael Eischer
694dfa026a add changelog for reliable B2 deletes 2022-08-26 21:20:46 +02:00
MichaelEischer
582167d671
Merge pull request #3882 from MichaelEischer/sftp-init-single-connection 
sftp: Only connect once to server during `init`
 2022-08-26 21:13:28 +02:00
MichaelEischer
3822ded0b3
Merge pull request #3877 from MichaelEischer/no-env-in-help 
Do not include the actual values of environment variables in help output
 2022-08-26 20:59:54 +02:00
Michael Eischer
cf0a8d7758 sftp: Only connect once for repository creation 
This is especially useful if ssh asks for a password or if closing the
initial connection could return an error due to a problematic server
implementation.
 2022-08-26 20:50:40 +02:00
Michael Eischer
dd7cd5b9b3 fuse: remove unused context parameter 2022-08-26 20:48:48 +02:00
Michael Eischer
a0c1ae9f90 mount: Correctly return context.Canceled for interrupted syscalls 
bazil/fuse expects us to return context.Canceled to signal that a
syscall was successfully interrupted. Returning a wrapped version of
that error however causes the fuse library to signal an EIO (input/output
error). Thus unwrap context.Canceled errors before returning them.
 2022-08-26 20:48:48 +02:00
Michael Eischer
5d0649faaf Update help output in docs 2022-08-26 20:44:01 +02:00