Michael Eischer
78d2312ee9
Merge pull request #3854 from MichaelEischer/sparsefiles
...
restore: Add support for sparse files
2022-09-24 22:04:02 +02:00
Michael Eischer
19afad8a09
restore: support sparse restores also on windows
2022-09-24 21:39:39 +02:00
Michael Eischer
0f89f443c7
update sparse restore changelog
2022-09-24 21:39:39 +02:00
Michael Eischer
c147422ba5
repository: special case SaveBlob for all zero chunks
...
Sparse files contain large regions containing only zero bytes. Checking
that a blob only contains zeros is possible with over 100GB/s for modern
x86 CPUs. Calculating sha256 hashes is only possible with 500MB/s (or
2GB/s using hardware acceleration). Thus we can speed up the hash
calculation for all zero blobs (which always have length
chunker.MinSize) by checking for zero bytes and then using the
precomputed hash.
The all zeros check is only performed for blobs with the minimal chunk
size, and thus should add no overhead most of the time. For chunks which
are not all zero but have the minimal chunks size, the overhead will be
below 2% based on the above performance numbers.
This allows reading sparse sections of files as fast as the kernel can
return data to us. On my system using BTRFS this resulted in about
4GB/s.
2022-09-24 21:39:39 +02:00
Michael Eischer
34fe1362da
restorer: move zeroPrefixLen to restic package
2022-09-24 21:39:39 +02:00
Michael Eischer
a5ebd5de4b
restorer: Fix race condition in partialFile.WriteAt
...
The restorer can issue multiple calls to WriteAt in parallel. This can
result in unexpected orderings of the Truncate and WriteAt calls and
sometimes too short restored files.
2022-09-24 21:39:39 +02:00
Michael Eischer
5b6a77058a
Enable sparseness only conditionally
...
We can either preallocate storage for a file or sparsify it. This
detects a pack file as sparse if it contains an all zero block or
consists of only one block. As the file sparsification is just an
approximation, hide it behind a `--sparse` parameter.
2022-09-24 21:20:00 +02:00
greatroar
3047bf611c
Changelog entry for sparse file restoring
2022-09-24 21:18:48 +02:00
greatroar
5d4568d393
Write sparse files in restorer
...
This writes files by using (*os.File).Truncate, which resolves to the
truncate system call on Unix.
Compared to the naive loop,
for _, b := range p {
if b != 0 {
return false
}
}
the optimized allZero is about 10× faster:
name old time/op new time/op delta
AllZero-8 1.09ms ± 1% 0.09ms ± 1% -92.10% (p=0.000 n=10+10)
name old speed new speed delta
AllZero-8 3.84GB/s ± 1% 48.59GB/s ± 1% +1166.51% (p=0.000 n=10+10)
2022-09-24 21:18:48 +02:00
Michael Eischer
eb83402d39
Merge pull request #3935 from miles170/master
...
Only display the message if there were locks to be removed
2022-09-24 20:53:13 +02:00
Michael Eischer
ef58ddd7b1
Merge pull request #3923 from MichaelEischer/fix-flaky-cache-test
...
cache: fix flaky TestFileSaveConcurrent on windows
2022-09-24 20:52:55 +02:00
Michael Eischer
7fc178aaf4
internal/cache: extend description of cache sharing test failure
2022-09-24 13:07:01 +02:00
Miles Liu
1acbda18f8
Only display the message if there were locks to be removed
...
`restic unlock` now only shows `successfully removed locks` if there were locks to be removed.
In addition, it also reports the number of the removed lock files.
2022-09-24 19:02:24 +08:00
Michael Eischer
da1a359c8b
Merge pull request #3927 from MichaelEischer/faster-index-each
...
Speed up MasterIndex.Each
2022-09-24 12:35:23 +02:00
Michael Eischer
041a51512a
Merge pull request #3780 from jkmw/fix/2578
...
Remove existing path before restoring a symlink
2022-09-24 12:34:42 +02:00
Michael Eischer
1ebd57247a
repository: optimize MasterIndex.Each
...
Sending data through a channel at very high frequency is extremely
inefficient. Thus use simple callbacks instead of channels.
> name old time/op new time/op delta
> MasterIndexEach-16 6.68s ±24% 0.96s ± 2% -85.64% (p=0.008 n=5+5)
2022-09-24 12:21:59 +02:00
Michael Eischer
825b95e313
repository: add benchmark for MasterIndex.Each
2022-09-24 12:21:59 +02:00
greatroar
1220fe9650
internal/cache: Concurrent use of cache not working on Windows
2022-09-17 19:49:44 +02:00
Jerome Küttner
ef618bdd3f
use os.Remove if path already exists on symlink restore
2022-09-14 08:14:31 +02:00
Michael Eischer
b48766d7b8
Merge pull request #3928 from restic/rawtaz-doc-b2-s3
...
doc: Clarify S3 recommendation for B2 slightly
2022-09-13 20:59:50 +02:00
rawtaz
20f1913ef7
doc: Clarify S3 recommendation for B2 slightly
...
This gives slightly more background to the recommendation for using restic's S3 backend with Backblaze B2.
2022-09-12 17:48:59 +02:00
rawtaz
d79e61ce5d
Merge pull request #3925 from hgraeber/add-powershell-completion
...
Add powershell completion
2022-09-11 01:04:57 +02:00
Herbert Graeber
988b386e8b
Add powershell completion
...
- Add code for powersehll complition available in cobra
- Add documentation for powershell completion
- Add changelog for pr3925
2022-09-11 00:44:12 +02:00
rawtaz
14d09a6081
Merge pull request #3912 from MichaelEischer/cleanup-snapshot-filter-options
...
Clean up snapshot filter options
2022-09-11 00:18:42 +02:00
Michael Eischer
381da0443a
tweak snapshot filter descriptions
2022-09-10 23:50:20 +02:00
Michael Eischer
8b9778d537
Merge pull request #3900 from MichaelEischer/b2-init-timeout
...
Add timeout for the initial connection to B2
2022-09-10 23:28:59 +02:00
Michael Eischer
17c27400f8
Merge pull request #3921 from MichaelEischer/filter-cleanup-error-handling
...
filter: deduplicate error handling for pattern validation
2022-09-10 23:24:50 +02:00
Michael Eischer
f76643bd2e
Merge pull request #3894 from MichaelEischer/filter-mount-exit-code
...
Mount should return exit code 0 after pressing Ctrl-C
2022-09-10 23:22:01 +02:00
Michael Eischer
be9ccc186e
Merge pull request #3875 from MichaelEischer/fix-fuse-context-cancel
...
mount: Fix input/output errors for canceled syscalls
2022-09-10 23:20:29 +02:00
Michael Eischer
2363e5c083
Merge pull request #3913 from MichaelEischer/better-migrate-error-message
...
migrate: Report why an migration cannot be applied
2022-09-09 23:37:25 +02:00
Michael Eischer
8e0ca80547
filter: deduplicate error handling for pattern validation
2022-09-09 23:12:41 +02:00
Michael Eischer
d6309961c5
deduplicate the snapshot filter cli option setup
2022-09-04 10:27:33 +02:00
Michael Eischer
8b4dd70013
migrate: Report why an migration cannot be applied
...
Just returning that `Migration upgrade cannot be applied: check failed`
is not too useful when running `migrate upgrade_repo_v2`.
2022-09-03 11:49:31 +02:00
Michael Eischer
7689d6c679
normalize help text for host, tag and path options
2022-09-03 00:06:38 +02:00
Michael Eischer
6c69f08a7b
Merge pull request #3905 from DRON-666/haspaths-linear
...
Reduce quadratic time complexity of `Snapshot.HasPaths`
2022-08-30 20:35:56 +02:00
Michael Eischer
3e70bac56e
Merge pull request #3898 from MichaelEischer/fix-copy-hang
...
don't hang when `copy` uses a single connection
2022-08-30 20:23:39 +02:00
DRON-666
2a630c51c1
Add changelog
2022-08-30 20:22:07 +02:00
DRON-666
d0f1060df7
Fix quadratic time complexity of Snapshot.HasPaths
2022-08-30 04:38:17 +03:00
Michael Eischer
f481ad64c8
Merge pull request #3904 from lbausch/add-newline
...
Add newline to keep prompt intact
2022-08-29 21:43:18 +02:00
Lorenz Bausch
7ddd803e46
Add newline to keep prompt intact
2022-08-29 17:37:49 +02:00
Michael Eischer
e5b2c4d571
b2: sniff the error that caused init retry loops
2022-08-28 17:46:03 +02:00
Michael Eischer
dc2db2de5e
b2: cancel connection setup after a minute
...
If the connection to B2 fails, the library enters an endless loop.
2022-08-28 14:56:17 +02:00
Michael Eischer
7682149c9d
repository: cleanup copy connection count check
2022-08-28 11:40:56 +02:00
Michael Eischer
b03277ead5
repository: don't hang when copying using a single connection
2022-08-28 11:40:31 +02:00
Michael Eischer
1b233c4e2e
Merge pull request #2661 from creativeprojects/issue-1734
...
"Everything is ok" message after retrying
2022-08-28 11:04:59 +02:00
Fred
4042db5169
Add changelog
2022-08-27 22:36:19 +02:00
Fred
be6baaec12
Add success callback to the backend
2022-08-27 22:27:15 +02:00
Fred
baf58fbaa8
Add unit tests
2022-08-27 22:21:06 +02:00
Fred
d629333efe
Add function to notify of success after retrying
2022-08-27 22:21:06 +02:00
Alexander Neumann
c169e37139
Merge pull request #3895 from MichaelEischer/refactor-cat-key
...
cat: Simplify implementation of 'cat key'
2022-08-27 18:40:46 +02:00
