alexvanin/restic
1
0
Fork 0
forked from TrueCloudLab/restic
Code Pull requests Activity
6017 commits 8 branches 49 tags 79 MiB
Commit graph

1279 commits

Author SHA1 Message Date
Michael Eischer
04c23fa95d rebuild-index: correctly rebuild index for mixed packs 
For mixed packs, data and tree blobs were stored in separate index
entries. This results in warning from the check command and maybe other
problems.
 2022-07-02 19:24:02 +02:00
MichaelEischer
bb5f196b09
Merge pull request #3733 from restic/improve-stats 
Improve stats
 2022-07-02 19:07:31 +02:00
MichaelEischer
c16f989d4a
Merge pull request #3470 from MichaelEischer/sanitize-debug-log 
Sanitize debug log
 2022-07-02 19:00:54 +02:00
Michael Eischer
a6e9e08034 Account for pack header overhead at each entry 
This will miss the pack header crypto overhead and the length field,
which only amount to a few bytes per pack file.
 2022-07-02 18:55:58 +02:00
Alexander Neumann
6c4ceaf1e7 Print number of bytes added to the repo 
This includes optional compression and crypto overhead.
 2022-07-02 18:55:12 +02:00
Alexander Neumann
99634c0936 Return real size from SaveBlob 2022-07-02 18:55:12 +02:00
MichaelEischer
fdc53a9d32
Merge pull request #3787 from MichaelEischer/refactor-repository 
repository: (Mostly) index-related cleanups
 2022-07-02 18:54:04 +02:00
Michael Eischer
6923353c43 redact swift auth token in debug output 2022-07-02 18:47:35 +02:00
Michael Eischer
5a11d14082 redacted keys/token in backend config debug log 2022-07-02 18:47:35 +02:00
Michael Eischer
0936d864a4 redact http authorization header in debug log output 2022-07-02 18:47:35 +02:00
Michael Eischer
ec7c9ce88b drop unused repository.Loader interface 2022-07-02 18:39:59 +02:00
Michael Eischer
2cd7e90ad1 repository: cleanup 2022-07-02 18:39:59 +02:00
Michael Eischer
c1a8fa4290 repository: remove unused packIDToIndex field 2022-07-02 18:39:59 +02:00
Michael Eischer
e68c3a4e62 repository: simplify CreateIndexFromPacks 2022-07-02 18:39:59 +02:00
Michael Eischer
1974ad7ce2 repository: hide MasterIndex.FinalizeFullIndexes / FinalizeNotFinalIndexes 2022-07-02 18:39:59 +02:00
Michael Eischer
ef53ca4a5a repository: remove MasterIndex.All() 2022-07-02 18:39:59 +02:00
Michael Eischer
bf81bf0795 repository: Properly set id for finalized index 
As MergeFinalIndex and index uploads can occur concurrently, it is
necessary for MergeFinalIndex to check whether the IDs for an index were
already set before merging it. Otherwise, we'd loose the ID of an index
which is set _after_ uploading it.
 2022-07-02 18:39:59 +02:00
Michael Eischer
e0a7852b8b repository: remove unused (Master)Index.Count 2022-07-02 18:39:58 +02:00
Michael Eischer
8ef2968f28 repository: remove unused index.ListPack 2022-07-02 18:39:12 +02:00
Michael Eischer
e4f20dea61 repository: inline index.encode 2022-07-02 18:39:12 +02:00
Michael Eischer
fe5a8e137a repository: remove unused index.Store 2022-07-02 18:39:12 +02:00
Michael Eischer
628ae799ca repository: make flushPacks private 2022-07-02 18:39:12 +02:00
Michael Eischer
ed8aa15376 repository: add Save method to MasterIndex interface 2022-07-02 18:38:56 +02:00
Michael Eischer
a77d5c4d11 repository: index saving belongs into the MasterIndex 2022-07-02 18:38:56 +02:00
greatroar
a0fa9c6e9f Revert "restic prune: Merge three loops over the index" 
This reverts commit 8bdfcf779f.
Should fix #3809. Also needed to make #3290 apply cleanly.
 2022-06-30 15:27:34 +02:00
MichaelEischer
bc96879d41
Merge pull request #3785 from MichaelEischer/replace-tomb-usage 
Remove usage of tomb package
 2022-06-19 14:42:48 +02:00
MichaelEischer
307f14604f
Merge pull request #3795 from greatroar/sema 
backend: Move semaphores to a dedicated package
 2022-06-18 17:12:01 +02:00
MichaelEischer
19581dbc18
Merge pull request #3786 from greatroar/prune 
restic prune: Merge three loops over the index
 2022-06-18 16:54:50 +02:00
greatroar
8bdfcf779f restic prune: Merge three loops over the index 
There were three loops over the index in restic prune, to find
duplicates, to determine sizes (in pack.Size) and to generate packInfos.
These three are now one loop. This way, prune doesn't need to construct
a set of duplicate blobs, pack.Size doesn't need to contain special
logic for prune's use case (the onlyHdr argument) and pack.Size doesn't
need to construct a map only to have it immediately transformed into a
different map.

Some quick testing on a 160GiB local repo doesn't show running time or
memory use of restic prune --dry-run changing significantly.
 2022-06-18 10:40:33 +02:00
greatroar
910d917b71 backend: Move semaphores to a dedicated package 
... called backend/sema. I resisted the temptation to call the main
type sema.Phore. Also, semaphores are now passed by value to skip a
level of indirection when using them.
 2022-06-18 10:01:58 +02:00
MichaelEischer
2c893fe43c
Merge pull request #3798 from greatroar/errors 
all: Move away from pkg/errors, easy cases
 2022-06-17 19:01:40 +02:00
greatroar
f92ecf13c9 all: Move away from pkg/errors, easy cases 
github.com/pkg/errors is no longer getting updates, because Go 1.13
went with the more flexible errors.{As,Is} function. Use those instead:
errors from pkg/errors already support the Unwrap interface used by 1.13
error handling. Also:

* check for io.EOF with a straight ==. That value should not be wrapped,
  and the chunker (whose error is checked in the cases changed) does not
  wrap it.
* Give custom Error methods pointer receivers, so there's no ambiguity
  when type-switching since the value type will no longer implement error.
* Make restic.ErrAlreadyLocked private, and rename it to
  alreadyLockedError to match the stdlib convention that error type
  names end in Error.
* Same with rest.ErrIsNotExist => rest.notExistError.
* Make s3.Backend.IsAccessDenied a private function.
 2022-06-14 08:36:38 +02:00
Jayson Wang
f144920ed5 fix handling of maxKeys in SearchKey 2022-06-12 14:19:06 +02:00
Alexander Neumann
1dd4b9b60e
Merge pull request #3788 from greatroar/sftp-posix-rename 
backend/sftp: Support atomic rename
 2022-06-06 19:39:48 +02:00
Alexander Neumann
07114ccb21
Merge pull request #3789 from greatroar/fix-loadblob 
internal/repository: Fix LoadBlob + fuzz test
 2022-06-06 19:33:42 +02:00
greatroar
c9557b2822 internal/repository: Fix LoadBlob + fuzz test 
When given a buf that is big enough for a compressed blob but not its
decompressed contents, the copy at the end of LoadBlob would skip the
last part of the contents.

Fixes #3783.
 2022-06-06 17:02:28 +02:00
greatroar
fa8f02292e backend/sftp: Support atomic rename 
... if the server has posix-rename@openssh.com.
OpenSSH introduced this extension in 2008:
7c29661471
 2022-06-06 13:40:42 +02:00
Michael Eischer
e002b09d57 archiver: free workers once finished 2022-06-05 15:48:10 +02:00
Michael Eischer
408ac1a0c2 archiver: remove tomb usage 2022-06-05 15:47:52 +02:00
Michael Eischer
5eba1217e7 migrate: Cleanup option to request repository check 2022-06-04 23:45:00 +02:00
MichaelEischer
60ca6b1418
Merge pull request #3774 from greatroar/archiver-pool 
archiver: Remove cleanup goroutine from BufferPool
 2022-06-04 18:50:24 +02:00
greatroar
0db1d11b2e archiver: Remove cleanup goroutine from BufferPool 
This isn't doing anything. Channels should get cleaned up by the GC when
the last reference to them disappears, just like all other data
structures. Also inlined BufferPool.Put in Buffer.Release, its only
caller.
 2022-05-29 17:09:16 +02:00
Alexander Neumann
d2c5843c68
Merge pull request #3704 from MichaelEischer/compression-migrations 
Support migration to repository format with compression
 2022-05-29 15:52:21 +02:00
Alexander Neumann
78a21bbccf
Merge pull request #3752 from MichaelEischer/fix-dir-sync-errors 
local: Ignore additional errors for directory syncing
 2022-05-29 12:54:51 +02:00
greatroar
dde8e9e296 internal/restic: Custom ID.MarshalJSON 
This skips an allocation. internal/archiver benchmarks, Linux/amd64:

name                     old time/op    new time/op    delta
ArchiverSaveFileSmall-8    3.94ms ± 6%    3.91ms ± 6%    ~     (p=0.947 n=20+20)
ArchiverSaveFileLarge-8     304ms ± 3%     301ms ± 4%    ~     (p=0.265 n=18+18)

name                     old speed      new speed      delta
ArchiverSaveFileSmall-8  1.04MB/s ± 6%  1.05MB/s ± 6%    ~     (p=0.803 n=20+20)
ArchiverSaveFileLarge-8   142MB/s ± 3%   143MB/s ± 4%    ~     (p=0.421 n=18+19)

name                     old alloc/op   new alloc/op   delta
ArchiverSaveFileSmall-8    17.9MB ± 0%    17.9MB ± 0%  -0.01%  (p=0.000 n=19+19)
ArchiverSaveFileLarge-8     382MB ± 2%     382MB ± 1%    ~     (p=0.687 n=20+19)

name                     old allocs/op  new allocs/op  delta
ArchiverSaveFileSmall-8       540 ± 1%       528 ± 0%  -2.19%  (p=0.000 n=19+19)
ArchiverSaveFileLarge-8     1.93k ± 3%     1.79k ± 4%  -7.06%  (p=0.000 n=20+20)
 2022-05-27 12:26:37 +02:00
MichaelEischer
88a8701fb5
Merge pull request #3734 from lbausch/validate-patterns 
Validate exclude patterns
 2022-05-14 16:20:15 +02:00
MichaelEischer
b2a2e5f727
Merge pull request #3753 from greatroar/indexmap-alloc 
repository: Re-tune indexmap allocation strategy
 2022-05-14 15:44:08 +02:00
MichaelEischer
b52c631bd3
Merge pull request #3754 from greatroar/simplify-hashing 
hashing: Fix up comments
 2022-05-14 15:33:51 +02:00
Lorenz Bausch
36bd464e8c
Add tests for validating exclude patterns 2022-05-11 22:41:00 +02:00
greatroar
39a335e690 hashing: Fix up comments 2022-05-11 21:36:10 +02:00