TrueCloudLab/restic
16
0
Fork 2
Code Issues 1 Pull requests Projects Releases Packages Wiki Activity Actions
8554 commits 9 branches 49 tags 79 MiB
Commit graph

8554 commits

This branch
This branch
All branches
Author SHA1 Message Date
Michael Eischer
50ec408302 index: move to repository package 2024-05-25 13:13:03 +02:00
Michael Eischer
58bbb27b4c downgrade minio to keep go 1.19 support 2024-05-25 12:52:08 +02:00
Michael Eischer
c5260373ca update backoff/xxhash libraries 2024-05-25 12:37:14 +02:00
Michael Eischer
b2f7a878cd update backend libraries 
minio-go is kept at 7.0.67 as later versions require Go 1.21.
 2024-05-25 12:36:16 +02:00
Michael Eischer
f20bf5eeba update golang.org/x packages 2024-05-25 12:35:28 +02:00
Michael Eischer
8e5d7d719c cache: move to backend package 2024-05-24 23:04:06 +02:00
Michael Eischer
80132e71d8
Merge pull request #4808 from MichaelEischer/insecure-no-password 
Implement `--insecure-no-password` option.
 2024-05-24 22:58:25 +02:00
Michael Eischer
130506250f document insecure-no-password 2024-05-24 22:38:20 +02:00
Michael Eischer
1d2277b4c3 Add --insecure-no-password option 
This also includes two derived options `--from-insecure-no-password`
used for commands that require specifying a source repository. And
`--new-insecure-no-password` for the `key add` and `key passwd`
commands.

Specifying `--insecure-no-password` disabled the password prompt and
immediately uses an empty password. Passing a password via CLI option or
environment variable at the same time is an error.
 2024-05-24 22:38:20 +02:00
Michael Eischer
d4b0d21199 key add/passwd: deduplicate options setup and remove globals 
The current pattern of using a global options variable is problematic.
 2024-05-24 22:38:20 +02:00
Michael Eischer
55cb8d174a
Merge pull request #4354 from MichaelEischer/associated-data 
Reduce prune memory usage
 2024-05-24 22:26:22 +02:00
Michael Eischer
436afbff23 add changelog for memory efficient prune 2024-05-24 22:18:14 +02:00
Michael Eischer
3c7b7efdc9 repository: remove prune plan parts once they are no longer necessary 2024-05-24 22:18:14 +02:00
Michael Eischer
462b82a060 index: reduce size of compressed indexes 
use the same index size for compressed and uncompressed indexes.
Otherwise, decoding the index of a compressed repository requires
significantly more memory.
 2024-05-24 22:18:14 +02:00
Michael Eischer
77873f5a9d repository: let prune control data structure of usedBlobs set 2024-05-24 22:18:14 +02:00
Michael Eischer
2033c02b09 index: replace CountedBlobSet with AssociatedSet 2024-05-24 22:18:14 +02:00
Michael Eischer
93098e9265 prune: hide implementation details of counted blob set 2024-05-24 21:42:56 +02:00
Michael Eischer
ff4775a15f
Merge pull request #4812 from MichaelEischer/streaming-index-rewrite 
Resumable prune & memory-efficient index rewrite
 2024-05-24 21:41:30 +02:00
Michael Eischer
860b595a8b backend: increase watchdog test timeout for deflaking 2024-05-24 21:33:17 +02:00
Michael Eischer
f680a2331d add changelog for streaming index rewrite 2024-05-24 21:33:17 +02:00
Michael Eischer
027cc64737 repository: fix prune heuristic to allow resuming interrupted runs 
Pack files created by interrupted prune runs, appear to consist only of
duplicate blobs on the next run. This caused the previous heuristic to
ignore those pack files. Now, a duplicate blob in a specific pack file
is also selected if that pack file only contains duplicate blobs. This
allows prune to select the already rewritten pack files.
 2024-05-24 21:33:17 +02:00
Michael Eischer
e52033a8bd index: slightly reduce Rewrite concurrency 
The index operations are likely CPU-bounded. Thus, reduce the
concurrency accordingly.
 2024-05-24 21:33:17 +02:00
Michael Eischer
57d69aa640 index: cleanup SaveIndex method 2024-05-24 21:33:17 +02:00
Michael Eischer
2ca1c37721 index: additional tests for new index save methods 2024-05-24 21:33:17 +02:00
Michael Eischer
5f7b48e65f index: replace Save() method with Rewrite and SaveFallback 
Rewrite implements a streaming rewrite of the index that excludes the
given packs. For this it loads all index files from the repository and
only modifies those that require changes. This will reduce the index
churn when running prune. Rewrite does not require the in-memory index
and thus can drop it to significantly reduce the memory usage.

However, `prune --unsafe-recovery` cannot use this strategy and requires
a separate method to save the whole in-memory index. This is now handled
using SaveFallback.
 2024-05-24 21:33:17 +02:00
Michael Eischer
ad98fbf7dd restic: add IDSet.Clone() method 2024-05-24 21:33:17 +02:00
Michael Eischer
72482ce5bd index: misc cleanups 2024-05-24 21:33:17 +02:00
Michael Eischer
68fa0e0305 prune: no longer disable automatic index updates 
this allows prune to resume an interrupted prune run.
 2024-05-24 21:33:17 +02:00
Michael Eischer
9aa0c90fb2 index: remove supersedes field 
Using the field with its current semantics is nearly impossible to get
right. Remove it as it will be replaced anyways in repository format 3.
 2024-05-24 21:33:17 +02:00
Michael Eischer
76e6719f2e repository: make CreateIndexFromPacks method private 2024-05-24 21:33:17 +02:00
Michael Eischer
04ad9f0c0c repository: remove Packer and SavePacker from public interface 2024-05-24 21:33:17 +02:00
Michael Eischer
550d1eeac3 repository: remove SaveIndex from interface 
The method is now only indirectly accessible via Prune or RepairIndex.
 2024-05-24 21:33:17 +02:00
Michael Eischer
fb59e00614 index: rewrite MasterIndex load/save test to be independent of repository 2024-05-24 21:33:17 +02:00
Michael Eischer
447b486c20 index: deduplicate index loading of check and repository 2024-05-24 21:33:17 +02:00
Michael Eischer
6ca12c1b4a archiver: replace most uses of restic.Repository 2024-05-24 21:33:17 +02:00
Michael Eischer
864995271e repository: unwrap BlobHandle parameters of LookupBlob 
The method now uses the same parameters as LookupBlobSize.
 2024-05-24 21:33:17 +02:00
Michael Eischer
1266a4932f repository: fix parameter order of LookupBlobSize 
All methods should use blobType followed by ID.
 2024-05-24 21:33:17 +02:00
Michael Eischer
e848ad651a restic: name parameters in restic interface 2024-05-24 21:33:17 +02:00
Michael Eischer
0bb0720348 test cleanups 2024-05-24 21:33:17 +02:00
Michael Eischer
c01bcb1001 archiver: remove unused masterIndex from test 2024-05-24 21:33:17 +02:00
Michael Eischer
0aa5c53842 repository: replace HasBlob with LookupBlobSize 2024-05-24 21:33:17 +02:00
Michael Eischer
8f1e70cd9b repository: remove clearIndex and packSize from public interface 2024-05-24 21:33:17 +02:00
Michael Eischer
4df887406f repository: inline MasterIndex interface into Repository interface 2024-05-24 21:33:17 +02:00
Michael Eischer
3eeb6723cd
Merge pull request #4764 from MichaelEischer/safe-keep-tag 
Prevent unsafe uses of `forget --keep-tag`
 2024-05-24 20:51:20 +02:00
Michael Eischer
3cc6827f09 forget: add feature flag for safe --keep-tags behavior 2024-05-24 20:45:33 +02:00
Michael Eischer
7948912b31 document --unsafe-allow-remove-all option 2024-05-24 20:45:33 +02:00
Michael Eischer
3f46808898 add forget safety net changelogs 2024-05-24 20:45:33 +02:00
Michael Eischer
82ca0030b7 forget: test --unsafe-allow-remove-all and --keep-tags safety check 2024-05-24 20:45:33 +02:00
Michael Eischer
57f9739573 forget: Add --unsafe-allow-remove-all option 
To prevent accidentally wiping all snapshots from a repository, that
option can only be used if either a snapshot filter or a keep policy is
specified.

Essentially, the option allows `forget --tag something
--unsafe-allow-remove-all` calls to remove all snapshots with a specific
tag.
 2024-05-24 20:45:33 +02:00
Michael Eischer
5b7952e426 forget: return error if no policy was specified 2024-05-24 20:45:33 +02:00