Commit graph

7993 commits

Author SHA1 Message Date
Michael Eischer
b2f7a878cd update backend libraries
minio-go is kept at 7.0.67 as later versions require Go 1.21.
2024-05-25 12:36:16 +02:00
Michael Eischer
f20bf5eeba update golang.org/x packages 2024-05-25 12:35:28 +02:00
Michael Eischer
80132e71d8
Merge pull request #4808 from MichaelEischer/insecure-no-password
Implement `--insecure-no-password` option.
2024-05-24 22:58:25 +02:00
Michael Eischer
130506250f document insecure-no-password 2024-05-24 22:38:20 +02:00
Michael Eischer
1d2277b4c3 Add --insecure-no-password option
This also includes two derived options `--from-insecure-no-password`
used for commands that require specifying a source repository. And
`--new-insecure-no-password` for the `key add` and `key passwd`
commands.

Specifying `--insecure-no-password` disabled the password prompt and
immediately uses an empty password. Passing a password via CLI option or
environment variable at the same time is an error.
2024-05-24 22:38:20 +02:00
Michael Eischer
d4b0d21199 key add/passwd: deduplicate options setup and remove globals
The current pattern of using a global options variable is problematic.
2024-05-24 22:38:20 +02:00
Michael Eischer
55cb8d174a
Merge pull request #4354 from MichaelEischer/associated-data
Reduce prune memory usage
2024-05-24 22:26:22 +02:00
Michael Eischer
436afbff23 add changelog for memory efficient prune 2024-05-24 22:18:14 +02:00
Michael Eischer
3c7b7efdc9 repository: remove prune plan parts once they are no longer necessary 2024-05-24 22:18:14 +02:00
Michael Eischer
462b82a060 index: reduce size of compressed indexes
use the same index size for compressed and uncompressed indexes.
Otherwise, decoding the index of a compressed repository requires
significantly more memory.
2024-05-24 22:18:14 +02:00
Michael Eischer
77873f5a9d repository: let prune control data structure of usedBlobs set 2024-05-24 22:18:14 +02:00
Michael Eischer
2033c02b09 index: replace CountedBlobSet with AssociatedSet 2024-05-24 22:18:14 +02:00
Michael Eischer
93098e9265 prune: hide implementation details of counted blob set 2024-05-24 21:42:56 +02:00
Michael Eischer
ff4775a15f
Merge pull request #4812 from MichaelEischer/streaming-index-rewrite
Resumable prune & memory-efficient index rewrite
2024-05-24 21:41:30 +02:00
Michael Eischer
860b595a8b backend: increase watchdog test timeout for deflaking 2024-05-24 21:33:17 +02:00
Michael Eischer
f680a2331d add changelog for streaming index rewrite 2024-05-24 21:33:17 +02:00
Michael Eischer
027cc64737 repository: fix prune heuristic to allow resuming interrupted runs
Pack files created by interrupted prune runs, appear to consist only of
duplicate blobs on the next run. This caused the previous heuristic to
ignore those pack files. Now, a duplicate blob in a specific pack file
is also selected if that pack file only contains duplicate blobs. This
allows prune to select the already rewritten pack files.
2024-05-24 21:33:17 +02:00
Michael Eischer
e52033a8bd index: slightly reduce Rewrite concurrency
The index operations are likely CPU-bounded. Thus, reduce the
concurrency accordingly.
2024-05-24 21:33:17 +02:00
Michael Eischer
57d69aa640 index: cleanup SaveIndex method 2024-05-24 21:33:17 +02:00
Michael Eischer
2ca1c37721 index: additional tests for new index save methods 2024-05-24 21:33:17 +02:00
Michael Eischer
5f7b48e65f index: replace Save() method with Rewrite and SaveFallback
Rewrite implements a streaming rewrite of the index that excludes the
given packs. For this it loads all index files from the repository and
only modifies those that require changes. This will reduce the index
churn when running prune. Rewrite does not require the in-memory index
and thus can drop it to significantly reduce the memory usage.

However, `prune --unsafe-recovery` cannot use this strategy and requires
a separate method to save the whole in-memory index. This is now handled
using SaveFallback.
2024-05-24 21:33:17 +02:00
Michael Eischer
ad98fbf7dd restic: add IDSet.Clone() method 2024-05-24 21:33:17 +02:00
Michael Eischer
72482ce5bd index: misc cleanups 2024-05-24 21:33:17 +02:00
Michael Eischer
68fa0e0305 prune: no longer disable automatic index updates
this allows prune to resume an interrupted prune run.
2024-05-24 21:33:17 +02:00
Michael Eischer
9aa0c90fb2 index: remove supersedes field
Using the field with its current semantics is nearly impossible to get
right. Remove it as it will be replaced anyways in repository format 3.
2024-05-24 21:33:17 +02:00
Michael Eischer
76e6719f2e repository: make CreateIndexFromPacks method private 2024-05-24 21:33:17 +02:00
Michael Eischer
04ad9f0c0c repository: remove Packer and SavePacker from public interface 2024-05-24 21:33:17 +02:00
Michael Eischer
550d1eeac3 repository: remove SaveIndex from interface
The method is now only indirectly accessible via Prune or RepairIndex.
2024-05-24 21:33:17 +02:00
Michael Eischer
fb59e00614 index: rewrite MasterIndex load/save test to be independent of repository 2024-05-24 21:33:17 +02:00
Michael Eischer
447b486c20 index: deduplicate index loading of check and repository 2024-05-24 21:33:17 +02:00
Michael Eischer
6ca12c1b4a archiver: replace most uses of restic.Repository 2024-05-24 21:33:17 +02:00
Michael Eischer
864995271e repository: unwrap BlobHandle parameters of LookupBlob
The method now uses the same parameters as LookupBlobSize.
2024-05-24 21:33:17 +02:00
Michael Eischer
1266a4932f repository: fix parameter order of LookupBlobSize
All methods should use blobType followed by ID.
2024-05-24 21:33:17 +02:00
Michael Eischer
e848ad651a restic: name parameters in restic interface 2024-05-24 21:33:17 +02:00
Michael Eischer
0bb0720348 test cleanups 2024-05-24 21:33:17 +02:00
Michael Eischer
c01bcb1001 archiver: remove unused masterIndex from test 2024-05-24 21:33:17 +02:00
Michael Eischer
0aa5c53842 repository: replace HasBlob with LookupBlobSize 2024-05-24 21:33:17 +02:00
Michael Eischer
8f1e70cd9b repository: remove clearIndex and packSize from public interface 2024-05-24 21:33:17 +02:00
Michael Eischer
4df887406f repository: inline MasterIndex interface into Repository interface 2024-05-24 21:33:17 +02:00
Michael Eischer
3eeb6723cd
Merge pull request #4764 from MichaelEischer/safe-keep-tag
Prevent unsafe uses of `forget --keep-tag`
2024-05-24 20:51:20 +02:00
Michael Eischer
3cc6827f09 forget: add feature flag for safe --keep-tags behavior 2024-05-24 20:45:33 +02:00
Michael Eischer
7948912b31 document --unsafe-allow-remove-all option 2024-05-24 20:45:33 +02:00
Michael Eischer
3f46808898 add forget safety net changelogs 2024-05-24 20:45:33 +02:00
Michael Eischer
82ca0030b7 forget: test --unsafe-allow-remove-all and --keep-tags safety check 2024-05-24 20:45:33 +02:00
Michael Eischer
57f9739573 forget: Add --unsafe-allow-remove-all option
To prevent accidentally wiping all snapshots from a repository, that
option can only be used if either a snapshot filter or a keep policy is
specified.

Essentially, the option allows `forget --tag something
--unsafe-allow-remove-all` calls to remove all snapshots with a specific
tag.
2024-05-24 20:45:33 +02:00
Michael Eischer
5b7952e426 forget: return error if no policy was specified 2024-05-24 20:45:33 +02:00
Michael Eischer
c0e1f36830 forget: refuse deleting the last snapshot in a snapshot group
`--keep-tag invalid-tag` was previously able to wipe all snapshots in a
repository. As a user specified a `--keep-*` option this is likely
unintentional. This forbid deleting all snapshot if a `--keep-*` option
was specified to prevent data loss. (Not specifying such an option
currently also causes the command to abort)
2024-05-24 20:45:33 +02:00
Michael Eischer
d106ad6921 restic: regenerate snapshot keep policy golden test files 2024-05-24 20:45:33 +02:00
Michael Eischer
16ef4d515b
Merge pull request #4784 from MichaelEischer/rework-backend-retries
Rework backend retries
2024-05-24 20:29:54 +02:00
Michael Eischer
e4a48085ae backend/retry: feature flag new retry behavior 2024-05-24 20:24:02 +02:00