Commit graph

1848 commits

Author SHA1 Message Date
Michael Eischer
2c80cfa4a5 walker: fix missing leaveDir if directory is partially skipped 2024-01-27 13:17:33 +01:00
Michael Eischer
9ecbda059c walker: add callback to inform about leaving a directory 2024-01-27 13:17:32 +01:00
Michael Eischer
3424088274
Merge pull request #4644 from MichaelEischer/refactor-repair-packs
Refactor and test `repair packs`
2024-01-27 13:00:51 +01:00
Michael Eischer
724ec179e3
Merge pull request #4648 from MichaelEischer/repository-removekey
repository: Introduce RemoveKey function
2024-01-27 13:00:42 +01:00
Michael Eischer
f0e1ad2285 fix linter warning 2024-01-27 12:51:45 +01:00
Michael Eischer
fd579421dd repository: deduplicate test 2024-01-27 12:51:45 +01:00
Michael Eischer
42c9318b9c repair pack: add tests 2024-01-27 12:51:45 +01:00
Michael Eischer
764b0bacd6 repair pack: add support for truncated files 2024-01-27 12:51:45 +01:00
Michael Eischer
7c351bc53c repair pack: reenable auto index updates
The method is not available on the restic.Repository interface that is
used for testing. Drop the call as a small amount of additional index
writes is not a problem.
2024-01-27 12:51:45 +01:00
Michael Eischer
feeab84204 repair pack: extract the repair logic into the repository package
Currently, the cmd/restic package contains a significant amount of code
that modifies repository internals. This code should in the mid-term
move into the repository package.
2024-01-27 12:51:45 +01:00
Michael Eischer
cb50832d50 index: let MasterIndex.Save also delete obsolete indexes 2024-01-27 12:51:08 +01:00
Michael Eischer
bedff1ed6d split deleteFiles into UI and logic parts 2024-01-27 12:51:08 +01:00
Michael Eischer
c13bf0b607 repository: Introduce RemoveKey function
This replaces directly removing keys via the backend.
2024-01-27 12:42:58 +01:00
Michael Eischer
446167ae80
Merge pull request #4643 from MichaelEischer/remove-redundant-poly1305-mask
Remove redundant poly1305 key masking
2024-01-23 19:46:40 +01:00
Michael Eischer
1419baf67a
Merge pull request #4645 from MichaelEischer/improve-lock-checking
lock: checkForOtherLocks processes each lock at most once
2024-01-23 19:46:00 +01:00
Michael Eischer
6bdca13603 ui/termstatus: simplify cleaning up on termination
`writeStatus` also cleans no longer used status lines.
The old code actually cleaned one line too much. However, as that line
was never used it makes no difference.
2024-01-21 21:27:27 +01:00
Michael Eischer
f1f34eb3e5 lock: checkForOtherLocks processes each lock at most once
If a lock could not be loaded, then restic would check all lock files
again. These repeated checks are not useful as the status of a lock file
cannot change unless its ID changes too. Thus, skip already check lock
files on retries.
2024-01-20 22:40:12 +01:00
Michael Eischer
fee83e1c09 Remove redundant poly1305 key masking
The implementation in crypto/poly1305 already performs the exact same
masking.
2024-01-20 12:36:59 +01:00
Michael Eischer
a763a5c67d s3: minor cleanups for archive storage class handling 2024-01-20 11:25:28 +01:00
Vladislav Belous
8ca58b487c S3: do not set storage class for metadata when using archive storage 2024-01-20 11:04:15 +01:00
Michael Eischer
2c310a526e repository: Replace StreamPack function with LoadBlobsFromPack method
LoadBlobsFromPack is now part of the repository struct. This ensures
that users of that method don't have to deal will internals of the
repository implementation.

The filerestorer tests now also contain far fewer pack file
implementation details.
2024-01-19 21:40:43 +01:00
Michael Eischer
6b7b5c89e9 repository: prepare StreamPack refactor 2024-01-19 21:40:43 +01:00
Michael Eischer
22d0c3f8dc check: Use PackBlobIterator instead of StreamPack
To only stream the content of a pack file once, check used StreamPack
with a custom pack load function. This combination was always brittle
and complicates using StreamPack everywhere else. Now that StreamPack
internally uses PackBlobIterator use that primitive instead, which is a
much better fit for what the check command requires.
2024-01-19 21:40:36 +01:00
Michael Eischer
fb422497af repository: split StreamPack implementation
Move the actual decoding of the pack data into a separate iterator.
2024-01-19 21:39:55 +01:00
Michael Eischer
54c5c72e5a
Merge pull request #4616 from MichaelEischer/fix-rest-connection-close
rest: fix and cleanup closing of http response body
2024-01-19 21:31:35 +01:00
Michael Eischer
bd883caae1 CI: enable bodyclose linter 2024-01-19 21:17:18 +01:00
Michael Eischer
b1a8fd1d03 rest: fix and cleanup closing of http response body
If client.Do returns an error, then there's no body that has to be
closed. For requests for which we are not interested in the response
body, immediately drain and close the body to make sure it isn't
forgotten later on.

This change in particular adds the missing `Close()` call for the
`List()` command.
2024-01-19 21:17:17 +01:00
Michael Eischer
fdcbb53017 walker: test skipping for root node 2024-01-19 21:16:06 +01:00
Michael Eischer
0b39940fdb walker: Remove ignoreTrees functionality
It was only used in two places:
- stats: apparently as a minor performance optimization, which is
  unlikely to be important
- find: filtered directories would be ignored. However, this
  optimization missed that it is possible that two directories have the
  exact same content. Such directories would be incorrectly ignored too.
  Example:
```
mkdir test test/a test/b
restic backup test
restic find latest test/b
-> incorrectly does not return anything
```

Thus, remove the functionality as it's apparently too complex to use
correctly.
2024-01-19 21:16:06 +01:00
Michael Eischer
7b2de84763
Merge pull request #4618 from MichaelEischer/workaround-rclone-list-errors
rclone: Workaround for incorrect "not found" errors while listing files
2024-01-09 18:28:31 +01:00
Michael Eischer
c31e9418ba
Merge pull request #4626 from MichaelEischer/reliable-large-restores
Improve reliability of large restores
2024-01-09 18:23:09 +01:00
Michael Eischer
2e8de9edfd rclone: Workaround for incorrect "not found" errors while listing files
rclone returns a "not found" error if an internal error occurs while
listing a folder. Ignoring this error lets restic erroneously think that
there are no files, which can cause `prune` to wipe the whole
repository.
2024-01-09 18:20:16 +01:00
Michael Eischer
e78be75d1e restore: separately restore blobs that are frequently referenced
Writing these blobs to their files can take a long time and consequently
cause the backend connection to time out. Avoid that by retrieving these
blobs separately.
2024-01-08 21:00:13 +01:00
Michael Eischer
2267910418 restore: split error reporting from downloadPack 2024-01-08 20:57:00 +01:00
Michael Eischer
00d18b7a88 restore: cleanup downloadPack 2024-01-08 20:53:08 +01:00
Michael Eischer
9328f34d43 restore: split downloadPack into smaller methods 2024-01-08 20:52:36 +01:00
Michael Eischer
77434c6e2b
Merge pull request #4474 from ekristen/aws-assume-role
Allow AWS Assume Role
2024-01-08 19:07:17 +00:00
Michael Eischer
e4a7eb09ef
Merge pull request #4624 from MichaelEischer/better-restorer-error-reporting
Improver restorer error reporting
2024-01-07 11:20:29 +01:00
Michael Eischer
f8b4e932ef
Merge pull request #4620 from MichaelEischer/improve-irregular-file-handling
Improve irregular file handling
2024-01-07 11:12:07 +01:00
Michael Eischer
dac3508170 restore: only report errors for blobs that actually failed to load
Previously, errors would be reported for all blobs of a packfile that
failed to stream. Now, only the not yet processed blobs are reported.
2024-01-07 10:54:56 +01:00
Michael Eischer
77b1c52673 repository: test that StreamPack only delivers blobs once 2024-01-07 10:54:53 +01:00
Michael Eischer
fe5c337ca2 repository: StreamPack delivers blobs at most once
If an error occurred while streaming a pack file, this could result in
passing some of the blobs multiple times to the callback function. This
significantly complicates using StreamPack correctly and is unnecessary.
Retries do not change the content of a blob and thus only deliver the
same result over and over again.
2024-01-07 10:54:49 +01:00
Michael Eischer
0360e540af
Merge pull request #4622 from MichaelEischer/fix-outdated-windows-import
termstatus: update import path of golang.org/x/term
2024-01-06 23:24:16 +01:00
Michael Eischer
e6dfefba13 termstatus: update import path of golang.org/x/term 2024-01-06 21:59:26 +01:00
Michael Eischer
02bc73f5eb s3: minor code cleanups 2024-01-06 21:44:53 +01:00
Michael Eischer
20cf4777cb s3: check for EnvAWS credentials before Static credentials
EnvAWS considers more environment variables, including AWS_SESSION_TOKEN
and thus should be checked first.
2024-01-06 21:43:47 +01:00
Erik Kristensen
5ffb536aae feat: support AWS assume role 2024-01-06 21:19:58 +01:00
Michael Eischer
33b7c84a7a deduplicate string formatting of snapshot metadata
This removes the spurious ")" bracket at the end and normalizes the
metadata format used by the `ls` command.
2024-01-06 20:20:51 +01:00
Michael Eischer
51419c51d3 archiver: Add filepath to error message if it is not included yet 2024-01-06 19:08:24 +01:00
Michael Eischer
6b79834cc8 archiver: improve error message for irregular files
Since Go 1.21, most reparse points are considered as irregular files.
Depending on the underlying driver these can exhibit nearly arbitrary
behavior. When encountering such a file, restic returned an
indecipherable error message: `error: invalid node type ""`.

Add the filepath to the error message and state that the file type is
not supported.
2024-01-06 19:03:11 +01:00