Commit graph

2252 commits

Author SHA1 Message Date
Aleksey Kravchenko
6939e0353f [#1] Add frostfs backend
Signed-off-by: Aleksey Kravchenko <al.kravchenko@yadro.com>
2024-12-13 16:47:25 +03:00
Michael Eischer
7bfe3d99ae fs: fallback to low privilege security descriptors on access denied 2024-11-03 21:45:52 +01:00
Michael Eischer
75f317eaf1 sftp: check for broken connection in Load/List operation 2024-11-03 21:44:45 +01:00
Michael Eischer
8aebea7ba2 fuse: test that the same fs.Node is used for the same file 2024-11-03 21:42:20 +01:00
Michael Eischer
0e9716a6e6 fuse: forget fs.Node instances on request by the kernel
Forget fs.Node instances once the kernel frees the corresponding nodeId.
This ensures that restic does not run out of memory on large snapshots.
2024-11-03 21:42:19 +01:00
Michael Eischer
de4f8b344e fuse: add missing type assertion for optional interfaces 2024-11-03 21:41:22 +01:00
Michael Eischer
75ec7d3269 fuse: cache fs.Node instances
A particular node should always be represented by a single instance.
This is necessary to allow the fuse library to assign a stable nodeId to
a node. macOS Sonoma trips over the previous, unstable behavior when
using fuse-t.
2024-11-03 21:41:13 +01:00
Michael Eischer
0aee70b496 restic: test path handling of volume shadow copy root path 2024-10-31 22:07:15 +01:00
Michael Eischer
e38f6794cd restic: fix error in fillGenericAttributes for vss volumes
Extended attributes and security descriptors apparently cannot be
retrieved from a vss volume. Fix the volume check to correctly detect
vss volumes and just completely disable extended attributes for volumes.
2024-10-31 22:07:15 +01:00
Michael Eischer
f77e67086c fs: add correct vss support to fixpath
Paths that only contain the volume shadow copy snapshot name require
special treatment. These paths must end with a slash for regular file
operations to work.
2024-10-31 22:07:15 +01:00
Michael Eischer
a7b13bd603 fs: remove file.Name() from interface
The only user was archiver.fileSaver.
2024-10-18 22:29:03 +02:00
Michael Eischer
0c711f5605 archiver: use correct filepath in fileSaver for vss
When using the VSS FS, then `f.Name()` contained the filename in the
snapshot. This caused a double mapping when calling NodeFromFileInfo.
2024-10-18 22:29:03 +02:00
Michael Eischer
4df2e33568 archiver: properly create node for vss backups
Previously, NodeFromFileInfo used the original file path to create the
node, which also meant that extended metadata was read from there
instead of within the vss snapshot.

This change is a temporary solution for restic 0.17.2 and will be
replaced with a clean fix in restic 0.18.0.
2024-10-18 22:26:18 +02:00
Connor Findlay
9553d873ff backend/azure: Add tests for both token types
Add two new test cases, TestBackendAzureAccountToken and
TestBackendAzureContainerToken, that ensure that the authorization using
both types of token works.

This introduces two new environment variables,
RESTIC_TEST_AZURE_ACCOUNT_SAS and RESTIC_TEST_AZURE_CONTAINER_SAS, that
contain the tokens to use when testing restic. If an environment
variable is missing, the related test is skipped.
2024-10-18 21:59:03 +02:00
Connor Findlay
d6e76a22a8 backend/azure: Handle Container SAS/SAT
Ignore AuthorizationFailure caused by using a container level SAS/SAT
token when calling GetProperties during the Create() call. This is because the
GetProperties call expects an Account Level token, and the container
level token simply lacks the appropriate permissions. Supressing the
Authorization Failure is OK, because if the token is actually invalid,
this is caught elsewhere when we try to actually use the token to do
work.
2024-10-18 21:59:03 +02:00
Michael Eischer
fc1fc00aa4 backup: exclude irregular files from backup
restic cannot backup irregular files as those don't behave like normal
files. Thus skip them with an error.
2024-10-18 21:56:41 +02:00
greatroar
3c82fe6ef5 fs: Include filename in mknod errors 2024-10-18 21:53:15 +02:00
Michael Eischer
0df2fa8135 fs: retry preallocate on Linux if interrupted by signal 2024-10-18 21:47:59 +02:00
Damien Clark
8c1d6a50c1 cache: fix race condition in cache cleanup
Fix multiple restic processes executing concurrently and racing to remove obsolete snapshots.

Co-authored-by: Michael Eischer <michael.eischer@fau.de>
2024-10-18 21:47:59 +02:00
Michael Eischer
ba71141f0a backup: support specifying volume instead of path on Windows
"C:" (volume name) versus "C:\" (path)
2024-08-31 16:50:06 +02:00
Michael Eischer
1f4c9d2806 cache: correctly ignore files whose filename is no ID
this can for example be the case for temporary files created by the
backend implementation.
2024-08-31 16:50:06 +02:00
Michael Eischer
ac5bc7c2f9 fs: fix error handling for retried get/set of security descriptor
The retry code path did not filter `ERROR_NOT_SUPPORTED`. Just call the
original function a second time to correctly follow the low privilege
code path.
2024-08-30 12:46:07 +02:00
Michael Eischer
3e4c1ea196 fs: fix race condition in get/set security descriptor
Calling `Load()` twice for an atomic variable can return different
values each time. This resulted in trying to read the security
descriptor with high privileges, but then not entering the code path to
switch to low privileges when another thread has already done so
concurrently.
2024-08-30 12:46:07 +02:00
Michael Eischer
8828c76f92 rest: improve handling of HTTP2 goaway
The HTTP client can only retry HTTP2 requests after receiving a GOAWAY
response if it can rewind the body. As we use a custom data type,
explicitly provide an implementation of `GetBody`.
2024-08-30 12:46:07 +02:00
Michael Eischer
64d628bd75 make timeout for slow requests configurable 2024-08-30 12:45:20 +02:00
Michael Eischer
6eece31dc3 lock: introduce short delay between failed locking retries
Failed locking attempts were immediately retried up to three times
without any delay between the retries. If a lock file is not found while
checking for other locks, with the reworked backend retries there is no
delay between those retries. This is a problem if a backend requires a
few seconds to reflect file deletions in the file listings. To work
around this problem, introduce a short exponentially increasing delay
between the retries. The number of retries is now increased to 4. This
results in delays of 5, 10 and 20 seconds between the retries.
2024-08-30 12:45:20 +02:00
Michael Eischer
8206cd19c8 backend/retry: don't trip circuit breaker if context is canceled
When the context used for a load operation is canceled, then the result
is always an error independent of whether the file could be retrieved
from the backend. Do not false positively trip the circuit breaker in
this case.

The old behavior was problematic when trying to lock a repository. When
`Lock.checkForOtherLocks` listed multiple lock files in parallel and one
of them fails to load, then all other loads were canceled. This
cancelation was remembered by the circuit breaker, such that locking
retries would fail.
2024-08-30 12:45:20 +02:00
Michael Eischer
cb16add8c8 restic: cleanup redundant code in test case 2024-08-30 12:45:20 +02:00
Michael Eischer
bc1aecfb15 restore: test timestamps for macOS resource forks are restored correctly 2024-08-30 12:45:20 +02:00
Michael Eischer
61aaddac28 restic: restore timestamps after extended attributes
restoring the xattr containing resource forks on macOS apparently
modifies the file modification timestamps. Thus, restore the timestamp
after xattrs.
2024-08-30 12:45:20 +02:00
Srigovind Nayak
a8032c932c cache: remove redundant index file cleanup
addressing code review comments
2024-08-30 12:43:13 +02:00
Srigovind Nayak
8a7ae17d4d Revert "repository: removed redundant prepareCache method from Repository"
This reverts commit 720609f8ba.
2024-08-30 12:43:13 +02:00
Srigovind Nayak
0ca9355bc0 cache: add test for the automated cache clear to cache backend 2024-08-30 12:43:13 +02:00
Srigovind Nayak
1e68fbca90 repository: removed redundant prepareCache method from Repository
* remove the prepareCache method from the Repository
* changed the signature of the SetIndex function to no longer return an error
2024-08-30 12:43:13 +02:00
Srigovind Nayak
0cf1737289 cache: check for context cancellation before clearing cache 2024-08-30 12:43:13 +02:00
Srigovind Nayak
fac1d9fea1 cache: backend add List method and a cache clear functionality
* removes files which are no longer in the repository, including index files, snapshot files and pack files from the cache.

cache: fix ids set initialisation with NewIDSet()
2024-08-30 12:43:13 +02:00
Michael Eischer
61e1f4a916 backend: return correct error on upload/request timeout 2024-08-30 12:37:10 +02:00
aneesh-n
7642e05eed Add test cases and handle volume GUID paths
Gracefully handle errors while checking for EA and add debug logs.
2024-08-30 12:37:10 +02:00
aneesh-n
51fad2eecb Gracefully handle invalid prepared volume names 2024-08-30 12:37:10 +02:00
aneesh-n
8861421cd6 Fix extended attributes handling for VSS snapshots 2024-08-30 12:37:10 +02:00
Andreas Deininger
c83b529c47 Fix typos 2024-08-30 12:37:10 +02:00
Shivashis Padhi
d15e693045 restore: Add progress bar to 'restore --verify' 2024-08-30 12:37:10 +02:00
Michael Eischer
283225f15f
Merge pull request #4994 from MichaelEischer/less-flaky-watchdog-test
backend: tweak timeouts to make watchdog timeout test less flaky
2024-08-10 19:35:38 +02:00
Michael Eischer
86390b453d
Merge pull request #4980 from zmanda/unsupported-ea-handling
Skip ExtendedAttribute processing in Windows for volumes that do not support EA
2024-08-10 17:11:12 +00:00
Michael Eischer
fa35e72214 backend: tweak timeouts to make watchdog timeout test less flaky 2024-08-10 19:08:03 +02:00
Michael Eischer
05571286b2
Merge pull request #4946 from mikix/json-errors
Improve error handling in --json mode
2024-08-10 17:04:56 +00:00
aneesh-n
18e9d71d7a
Fix review comments 2024-08-10 10:38:04 -06:00
Michael Eischer
a164789321
Merge pull request #4977 from MichaelEischer/warn-on-incomplete-metadata
backup: warn but store store item if extended metadata is incomplete
2024-08-10 17:59:25 +02:00
aneesh-n
71632a8197
Handle extended length paths 2024-08-05 16:03:43 -06:00
aneesh-n
85639f5159
Add handling for relative paths, vss paths, UNC paths 2024-08-04 13:19:13 -06:00