Connor Findlay
2e704c69ac
backend/azure: Handle Container SAS/SAT
...
Ignore AuthorizationFailure caused by using a container level SAS/SAT
token when calling GetProperties during the Create() call. This is because the
GetProperties call expects an Account Level token, and the container
level token simply lacks the appropriate permissions. Supressing the
Authorization Failure is OK, because if the token is actually invalid,
this is caught elsewhere when we try to actually use the token to do
work.
2024-10-17 20:38:03 +02:00
Michael Eischer
bcd5ac34bb
Merge pull request #5060 from MichaelEischer/proper-nodefromfileinfo
...
fs: move NodeFromFileInfo into FS interface
2024-10-16 21:34:37 +02:00
Michael Eischer
618f306f13
Merge pull request #5054 from phillipp/dump-compress-zip
...
dump: add --compress flag to compress archives
2024-10-16 19:17:47 +00:00
Michael Eischer
75711446e1
fs: move NodeFromFileInfo into FS interface
2024-10-16 21:17:21 +02:00
Michael Eischer
e29d38f8bf
dump/zip: test that files are compressed
2024-10-16 21:11:24 +02:00
Michael Eischer
da3c02405b
dump/zip: only compress regular files
2024-10-16 21:09:05 +02:00
Michael Eischer
f44b7cdf8c
backup: exclude irregular files from backup
...
restic cannot backup irregular files as those don't behave like normal
files. Thus skip them with an error.
2024-10-16 20:54:08 +02:00
Michael Eischer
e91a456656
Merge pull request #5061 from MichaelEischer/fix-timestamp-restore-windows
...
fs: fix restoring timestamps on older Windows versions for long paths
2024-10-16 20:47:17 +02:00
Michael Eischer
e21496f217
Merge pull request #5074 from greatroar/dump
...
dump: Simplify writeNode and use fewer goroutines
2024-10-16 18:33:35 +00:00
Michael Eischer
0c0d8b8cfd
Merge pull request #5083 from greatroar/errors
...
Some error handling patches
2024-10-16 18:22:49 +00:00
greatroar
2b609d3e77
errors, fs: Replace CombineErrors with stdlib Join
...
This does not produce exactly the same messages, as it inserts newlines
instead of "; ". But given how long our error messages can be, that
might be a good thing.
2024-10-05 10:56:40 +02:00
greatroar
19653f9e06
fs: Simplify NodeCreateAt
2024-10-05 10:56:39 +02:00
greatroar
e10e2bb50f
fs: Include filename in mknod errors
2024-10-05 10:56:39 +02:00
greatroar
b5c28a7ba2
internal/restic: Use IDSet.Clone + use maps package
...
One place where IDSet.Clone is useful was reinventing it, using a
conversion to list, a sort, and a conversion back to map.
Also, use the stdlib "maps" package to implement as much of IDSet as
possible. This requires changing one caller, which assumed that cloning
nil would return a non-nil IDSet.
2024-10-03 21:14:29 +02:00
greatroar
8c7a6daa47
dump: Simplify writeNode and use fewer goroutines
...
This changes Dumper.writeNode to spawn loader goroutines as needed
instead of as a pool. The code is shorter, fewer goroutines are spawned
for small files, and crash dumps (also for unrelated errors) should be
smaller.
2024-09-30 17:24:05 +02:00
Phillipp Röll
1a7fafc7eb
dump: compress zip archives
2024-09-15 21:04:54 +02:00
Michael Eischer
4469fe1575
fs: fix restoring timestamps on Windows for long paths
2024-09-15 18:28:11 +02:00
Phillipp Röll
bad6c54a33
dump: add --compress-zip flag to compress zip archives
2024-09-15 14:25:02 +02:00
Michael Eischer
efec1a5e96
Merge pull request #5045 from MichaelEischer/fix-preallocate-eintr
...
Linux: retry preallocate if interrutped by signal
2024-09-14 19:17:51 +02:00
Michael Eischer
5ce6ca2219
fuse: test that the same fs.Node is used for the same file
2024-09-14 18:11:44 +02:00
Michael Eischer
51173c5003
fuse: forget fs.Node instances on request by the kernel
...
Forget fs.Node instances once the kernel frees the corresponding nodeId.
This ensures that restic does not run out of memory on large snapshots.
2024-09-14 18:11:44 +02:00
Michael Eischer
e9940f39dc
fuse: add missing type assertion for optional interfaces
2024-09-14 18:11:44 +02:00
Michael Eischer
6ec2b62ec5
fuse: cache fs.Node instances
...
A particular node should always be represented by a single instance.
This is necessary to allow the fuse library to assign a stable nodeId to
a node. macOS Sonoma trips over the previous, unstable behavior when
using fuse-t.
2024-09-14 18:11:44 +02:00
Damien Clark
4795143d6d
cache: fix race condition in cache cleanup
...
Fix multiple restic processes executing concurrently and racing to remove obsolete snapshots.
Co-authored-by: Michael Eischer <michael.eischer@fau.de>
2024-09-14 18:07:46 +02:00
Michael Eischer
34fe73ea42
fs: retry preallocate on Linux if interrupted by signal
2024-09-07 16:39:40 +02:00
Srigovind Nayak
b69c6408a6
forget: make oldest snapshot marker more strict
...
Now, a snapshot is only marked as oldest if it's the last in the list AND its values matches the last seen value for that bucket.
Also, updated the corresponding golden files for the tests.
2024-09-07 15:07:23 +05:30
Srigovind Nayak
d656a50852
forget: update tests to reflect specific reasons for keeping oldest snapshots in a group
2024-09-07 15:07:23 +05:30
Srigovind Nayak
87f30bc787
forget: indicate why the oldest snapshot in a group is kept
...
When the oldest snapshot in the
list is retained, the reason is now prefixed with "oldest" to clearly
indicate why it's being kept.
2024-09-07 15:07:23 +05:30
Michael Eischer
9a6059eb71
Merge pull request #5032 from dropbigfish/master
...
chore: fix some function name comments
2024-09-01 21:52:26 +00:00
dropbigfish
6f9513d88c
chore: fix some function names
...
Signed-off-by: dropbigfish <fillfish@foxmail.com>
2024-09-01 00:54:39 +08:00
Michael Eischer
b91ef3f1ff
fs: remove dead code
2024-08-31 18:40:36 +02:00
Michael Eischer
e2bce1b9ee
fs: move WindowsAttributes definition back to restic package
2024-08-31 18:40:36 +02:00
Michael Eischer
ebdd946ac1
fs: unexport nodeRestoreTimestamps
2024-08-31 18:40:36 +02:00
Michael Eischer
2aa1e2615b
fs: fix comments
2024-08-31 18:40:36 +02:00
Michael Eischer
6c16733dfd
fs: remove unused methods from File interface
2024-08-31 18:40:36 +02:00
Michael Eischer
f0329bb4e6
fs: replace statT with ExtendedFileInfo
2024-08-31 18:40:36 +02:00
Michael Eischer
6d3a5260d3
fs: unexport a several windows functions
2024-08-31 18:40:36 +02:00
Michael Eischer
cf051e777a
fs: remove Readdir method from File interface
2024-08-31 18:20:41 +02:00
Michael Eischer
cc7f99125a
minimize usage of internal/fs in tests
2024-08-31 18:20:41 +02:00
Michael Eischer
65a7157383
mount: use os instead of fs package
2024-08-31 18:20:41 +02:00
Michael Eischer
24f4e780f1
backend: consistently use os package for filesystem access
...
The go std library should be good enough to manage the files in the
backend and cache folders.
2024-08-31 18:20:40 +02:00
Michael Eischer
ca1e5e10b6
add proper constants for node type
2024-08-31 18:20:01 +02:00
Michael Eischer
7bb92dc7bd
archiver: use ExtendedStat from FS interface
...
With this change, NodeFromFileInfo is the last function that bypasses
the FS interface in the archiver.
2024-08-31 18:05:09 +02:00
Michael Eischer
e79dca644e
fs: unexport DeviceID
2024-08-31 18:04:53 +02:00
Michael Eischer
70fbad6623
archiver: minimize imports
2024-08-31 18:04:37 +02:00
Michael Eischer
6fd5d5f2d5
archiver: move helper functions to combine rejects
2024-08-31 18:04:22 +02:00
Michael Eischer
f1585af0f2
move include/exclude options to filter package
2024-08-31 18:04:07 +02:00
Michael Eischer
41c031a19e
backup: move RejectFuncs to archiver package
2024-08-31 18:03:35 +02:00
Michael Eischer
f9dbcd2531
backup: convert reject funcs to use FS interface
...
Depending on parameters the paths in a snapshot do not directly
correspond to real paths on the filesystem. Therefore, reject funcs must
use the FS interface to work correctly.
2024-08-31 18:03:02 +02:00
Michael Eischer
c6fae0320e
archiver: hide implementation details
2024-08-31 17:52:45 +02:00
Michael Eischer
507842b614
fs: remove Open method from FS interface
2024-08-31 17:37:25 +02:00
Michael Eischer
263709da8c
fs: unexport isListxattrPermissionError
2024-08-31 17:37:25 +02:00
Michael Eischer
80ed863aab
repository: remove redundant cleanup code
...
The temp files used by the packer manager are either delete after
creation (unix) or marked as delete on close (windows). Thus, no
explicit cleanup is necessary.
2024-08-31 17:37:25 +02:00
Michael Eischer
0ddb4441d7
fs: clean up helper functions
2024-08-31 17:37:25 +02:00
Michael Eischer
fc549c9462
cleanup imports
2024-08-31 17:37:25 +02:00
Michael Eischer
b9b32e5647
restic: extract Node filesystem code to fs package
2024-08-31 17:37:25 +02:00
Michael Eischer
a2e54eac64
restic: simplify nodeCreateFileAt
...
The code to write the file content is never used.
2024-08-31 17:37:25 +02:00
Michael Eischer
5644079707
restic: prepare extraction of fs code from Node
2024-08-31 17:37:25 +02:00
Michael Eischer
97f696b937
backend: remove dead code
2024-08-31 17:25:24 +02:00
Michael Eischer
af989aab4e
backend/layout: unexport fields and simplify rest layout
2024-08-31 17:25:24 +02:00
Michael Eischer
6024597028
drop support for s3legacy layout
2024-08-31 17:25:24 +02:00
Michael Eischer
943b6ccfba
index: remove support for legacy index format
2024-08-31 17:12:43 +02:00
Michael Eischer
a5533344f9
Merge pull request #5028 from MichaelEischer/windows-allow-specifying-volumes
...
backup: support specifying volume instead of path on Windows
2024-08-31 16:43:20 +02:00
Michael Eischer
ddf35a60ad
Merge pull request #5026 from MichaelEischer/fix-handling-invalid-filenames
...
cache: Fix handling of invalid filenames
2024-08-31 16:42:13 +02:00
Michael Eischer
4fcedb4bae
backup: support specifying volume instead of path on Windows
...
"C:" (volume name) versus "C:\" (path)
2024-08-30 11:35:43 +02:00
Michael Eischer
a0f2dfbc19
Merge pull request #5019 from MichaelEischer/fix-windows-sd-race
...
backup: Fix spurious "A Required Privilege Is Not Held by the Client" error
2024-08-29 16:59:06 +02:00
Michael Eischer
0aadfe32bb
Merge pull request #5018 from MichaelEischer/rest-retry-http2-goaway
...
rest: improve handling of HTTP2 goaway
2024-08-29 16:58:04 +02:00
Michael Eischer
7bbf75237d
Merge pull request #5014 from MichaelEischer/configurable-slow-request-timeout
...
Make timeout for slow requests configurable
2024-08-29 16:52:24 +02:00
Michael Eischer
8eff4e0e5c
cache: correctly ignore files whose filename is no ID
...
this can for example be the case for temporary files created by the
backend implementation.
2024-08-29 16:32:15 +02:00
Michael Eischer
9c70794886
fs: fix error handling for retried get/set of security descriptor
...
The retry code path did not filter `ERROR_NOT_SUPPORTED`. Just call the
original function a second time to correctly follow the low privilege
code path.
2024-08-26 19:36:43 +02:00
Michael Eischer
6fbfccc2d3
fs: fix race condition in get/set security descriptor
...
Calling `Load()` twice for an atomic variable can return different
values each time. This resulted in trying to read the security
descriptor with high privileges, but then not entering the code path to
switch to low privileges when another thread has already done so
concurrently.
2024-08-26 19:31:21 +02:00
Michael Eischer
2296fdf668
lock: introduce short delay between failed locking retries
...
Failed locking attempts were immediately retried up to three times
without any delay between the retries. If a lock file is not found while
checking for other locks, with the reworked backend retries there is no
delay between those retries. This is a problem if a backend requires a
few seconds to reflect file deletions in the file listings. To work
around this problem, introduce a short exponentially increasing delay
between the retries. The number of retries is now increased to 4. This
results in delays of 5, 10 and 20 seconds between the retries.
2024-08-26 16:31:42 +02:00
Michael Eischer
89d216ca76
Merge pull request #5011 from MichaelEischer/fix-canceled-retry
...
backend/retry: don't trip circuit breaker if context is canceled
2024-08-26 16:30:03 +02:00
Michael Eischer
e24dd5a162
backend/retry: don't trip circuit breaker if context is canceled
...
When the context used for a load operation is canceled, then the result
is always an error independent of whether the file could be retrieved
from the backend. Do not false positively trip the circuit breaker in
this case.
The old behavior was problematic when trying to lock a repository. When
`Lock.checkForOtherLocks` listed multiple lock files in parallel and one
of them fails to load, then all other loads were canceled. This
cancelation was remembered by the circuit breaker, such that locking
retries would fail.
2024-08-26 16:22:21 +02:00
Michael Eischer
36c4475ad9
rest: improve handling of HTTP2 goaway
...
The HTTP client can only retry HTTP2 requests after receiving a GOAWAY
response if it can rewind the body. As we use a custom data type,
explicitly provide an implementation of `GetBody`.
2024-08-26 15:44:17 +02:00
Michael Eischer
b8f409723d
make timeout for slow requests configurable
2024-08-26 14:14:43 +02:00
Michael Eischer
311b27ced8
restic: cleanup redundant code in test case
2024-08-25 23:18:55 +02:00
Michael Eischer
43b36ad2b0
restore: test timestamps for macOS resource forks are restored correctly
2024-08-25 23:18:55 +02:00
Michael Eischer
2e55209b34
restic: restore timestamps after extended attributes
...
restoring the xattr containing resource forks on macOS apparently
modifies the file modification timestamps. Thus, restore the timestamp
after xattrs.
2024-08-25 23:18:55 +02:00
Michael Eischer
5cca6e66be
Merge pull request #4981 from konidev20/fix-gh-4934-cleanup-removed-snaphots-from-cache
...
cache: clear snapshot files from cache during load index
2024-08-16 19:04:59 +00:00
Srigovind Nayak
88174cd0a4
cache: remove redundant index file cleanup
...
addressing code review comments
2024-08-17 00:21:49 +05:30
Srigovind Nayak
b7d014b685
Revert "repository: removed redundant prepareCache method from Repository"
...
This reverts commit 720609f8ba
2024-08-17 00:18:13 +05:30
Michael Eischer
74d3f92cc7
Merge pull request #4993 from MichaelEischer/fix-timeout-error
...
backend: return correct error on upload/request timeout
2024-08-15 22:07:37 +02:00
Michael Eischer
80f24584a5
Merge pull request #4998 from zmanda/ea_vss_fix
...
Fix extended attributes handling for VSS snapshots
2024-08-15 19:51:35 +00:00
Michael Eischer
8e00158c34
Merge pull request #5000 from deining/fix-typo
...
Fix typos
2024-08-15 19:42:14 +00:00
Michael Eischer
36b5580c1c
Merge pull request #4989 from plant99/progress-bar-for-restore-verify
...
restore: Add progress bar to 'restore --verify'
2024-08-15 19:34:05 +00:00
aneesh-n
19f487750e
Add test cases and handle volume GUID paths
...
Gracefully handle errors while checking for EA and add debug logs.
2024-08-11 19:25:58 -06:00
Shivashis Padhi
f1407afd1f
restore: Add progress bar to 'restore --verify'
2024-08-11 22:25:21 +02:00
Andreas Deininger
4401265e36
Fix typos
2024-08-11 21:38:15 +02:00
Srigovind Nayak
5fd984ba6f
cache: add test for the automated cache clear to cache backend
2024-08-11 23:41:07 +05:30
Srigovind Nayak
720609f8ba
repository: removed redundant prepareCache method from Repository
...
* remove the prepareCache method from the Repository
* changed the signature of the SetIndex function to no longer return an error
2024-08-11 23:41:07 +05:30
Srigovind Nayak
a23e7bfb82
cache: check for context cancellation before clearing cache
2024-08-11 23:41:07 +05:30
Srigovind Nayak
f66624f5bf
cache: backend add List method and a cache clear functionality
...
* removes files which are no longer in the repository, including index files, snapshot files and pack files from the cache.
cache: fix ids set initialisation with NewIDSet()
2024-08-11 23:40:52 +05:30
aneesh-n
849c441455
Gracefully handle invalid prepared volume names
2024-08-11 01:48:25 -06:00
aneesh-n
1d392a36f9
Fix extended attributes handling for VSS snapshots
2024-08-11 01:23:47 -06:00
Michael Eischer
3f5e2160de
Merge pull request #4938 from MichaelEischer/bump-go-version
...
Bump go version to 1.21
2024-08-10 19:57:59 +02:00
Michael Eischer
283225f15f
Merge pull request #4994 from MichaelEischer/less-flaky-watchdog-test
...
backend: tweak timeouts to make watchdog timeout test less flaky
2024-08-10 19:35:38 +02:00
Michael Eischer
400ae55940
replace deprecated usages of math/rand
2024-08-10 19:34:49 +02:00
Michael Eischer
0b19f6cf5a
Switch back to sha256 from the std library
...
The std library now also supports the sha assembly instructions on
ARM64. Thus, sha256-simd no longer provides a performance benefit.
2024-08-10 19:16:10 +02:00
Michael Eischer
ad48751adb
bump required go version to 1.21
2024-08-10 19:16:10 +02:00
