Michael Eischer
e4a48085ae
backend/retry: feature flag new retry behavior
2024-05-24 20:24:02 +02:00
Michael Eischer
b1266867d2
repository: wait max 1 minutes for lock removal if context is canceled
...
The toplevel context in restic only canceled if the user interrupts a
restic operation. If the network connection has failed this can require
waiting the full retry duration of 15 minutes which is a bad user
experience for interactive usage. Thus limit the delay to one minute in
this case.
2024-05-24 20:24:02 +02:00
Michael Eischer
98709a4372
retry: reduce total number of retries
...
Retries in restic try to solve two main problems:
- retry a temporarily failed operation
- tolerate temporary network interruptions
The first problem only requires a few retries, whereas the last one benefits
primarily from spreading the requests over a longer duration.
Increasing the default multiplier and the initial interval works for
both cases. The first few retries only take a few seconds, while later
retries quickly reach the maximum interval of one minute. This ensures
that the total number of retries issued by restic will remain at around
21 retries for a 15 minute period. As the concurrency in restic is
bounded, retries drastically reduce the number of requests sent to a
backend. This helps to prevent overloading the backend.
2024-05-24 20:24:02 +02:00
Michael Eischer
512cd6ef07
retry: ensure that there's always at least one retry
...
Previously, if an operation failed after 15 minutes, then it would never
be retried. This means that large backend requests are more unreliable
than smaller ones.
2024-05-24 20:24:02 +02:00
Michael Eischer
a60ee9b764
retry: limit retries based on elapsed time not count
...
Depending on how long an operation takes to fail, the total retry
duration can currently vary between 1.5 and 15 minutes. In particular
for temporarily interrupted network connections, the former timeout is
too short. Thus always use a limit of 15 minutes.
2024-05-24 20:24:02 +02:00
Michael Eischer
a3633cad9e
retry: explicitly log failed requests
...
This simplifies finding the request in the log output that cause an
operation to fail.
2024-05-24 20:24:02 +02:00
greatroar
66d03c797e
ui, termstatus: Move WrapStdio
...
Saves some imports. ui still needs to import ui/termstatus from
message.go.
2024-05-21 11:25:16 +02:00
greatroar
0b56214473
ui: Simplify stdio wrapper
...
The StdioWrapper type is really just a pair of io.WriteClosers, so
remove it in favor of a function that returns two of those. Test
coverage increases because the removed code was not tested.
2024-05-21 11:23:32 +02:00
greatroar
7f439a9c34
ui: Inline lineWriter into StdioWrapper
2024-05-21 11:00:09 +02:00
Michael Eischer
c56ecec9aa
azure: deduplicate cli and default credentials case
2024-05-18 22:15:54 +02:00
Maik Riechert
355f520936
Azure: add option to force use of CLI credential
2024-05-18 22:15:54 +02:00
Michael Eischer
223aa22cb0
replace some uses of restic.Repository with finegrained interfaces
2024-05-18 21:42:51 +02:00
Michael Eischer
291c9677de
restic/repository: remove Backend() method
2024-05-18 21:42:51 +02:00
Michael Eischer
673496b091
repository: clean cache between CheckPack retries
...
The cache cleanup pattern is also used in ListPack etc.
2024-05-18 21:42:51 +02:00
Michael Eischer
3d2410ed50
Replace some repo.RemoveUnpacked usages
...
These will eventually be blocked as they do not delete Snapshots.
2024-05-18 21:42:51 +02:00
Michael Eischer
d2c26e33f3
repository: remove further usages of repo.Backend()
2024-05-18 21:42:51 +02:00
Michael Eischer
8a425c2f0a
remove usages of repo.Backend() from tests
2024-05-18 21:42:51 +02:00
Michael Eischer
aa4647f773
repository: unexport PackBlobIterator
2024-05-18 21:42:51 +02:00
Michael Eischer
94e863885c
check: move verification of individual pack file to repository
2024-05-18 21:42:50 +02:00
Michael Eischer
e40943a75d
restic: remove backend usage from lock test
2024-05-18 21:38:31 +02:00
Michael Eischer
67e2ba0d40
repository: Lock requires *repository.Repository
...
This allows the Lock function to access the backend, even once the
Backend method is removed from the interface.
2024-05-18 21:38:31 +02:00
Michael Eischer
d8b184b3d3
repository: convert test helper to return *repository.Repository
2024-05-18 21:38:31 +02:00
Michael Eischer
a1ca5e15c4
migrations: add temporary hack for s3_layout
...
The migration will be removed after the next restic release anyways.
Thus, there's no need for a clean implementation.
2024-05-18 21:38:31 +02:00
Michael Eischer
34d90aecf9
migrations: move logic of upgrade_repo_v2 to repository package
...
The migration modifies repository internals and thus should live within
the repository package.
2024-05-18 21:38:31 +02:00
Michael Eischer
ab9077bc13
replace usages of backend.Remove() with repository.RemoveUnpacked()
...
RemoveUnpacked will eventually block removal of all filetypes other than
snapshots. However, getting there requires a major refactor to provide
some components with privileged access.
2024-05-18 21:38:31 +02:00
Michael Eischer
0c1ba6d95d
backend: remove unused Location method
2024-05-18 21:38:31 +02:00
Michael Eischer
74d90653e0
check: use ReadFull to load pack header in checkPack
...
This ensures that the pack header is actually read completely.
Previously, for a truncated file it was possible to only read a part of
the header, as backend.Load(...) is not guaranteed to return as many
bytes as requested by the length parameter.
2024-05-18 21:28:54 +02:00
Michael Eischer
8f8d872a68
fix compatibility with go 1.19
2024-05-18 21:28:54 +02:00
Michael Eischer
ff0744b3af
check: test checkPack retries
2024-05-18 21:28:54 +02:00
Michael Eischer
987c3b250c
repository: test retries of ListPack
2024-05-18 21:28:54 +02:00
Michael Eischer
bf16096771
repository: test LoadBlob retries
2024-05-18 21:28:54 +02:00
Michael Eischer
4f45668b7c
repository: rework and extend LoadRaw tests
2024-05-18 21:28:54 +02:00
Michael Eischer
ac805d6838
cache: cleanup debug logs
2024-05-18 21:28:54 +02:00
Michael Eischer
5214af88e2
cache: test forget behavior
2024-05-18 21:28:54 +02:00
Michael Eischer
3ff063e913
check: verify pack a second time if broken
2024-05-18 21:28:54 +02:00
Michael Eischer
385cee09dc
repository: fix caching of tree packs in LoadBlobsFromPack
2024-05-18 21:28:54 +02:00
Michael Eischer
e734746f75
cache: forget cached file at most once
...
This is inspired by the circuit breaker pattern used for distributed
systems. If too many requests fails, then it is better to immediately
fail new requests for a limited time to give the backend time to
recover.
By only forgetting a file in the cache at most once, we can ensure that
a broken file is only retrieved once again from the backend. If the file
stored there is broken, previously it would be cached and deleted
continuously. Now, it is retrieved only once again, all later requests
just use the cached copy and either succeed or fail immediately.
2024-05-18 21:28:54 +02:00
Michael Eischer
97a307df1a
cache: Always use cached file if it exists
...
A file is always cached whole. Thus, any out of bounds access will also
fail when directed at the backend. To handle case in which the cached
file is broken, then caller must call Cache.Forget(h) for the file in
question.
2024-05-18 21:28:54 +02:00
Michael Eischer
433a6aad29
repository: remove redundant blob loading fallback from RepairPacks
...
LoadBlobsFromPack already implements the same fallback behavior.
2024-05-18 21:28:54 +02:00
Michael Eischer
e401af07b2
check: fix error message formatting
2024-05-18 21:28:54 +02:00
Michael Eischer
7017adb7e9
repository: retry failed ListPack once
2024-05-18 21:28:54 +02:00
Michael Eischer
e33ce7f408
repository: retry failed LoadBlob once
2024-05-18 21:28:54 +02:00
Michael Eischer
2ace242f36
repository: make reloading broken files explicit
2024-05-18 21:28:54 +02:00
Michael Eischer
e9390352a7
cache: code cleanups
2024-05-18 21:26:00 +02:00
Michael Eischer
503c8140b1
repository: unify blob decoding code
2024-05-18 21:26:00 +02:00
Michael Eischer
6563f1d2ca
repository: remove redundant debug log
2024-05-18 21:26:00 +02:00
Michael Eischer
021fb49559
repository: Implement repository.LoadUnpacked using LoadRaw
...
Both functions were using a similar implementation.
2024-05-18 21:26:00 +02:00
Michael Eischer
779c8d3527
debug/repair packs/upgrade repo v2: use repository.LoadRaw
...
This replaces calling the low-level backend.Load() method.
2024-05-18 21:26:00 +02:00
Michael Eischer
1d6d3656b0
repository: move backend.LoadAll to repository.LoadRaw
...
LoadRaw also includes improved context cancellation handling similar to the
implementation in repository.LoadUnpacked.
The removed cache backend test will be added again later on.
2024-05-18 21:26:00 +02:00
Michael Eischer
47232bf8b0
backend: move LimitReadCloser to util package
...
The helper is only intended for usage by backend implementations.
2024-05-18 21:26:00 +02:00
Michael Eischer
53d15bcd1b
retry: add circuit breaker to load method
...
If a file exhausts its retry attempts, then it is likely not accessible
the next time. Thus, immediately fail all load calls for that file to
avoid useless retries.
2024-05-18 19:59:26 +02:00
Michael Eischer
394c8ca3ed
rest/rclone/s3/sftp/swift: move short file detection behind feature gate
...
These backends tend to use a large variety of server implementations.
Some of those implementations might prove problematic with the new
checks.
2024-05-18 19:59:26 +02:00
Michael Eischer
6328b7e1f5
replace "too small" with "too short" in error messages
2024-05-18 19:59:26 +02:00
Michael Eischer
aeb7eb245c
retry: do not retry permanent errors
...
This is currently gated behind a feature flag as some unexpected
interactions might show up in the wild.
2024-05-18 19:59:26 +02:00
Michael Eischer
bf8cc59889
Use generic backend-error-redesign feature flag instead of http-timeouts
...
An individual flag for each change of the backend error handling would
be too finegrained. Thus, add a generic flag.
2024-05-18 19:54:52 +02:00
Michael Eischer
4740528a0b
backend: add tests for IsPermanentError
2024-05-18 19:54:52 +02:00
Michael Eischer
6a85df7297
backend: add IsPermanentError() method to interface
2024-05-18 19:54:52 +02:00
Michael Eischer
cfc420664a
mem: stricter handling of out of bounds requests
2024-05-18 19:54:52 +02:00
Michael Eischer
d40f23e716
azure/b2/gs/s3/swift: adapt cloud backend
2024-05-18 19:54:51 +02:00
Michael Eischer
e793c002ec
local: stricter handling of short files
2024-05-18 19:54:21 +02:00
Michael Eischer
b4895ebd76
rest: rework error reporting and report too short files
2024-05-18 19:54:21 +02:00
Michael Eischer
eaa3f81d6b
sftp: check for truncated files without an extra backend request
2024-05-18 19:54:21 +02:00
Michael Eischer
c6d74458ee
sftp: improve handling of too short files
2024-05-18 19:54:21 +02:00
aneesh-n
9d1b23964d
Update comment for licensing
2024-05-17 14:19:05 -06:00
aneesh-n
5cff6e084e
Add support for Windows EA in node
...
Refactor Extended Attribute related functions in node files as windows apis get and set EA in bulk
2024-05-17 14:18:20 -06:00
aneesh-n
d6708505b9
Add helpers for Windows Extended Attributes
2024-05-17 14:15:03 -06:00
Michael Eischer
7ed560a201
Merge pull request #4796 from MichaelEischer/parallel-dump-load
...
dump: Parallelize loading large files
2024-05-14 22:35:44 +02:00
Michael Eischer
92221c2a6d
Merge pull request #4708 from zmanda/windows-securitydesc
...
Back up and restore SecurityDescriptors on Windows
2024-05-12 14:14:39 +00:00
Michael Eischer
b5fdb1d637
Merge pull request #4782 from MichaelEischer/fix-sftp-performance
...
Fix sftp upload performance
2024-05-12 15:28:33 +02:00
Michael Eischer
ebd01a4675
backend: add tests for watchdogRoundTripper
2024-05-09 23:46:17 +02:00
Michael Eischer
8778670232
backend: cancel stuck http requests
...
requests that make no upload or download progress within a timeout are
canceled.
2024-05-09 23:46:17 +02:00
Michael Eischer
0987c731ec
backend: configure protocol-level connection health checks
...
This should detect a connection that is stuck for more than 2 minutes.
2024-05-09 23:46:17 +02:00
aneesh-n
a4fd1b91e5
Fix review comments
...
Change lowerPrivileges from bool to atomic.Bool.
Add missing cleanup from upstream go-winio.
Add handling for ERROR_NOT_ALL_ASSIGNED warning.
2024-05-06 16:54:08 -06:00
Michael Eischer
4d55a62ada
bloblru: add test for GetOrCompute
2024-05-05 12:00:25 +02:00
Michael Eischer
7cce667f92
fuse: switch to use bloblru.GetOrCompute
2024-05-05 11:38:42 +02:00
Michael Eischer
bd03af2feb
dump: add GetOrCompute to bloblru cache
2024-05-05 11:38:42 +02:00
Michael Eischer
45509eafc8
dump: load blobs of a file from repository in parallel
2024-05-05 11:38:42 +02:00
Michael Eischer
ffe5439149
Merge pull request #4605 from MichaelEischer/better-restorer-error-handling
...
Rework repository.StreamPacks & better restorer error handling
2024-05-01 16:37:41 +02:00
aneesh-n
672f6cd776
Fix review comments for privileges and security flags
2024-04-29 17:29:51 -06:00
aneesh-n
08c6945d61
Fix review comments
2024-04-29 16:21:38 -06:00
Aneesh N
3f76b902e5
Merge branch 'master' into windows-securitydesc
2024-04-29 14:40:34 -06:00
DRON-666
125dba23c5
Rearange code
2024-04-29 01:27:34 +03:00
DRON-666
7ee889bb0d
Use S_FALSE and MaxInt
2024-04-29 01:25:25 +03:00
DRON-666
90b168eb6c
isMountPointExcluded to isMountPointIncluded
2024-04-29 01:23:50 +03:00
DRON-666
24330c19a8
Use kebab case in option names
2024-04-29 01:21:33 +03:00
DRON-666
5703e5a652
Fix texts and comments
2024-04-29 01:18:46 +03:00
DRON-666
0a8f9c5d9c
vss: Add tests for "provider" option
2024-04-28 22:45:21 +03:00
DRON-666
bb0f93ef3d
vss: Add "provider" option
2024-04-28 22:45:21 +03:00
DRON-666
3bac1f0135
vss: Fix issues reported by linters
2024-04-28 22:45:21 +03:00
DRON-666
88c509e3e9
vss: Change ErrorHandler
signature
...
We don't need `error` here: the only existing implementation
of `ErrorHandler` always call `Backup.Error` and all
implementations of `Backup.Error` always return nil.
2024-04-28 22:44:16 +03:00
DRON-666
9d3d915e2c
vss: Add some tests
2024-04-28 22:44:16 +03:00
DRON-666
c4f67c0064
vss: Add volume filtering
...
Add options to exclude all mountpoints and arbitrary volumes from snapshotting.
2024-04-28 22:44:15 +03:00
DRON-666
7470e5356e
vss: Add "timeout" option
...
Changing multiple "callAsyncFunctionAndWait" with fixed timeout
to calculated timeout based on deadline.
2024-04-28 22:44:15 +03:00
DRON-666
78dbc5ec58
vss: Add initial support for extended options
2024-04-28 22:44:15 +03:00
Michael Eischer
935327d480
sftp: slightly increase write concurrency
...
This should increase upload throughput for high latency links a bit.
2024-04-28 11:50:09 +02:00
Michael Eischer
669a669603
sftp: Fix upload performance issue
...
Since pkg/sftp 1.13.0 files were uploaded sequentially using 32kb chunks
instead of sending 64 chunks in parallel.
2024-04-28 11:48:26 +02:00
Michael Eischer
940a3159b5
let index.Each() and pack.Size() return error on canceled context
...
This forces a caller to actually check that the function did complete.
2024-04-22 22:39:32 +02:00
Michael Eischer
31624aeffd
Improve command shutdown on context cancellation
2024-04-22 22:31:38 +02:00
Michael Eischer
b15d867414
Merge pull request #4763 from MichaelEischer/refactor-prune
...
Refactor repair index / prune into the repository package
2024-04-22 22:24:53 +02:00
Michael Eischer
2e6c43c695
Merge pull request #4761 from MichaelEischer/fix-cache-race
...
cache: ignore ErrNotExist during cleanup of old files
2024-04-22 21:46:06 +02:00
Michael Eischer
f7632de3d6
Merge pull request #4772 from MichaelEischer/better-error-on-too-large-blob
...
repository: Better error message if blob is larger than 4GB
2024-04-22 21:45:06 +02:00
Michael Eischer
20d8eed400
repository: streamPack: separate requests for gap larger than 1MB
...
With most cloud providers, traffic is much more expensive than API
calls. Thus slightly bias streamPack towards a bit more API calls in
exchange for slightly less traffic.
2024-04-22 21:21:23 +02:00
Michael Eischer
cf700d8794
repository: streamPack: reuse zstd decoder
2024-04-22 21:21:23 +02:00
Michael Eischer
666a0b0bdb
repository: streamPack: replace streaming with chunked download
...
Due to the interface of streamPack, we cannot guarantee that operations
progress fast enough that the underlying connections remains open. This
introduces partial failures which massively complicate the error
handling.
Switch to a simpler approach that retrieves the pack in chunks of 32MB.
If a blob is larger than this limit, then it is downloaded separately.
To avoid multiple copies in memory, an auxiliary interface
`discardReader` is introduced that allows directly accessing the
downloaded byte slices, while still supporting the streaming used by the
`check` command.
2024-04-22 21:21:23 +02:00
Michael Eischer
621012dac0
repository: Add blob loading fallback to LoadBlobsFromPack
...
Try to retrieve individual blobs via LoadBlob if streaming did not work.
2024-04-21 21:35:55 +02:00
Michael Eischer
10355c3fb6
repository: Better error message if blob is larger than 4GB
2024-04-19 22:00:35 +02:00
coderwander
a82ed71de7
Fix struct names
...
Signed-off-by: coderwander <770732124@qq.com>
2024-04-18 10:02:09 +08:00
Michael Eischer
09587e6c08
repository: duplicate a few blobs in prune tests
2024-04-14 13:57:19 +02:00
Michael Eischer
defd7ae729
prune/repair index: reset in-memory index after command
...
The current in-memory index becomes stale after prune or repair index
have run. Thus, just drop the in-memory index altogether once these
commands have finished.
2024-04-14 13:46:24 +02:00
Michael Eischer
038586dc9d
repository: add minimal test for prune
2024-04-14 13:45:17 +02:00
Michael Eischer
d8622c86eb
prune: clean up internal interface
2024-04-14 13:45:15 +02:00
Michael Eischer
8d507c1372
repository: add basic test for RepairIndex
2024-04-14 13:45:15 +02:00
Michael Eischer
310db03c0e
repair index: improve log output if index cannot be deleted
...
The operation will always fail with an error if an index cannot be
deleted. Thus, this change is purely cosmetic.
2024-04-14 13:45:13 +02:00
Michael Eischer
7d1b9cde34
repository: use normal Init method in tests
2024-04-14 13:45:11 +02:00
Michael Eischer
b25fc2c89d
repository: remove redundant flushes from tests
2024-04-14 13:45:10 +02:00
Michael Eischer
c65459cd8a
repository: speed up tests
2024-04-14 13:45:10 +02:00
Michael Eischer
eda9f7beb4
ui/progress: add helper to print messages during tests
2024-04-14 13:45:08 +02:00
Michael Eischer
35277b7797
backend/mem: cleanup not found error message
2024-04-14 13:45:06 +02:00
Michael Eischer
7ba5e95a82
check: allow tests to only verify pack&index integrity
2024-04-14 13:45:04 +02:00
Michael Eischer
4c9a10ca37
repair packs: deduplicate index rebuild
2024-04-14 13:45:02 +02:00
Michael Eischer
85e4021619
prune: move additional option checks to repository
2024-04-14 13:44:58 +02:00
Michael Eischer
55d56db31b
Merge pull request #4743 from MichaelEischer/deprecate-s3legacy-layout
...
Deprecate s3legacy layout
2024-04-11 22:09:34 +02:00
Michael Eischer
fc3b548625
prune: move logic into repository package
2024-04-10 21:30:52 +02:00
Michael Eischer
866ddf5698
repair index: refactor code into repository package
2024-04-10 21:30:52 +02:00
Michael Eischer
591b421c4a
Deprecate s3legacy layout
2024-04-10 21:27:56 +02:00
Michael Eischer
bf054c09d2
backup: Ignore xattr.list permission error for parent directories
...
On FreeBSD, limited users may not be able to even list xattrs for the
parent directories above the snapshot source paths. As this can cause
the backup to fail, just ignore those errors.
2024-04-10 20:46:15 +02:00
Michael Eischer
0747cf5319
cache: ignore ErrNotExist during cleanup of old files
...
Two restic processes running concurrently can try to remove the same
files from the cache. This could cause one process to fail with an error
if the other one has already remove a file that the current process also
tries to delete.
2024-04-10 19:25:51 +02:00
Michael Eischer
49f98f25fc
Merge pull request #4742 from MichaelEischer/consistent-rtest-import
...
Use consistent alias for interal/test package
2024-04-03 21:47:32 +02:00
Michael Eischer
6cca1d5705
Merge pull request #4655 from ae-govau/unixsocket
...
Enhancement: option to send HTTP over unix socket
2024-04-03 19:29:21 +00:00
Michael Eischer
ec2b79834a
use consistent alias for interal/test package
2024-03-29 00:24:03 +01:00
Michael Eischer
5e98f1e2eb
repository: fix test setup race conditions
2024-03-28 23:17:02 +01:00
Michael Eischer
dc441c57a7
repository: unify repository initialization in tests
...
Tests should use a helper from internal/repository/testing.go to
construct a Repository object.
2024-03-28 23:17:02 +01:00
Michael Eischer
3ba1fa3cee
repository: remove a few global variables
2024-03-28 23:17:02 +01:00
Michael Eischer
044e8bf821
repository: parallelize lock tests
2024-03-28 23:17:02 +01:00
Michael Eischer
e8df50fa3c
repository: remove global list of locks
2024-03-28 22:46:33 +01:00
Michael Eischer
cbb5f89252
lock: move code to repository package
2024-03-28 22:46:33 +01:00
Michael Eischer
7f9ad1c3db
Merge pull request #4705 from MichaelEischer/snapshot-statistics
...
Store snapshot statistics & print snapshot size
2024-03-28 22:41:45 +01:00
Michael Eischer
d705741571
backup: test that deviceID is only stored for hardlinks
2024-03-28 19:32:50 +01:00
Michael Eischer
a9b3d86c4f
features: remove example feature
2024-03-28 19:12:07 +01:00
Michael Eischer
a26d6ffa72
archiver: move deviceID handling behind feature flag
2024-03-28 19:12:07 +01:00
Michael Eischer
2ba21fe72b
archiver: only store deviceID for hardlinks
...
The deviceID can change e.g. when backing up from filesystem snapshot.
It is only used for hardlink detection. Thus there it is not necessary
to store it for everything else.
2024-03-28 19:12:07 +01:00
Michael Eischer
8876e3025b
Merge pull request #4724 from MichaelEischer/disable-old-index
...
Deprecated legacy index format
2024-03-28 18:24:47 +01:00
Adam Eijdenberg
6e775d3787
Enhancement: option to send HTTP over unix socket
...
add tests for unix socket connection
switch HTTP rest-server test to use any free port
allow rest-server test graceful shutdown opportunity
2024-03-28 17:41:41 +01:00
avoidalone
ac948fccda
fix some typos
...
Signed-off-by: avoidalone <wuguangdong@outlook.com>
2024-03-11 14:35:12 +08:00
Michael Eischer
f8852f0eb6
repair index: fix deletion of legacy indexes
2024-03-09 18:21:22 +01:00
Michael Eischer
1a8bf358f1
index: deprecate legacy index format
2024-03-09 18:21:14 +01:00
Michael Eischer
a9b64cd7ad
features: print warning for stable/depreacted feature flags
2024-03-09 17:29:52 +01:00
Michael Eischer
70839155f2
features: add tests
2024-03-09 17:29:52 +01:00
Michael Eischer
5974a79497
features: add basic feature flag implementation
2024-03-09 17:29:52 +01:00
Michael Eischer
e71660cd1e
backup: rename data_added_in_repo statistic to data_added_packed
2024-02-25 20:40:52 +01:00
Aneesh Nireshwalia
062d408987
Clean up SecurityDescriptor helper
2024-02-24 14:23:04 -07:00
Aneesh Nireshwalia
5764300022
Add changelog and fix lint error
2024-02-24 13:47:49 -07:00
Aneesh Nireshwalia
90916f53de
Add test cases for security descriptors
2024-02-24 13:27:01 -07:00
Aneesh Nireshwalia
70cf8e3788
Add support for backup/restore of security descriptors
2024-02-24 13:25:28 -07:00
Aneesh Nireshwalia
e3e59fef24
Fix CombineErrors and fillExtendedAttr error handling
2024-02-24 13:22:34 -07:00
Aneesh Nireshwalia
09ce1b4e58
Create helper for SecurityDescriptor related functions
2024-02-24 13:16:25 -07:00
Michael Eischer
a8f5684f68
archiver: test statistics in snapshot
2024-02-23 22:05:15 +01:00
Michael Eischer
681395955e
archiver: test backup summary calculation
2024-02-23 21:46:39 +01:00
Michael Eischer
38f91d3b5e
backup: store statistics in snapshot
2024-02-23 20:27:13 +01:00
Michael Eischer
86897314d5
backup: expose data_added_in_repo in JSON output
...
The value describes how much data was added after compression.
Previously, it was only available in the text output.
2024-02-23 20:27:13 +01:00
Michael Eischer
a59f654fa6
archiver: refactor summary collection from ui into the archiver
2024-02-23 20:27:13 +01:00
Michael Eischer
8b1a85711f
archiver: unexport save/saveDir/saveTree methods
2024-02-23 20:24:21 +01:00
Aneesh Nireshwalia
4bbd25a37f
Add tests for generic attribute changes
2024-02-22 17:55:50 -07:00
Aneesh Nireshwalia
d4be734c73
Handle readonly empty files in windows
2024-02-22 17:54:43 -07:00
Aneesh Nireshwalia
eeb1aa5388
Add ability to report warnings to terminal
...
Report warnings to terminal when unrecognized generic attributes are found in the repository.
2024-02-22 17:52:26 -07:00
Aneesh Nireshwalia
0962917974
Support windows metadata using generic attribs
...
Add new generic_attributes attribute in Node.
Use the generic attributes to add support for creation time and file attributes like hidden, readonly, encrypted in windows. Handle permission errors for readonly files in windows.
Handle backup and restore of encrypted attributes using windows system calls.
2024-02-22 17:31:20 -07:00
Aneesh Nireshwalia
62a8a599f1
Add optional messages for Equals helper
2024-02-22 16:58:12 -07:00
Aneesh Nireshwalia
94de87d4b7
Add CombineErrors helper function
2024-02-22 16:57:00 -07:00
Michael Eischer
4c3218ef9f
repository: include packID in StreamPack for decrypt/decompress errors
2024-02-17 19:38:01 +01:00
Michael Eischer
18b0bbbf42
repository: use fmt.Errorf in StreamPacks
2024-02-17 19:37:32 +01:00
Michael Eischer
ed4a4f8748
check: exclude inaccessible files from the repair pack suggestion
2024-02-12 20:25:15 +01:00
Michael Eischer
4073299a7c
check: fix missing error if blob is invalid
2024-02-12 20:20:13 +01:00
Michael Eischer
22a3cea1b3
checker: wrap all pack errors in ErrPackData
2024-02-12 20:19:32 +01:00
Michael Eischer
5b5d506472
backup: report files whose chunks failed to upload
2024-02-11 22:43:26 +01:00
Alexander Neumann
c0514dd8ba
Fix linter errors (except for tests)
2024-02-10 22:58:10 +01:00
Michael Eischer
5957417b1f
Apply changelog entry / documentation improvements from review
2024-02-04 18:55:41 +01:00
Michael Eischer
86b38a0b17
rename --no-verify-pack
to --no-extra-verify
2024-02-04 17:01:05 +01:00
Michael Eischer
c32e5e2abb
pack: verify integrity of pack file header
2024-02-04 15:31:42 +01:00
Michael Eischer
c97a271e89
repository: ask users to report corrupted data while saving blobs
2024-02-04 15:31:42 +01:00
Michael Eischer
193140525c
repository: test verification of blobs/unpacked data
2024-02-04 15:31:42 +01:00
Michael Eischer
2dbb18128c
repository: Allow skipping verification for tests
...
Some tests have to explicitly create pack files with blobs that don't
match their ID. For those blobs the builtin verification of the
repository must be disabled.
2024-02-03 18:22:47 +01:00
Michael Eischer
30a84e9003
backup: verify unpacked files before upload
2024-02-03 18:22:47 +01:00
Michael Eischer
c01a0c6da7
backup: verify blobs before upload
...
This only covers the blobs themselves, the pack header is not verified
so far. Unpacked files are also not covered by the integrity check.
2024-02-03 18:22:47 +01:00
Michael Eischer
16e3f79e8b
repository: make repo.Options configurable for test repos
2024-02-03 18:22:47 +01:00
Michael Eischer
bb92b487f7
repository: fix repack test
2024-02-03 18:22:47 +01:00
Michael Eischer
e44e4b00a6
Merge pull request #4550 from ndecker/ls-ncdu
...
Ls ncdu
2024-01-27 12:27:35 +00:00
Michael Eischer
d4ed7c8858
walker: add tests for leaveDir
2024-01-27 13:17:33 +01:00
Michael Eischer
2c80cfa4a5
walker: fix missing leaveDir if directory is partially skipped
2024-01-27 13:17:33 +01:00
Michael Eischer
9ecbda059c
walker: add callback to inform about leaving a directory
2024-01-27 13:17:32 +01:00
Michael Eischer
246559e654
check: cleanup s3 legacy detection
2024-01-27 13:02:04 +01:00
Michael Eischer
1dfd854769
lock: switch to repository.List
2024-01-27 13:02:04 +01:00
Michael Eischer
bfb56b78e1
replace some usages of restic.Repository with more specific interface
...
This should eventually make it easier to test the code.
2024-01-27 13:02:02 +01:00
Michael Eischer
3424088274
Merge pull request #4644 from MichaelEischer/refactor-repair-packs
...
Refactor and test `repair packs`
2024-01-27 13:00:51 +01:00
Michael Eischer
724ec179e3
Merge pull request #4648 from MichaelEischer/repository-removekey
...
repository: Introduce RemoveKey function
2024-01-27 13:00:42 +01:00
Michael Eischer
f0e1ad2285
fix linter warning
2024-01-27 12:51:45 +01:00
Michael Eischer
fd579421dd
repository: deduplicate test
2024-01-27 12:51:45 +01:00
Michael Eischer
42c9318b9c
repair pack: add tests
2024-01-27 12:51:45 +01:00
Michael Eischer
764b0bacd6
repair pack: add support for truncated files
2024-01-27 12:51:45 +01:00
Michael Eischer
7c351bc53c
repair pack: reenable auto index updates
...
The method is not available on the restic.Repository interface that is
used for testing. Drop the call as a small amount of additional index
writes is not a problem.
2024-01-27 12:51:45 +01:00
Michael Eischer
feeab84204
repair pack: extract the repair logic into the repository package
...
Currently, the cmd/restic package contains a significant amount of code
that modifies repository internals. This code should in the mid-term
move into the repository package.
2024-01-27 12:51:45 +01:00
Michael Eischer
cb50832d50
index: let MasterIndex.Save also delete obsolete indexes
2024-01-27 12:51:08 +01:00
Michael Eischer
bedff1ed6d
split deleteFiles into UI and logic parts
2024-01-27 12:51:08 +01:00
Michael Eischer
c13bf0b607
repository: Introduce RemoveKey function
...
This replaces directly removing keys via the backend.
2024-01-27 12:42:58 +01:00
Michael Eischer
446167ae80
Merge pull request #4643 from MichaelEischer/remove-redundant-poly1305-mask
...
Remove redundant poly1305 key masking
2024-01-23 19:46:40 +01:00
Michael Eischer
1419baf67a
Merge pull request #4645 from MichaelEischer/improve-lock-checking
...
lock: checkForOtherLocks processes each lock at most once
2024-01-23 19:46:00 +01:00
Michael Eischer
6bdca13603
ui/termstatus: simplify cleaning up on termination
...
`writeStatus` also cleans no longer used status lines.
The old code actually cleaned one line too much. However, as that line
was never used it makes no difference.
2024-01-21 21:27:27 +01:00
Michael Eischer
f1f34eb3e5
lock: checkForOtherLocks processes each lock at most once
...
If a lock could not be loaded, then restic would check all lock files
again. These repeated checks are not useful as the status of a lock file
cannot change unless its ID changes too. Thus, skip already check lock
files on retries.
2024-01-20 22:40:12 +01:00
Michael Eischer
fee83e1c09
Remove redundant poly1305 key masking
...
The implementation in crypto/poly1305 already performs the exact same
masking.
2024-01-20 12:36:59 +01:00
Michael Eischer
a763a5c67d
s3: minor cleanups for archive storage class handling
2024-01-20 11:25:28 +01:00
Vladislav Belous
8ca58b487c
S3: do not set storage class for metadata when using archive storage
2024-01-20 11:04:15 +01:00
Michael Eischer
2c310a526e
repository: Replace StreamPack function with LoadBlobsFromPack method
...
LoadBlobsFromPack is now part of the repository struct. This ensures
that users of that method don't have to deal will internals of the
repository implementation.
The filerestorer tests now also contain far fewer pack file
implementation details.
2024-01-19 21:40:43 +01:00
Michael Eischer
6b7b5c89e9
repository: prepare StreamPack refactor
2024-01-19 21:40:43 +01:00
Michael Eischer
22d0c3f8dc
check: Use PackBlobIterator instead of StreamPack
...
To only stream the content of a pack file once, check used StreamPack
with a custom pack load function. This combination was always brittle
and complicates using StreamPack everywhere else. Now that StreamPack
internally uses PackBlobIterator use that primitive instead, which is a
much better fit for what the check command requires.
2024-01-19 21:40:36 +01:00
Michael Eischer
fb422497af
repository: split StreamPack implementation
...
Move the actual decoding of the pack data into a separate iterator.
2024-01-19 21:39:55 +01:00
Michael Eischer
54c5c72e5a
Merge pull request #4616 from MichaelEischer/fix-rest-connection-close
...
rest: fix and cleanup closing of http response body
2024-01-19 21:31:35 +01:00
Michael Eischer
bd883caae1
CI: enable bodyclose linter
2024-01-19 21:17:18 +01:00
Michael Eischer
b1a8fd1d03
rest: fix and cleanup closing of http response body
...
If client.Do returns an error, then there's no body that has to be
closed. For requests for which we are not interested in the response
body, immediately drain and close the body to make sure it isn't
forgotten later on.
This change in particular adds the missing `Close()` call for the
`List()` command.
2024-01-19 21:17:17 +01:00
Michael Eischer
fdcbb53017
walker: test skipping for root node
2024-01-19 21:16:06 +01:00
Michael Eischer
0b39940fdb
walker: Remove ignoreTrees functionality
...
It was only used in two places:
- stats: apparently as a minor performance optimization, which is
unlikely to be important
- find: filtered directories would be ignored. However, this
optimization missed that it is possible that two directories have the
exact same content. Such directories would be incorrectly ignored too.
Example:
```
mkdir test test/a test/b
restic backup test
restic find latest test/b
-> incorrectly does not return anything
```
Thus, remove the functionality as it's apparently too complex to use
correctly.
2024-01-19 21:16:06 +01:00
Michael Eischer
7b2de84763
Merge pull request #4618 from MichaelEischer/workaround-rclone-list-errors
...
rclone: Workaround for incorrect "not found" errors while listing files
2024-01-09 18:28:31 +01:00
Michael Eischer
c31e9418ba
Merge pull request #4626 from MichaelEischer/reliable-large-restores
...
Improve reliability of large restores
2024-01-09 18:23:09 +01:00
Michael Eischer
2e8de9edfd
rclone: Workaround for incorrect "not found" errors while listing files
...
rclone returns a "not found" error if an internal error occurs while
listing a folder. Ignoring this error lets restic erroneously think that
there are no files, which can cause `prune` to wipe the whole
repository.
2024-01-09 18:20:16 +01:00
Michael Eischer
e78be75d1e
restore: separately restore blobs that are frequently referenced
...
Writing these blobs to their files can take a long time and consequently
cause the backend connection to time out. Avoid that by retrieving these
blobs separately.
2024-01-08 21:00:13 +01:00
Michael Eischer
2267910418
restore: split error reporting from downloadPack
2024-01-08 20:57:00 +01:00
Michael Eischer
00d18b7a88
restore: cleanup downloadPack
2024-01-08 20:53:08 +01:00
Michael Eischer
9328f34d43
restore: split downloadPack into smaller methods
2024-01-08 20:52:36 +01:00
Michael Eischer
77434c6e2b
Merge pull request #4474 from ekristen/aws-assume-role
...
Allow AWS Assume Role
2024-01-08 19:07:17 +00:00
Michael Eischer
e4a7eb09ef
Merge pull request #4624 from MichaelEischer/better-restorer-error-reporting
...
Improver restorer error reporting
2024-01-07 11:20:29 +01:00
Michael Eischer
f8b4e932ef
Merge pull request #4620 from MichaelEischer/improve-irregular-file-handling
...
Improve irregular file handling
2024-01-07 11:12:07 +01:00
Michael Eischer
dac3508170
restore: only report errors for blobs that actually failed to load
...
Previously, errors would be reported for all blobs of a packfile that
failed to stream. Now, only the not yet processed blobs are reported.
2024-01-07 10:54:56 +01:00
Michael Eischer
77b1c52673
repository: test that StreamPack only delivers blobs once
2024-01-07 10:54:53 +01:00
Michael Eischer
fe5c337ca2
repository: StreamPack delivers blobs at most once
...
If an error occurred while streaming a pack file, this could result in
passing some of the blobs multiple times to the callback function. This
significantly complicates using StreamPack correctly and is unnecessary.
Retries do not change the content of a blob and thus only deliver the
same result over and over again.
2024-01-07 10:54:49 +01:00
Michael Eischer
0360e540af
Merge pull request #4622 from MichaelEischer/fix-outdated-windows-import
...
termstatus: update import path of golang.org/x/term
2024-01-06 23:24:16 +01:00
Michael Eischer
e6dfefba13
termstatus: update import path of golang.org/x/term
2024-01-06 21:59:26 +01:00
Michael Eischer
02bc73f5eb
s3: minor code cleanups
2024-01-06 21:44:53 +01:00
Michael Eischer
20cf4777cb
s3: check for EnvAWS credentials before Static credentials
...
EnvAWS considers more environment variables, including AWS_SESSION_TOKEN
and thus should be checked first.
2024-01-06 21:43:47 +01:00
Erik Kristensen
5ffb536aae
feat: support AWS assume role
2024-01-06 21:19:58 +01:00
Michael Eischer
33b7c84a7a
deduplicate string formatting of snapshot metadata
...
This removes the spurious ")" bracket at the end and normalizes the
metadata format used by the `ls` command.
2024-01-06 20:20:51 +01:00
Michael Eischer
51419c51d3
archiver: Add filepath to error message if it is not included yet
2024-01-06 19:08:24 +01:00
Michael Eischer
6b79834cc8
archiver: improve error message for irregular files
...
Since Go 1.21, most reparse points are considered as irregular files.
Depending on the underlying driver these can exhibit nearly arbitrary
behavior. When encountering such a file, restic returned an
indecipherable error message: `error: invalid node type ""`.
Add the filepath to the error message and state that the file type is
not supported.
2024-01-06 19:03:11 +01:00
Michael Eischer
0018bb7854
restic: cleanup node type determination
...
os.ModeCharDevice is already included in os.ModeType
2024-01-06 18:43:16 +01:00
Michael Eischer
649a6409ee
rewrite: cleanup tests
2023-12-24 15:36:22 +01:00
Michael Eischer
30e6ed038c
Merge pull request #4586 from Gelma/typos
...
Fix typos
2023-12-23 13:21:50 +00:00
Michael Eischer
53ebe91a50
Move changelog typo fixes to changelog files
2023-12-23 14:05:47 +01:00
Michael Eischer
dbbd31bc3a
CI: update golangci-lint
...
Necessary to properly support Go 1.21.
2023-12-23 13:41:30 +01:00
Andrea Gelmini
241916d55b
Fix typos
2023-12-06 13:11:55 +01:00
Michael Eischer
ee305e6041
backup: rework error reporting for subcommand
2023-10-27 23:58:52 +02:00
Michael Eischer
8bceb8e359
fs: add tests for CommandReader
2023-10-27 23:58:52 +02:00
Michael Eischer
317144c1d6
fs: merge command startup into CommandReader
2023-10-27 23:58:51 +02:00
Michael Eischer
7d879705ad
fs: cleanup CommandReader implementation
2023-10-27 23:58:51 +02:00
Enrico204
37a312e505
restic-from-command: use standard behavior when no output and exit code 0 from command
...
The behavior of the new option should reflect the behavior of normal backups: when the command exit code is zero and there is no output in the stdout, emit a warning but create the snapshot. This commit fixes the integration tests and the ReadCloserCommand struct.
2023-10-27 23:58:51 +02:00
Enrico204
81f8d473df
restic-from-command: abort snapshot on non-zero exit codes
2023-10-27 23:58:51 +02:00
Enrico204
4e5caab114
stdin-from-command: implemented suggestions in #4254
...
The code has been refactored so that the archiver is back to the original code, and the stderr is handled using a go routine to avoid deadlock.
2023-10-27 23:58:51 +02:00
Sebastian Hoß
c133065a9f
Check command result before snapshotting
...
Return with an error containing the stderr of the given command in case it fails. No new snapshot will be created and future prune operations on the repository will remove the unreferenced data.
Signed-off-by: Sebastian Hoß <seb@xn--ho-hia.de>
2023-10-27 23:58:51 +02:00
Sebastian Hoß
25350a9c55
Extend SnapshotOptions w/ command data
...
In order to determine whether to save a snapshot, we need to capture the exit code returned by a command. In order to provide a nice error message, we supply stderr as well.
Signed-off-by: Sebastian Hoß <seb@xn--ho-hia.de>
2023-10-27 23:58:51 +02:00
Michael Eischer
a8fdcf79b7
restorer: Make hardlink index generic
...
This will allow reusing it for the stats command without regressing the
memory usage due to storing an unnecessary file path.
2023-10-27 23:40:42 +02:00
Michael Eischer
45962c2847
Merge pull request #4499 from MichaelEischer/modular-backend-code
...
Split backend code from restic package
2023-10-27 20:19:20 +02:00
Leo R. Lundgren
aafb806a8c
doc: Correct two typos
2023-10-27 18:56:32 +02:00
Martin Michlmayr
41e6a02bcc
Fix typos
2023-10-27 18:56:32 +02:00
Michael Eischer
c7b770eb1f
convert MemorizeList to be repository based
...
Ideally, code that uses a repository shouldn't directly interact with
the underlying backend. Thus, move MemorizeList one layer up.
2023-10-25 23:01:35 +02:00
Michael Eischer
1b8a67fe76
move Backend interface to backend package
2023-10-25 23:00:18 +02:00
Michael Eischer
ceb0774af1
backend: make LoadAll independent of restic package
2023-10-25 22:58:39 +02:00
Michael Eischer
b6d79bdf6f
restic: decouple restic.Handle
2023-10-25 22:54:07 +02:00
Michael Eischer
7881309d63
backend: move backend implementation helpers to util package
...
This removes code that is only used within a backend implementation from
the backend package. The latter now only contains code that also has
external users.
2023-10-25 22:54:07 +02:00
Michael Eischer
8e6fdf5edf
Merge pull request #4520 from awannabeengineer/load-retry-nonexistent
...
retry: Do not retry Load() if file does not exist
2023-10-25 20:42:05 +00:00
Michael Eischer
a28940ea29
check: Suggest usage of restic repair packs
for corrupted blobs
...
For now, the guide is only shown if the blob content does not match its
hash. The main intended usage is to handle data corruption errors when
using maximum compression in restic 0.16.0
2023-10-23 18:36:28 +02:00
Arash Farr
d15ffd9c92
retry: Do not retry Load() if file does not exist
2023-10-22 13:25:32 -05:00
Michael Eischer
7f05af02b9
Merge pull request #4480 from AgathaSorceress/add-rest-auth-env
...
Support reading basic auth credentials for REST server from environment variables
2023-10-21 17:41:08 +00:00
Michael Eischer
ab6defbace
Merge pull request #4519 from adamantike/backend/sftp/add-args-option
...
backend/sftp: Add sftp.args option
2023-10-21 17:37:46 +00:00
Michael Eischer
6a4d6d5da4
sftp: test that Args and Command options cannot be set at the same time
2023-10-21 19:26:39 +02:00
Michael Manganiello
41f70f1f4f
backend/sftp: Add sftp.args option
...
Allow setting custom arguments for the `sftp` backend, by using the
`sftp.args` option. This is similar to the approach already implemented
in the `rclone` backend, to support new arguments without requiring
future code changes for each different SSH argument.
Closes #4241
2023-10-21 19:25:44 +02:00
Michael Eischer
6c7560e537
backend/b2: Switch to official library Backblaze/blazer
...
According to https://github.com/kurin/blazer Readme, the repository is
no longer maintained, and the official one at
https://github.com/Backblaze/blazer took over.
Closes #4148
2023-10-21 19:18:13 +02:00
Agatha V. Lovelace
f1877e721e
feat: support reading REST credentials from env
2023-10-20 13:04:58 +02:00
Michael Eischer
d27cfd10a9
Merge pull request #4487 from lbausch/tweak-restore-summary
...
Tweak restore summary
2023-10-19 20:14:14 +00:00
Michael Eischer
d173d1c532
Merge pull request #4486 from MichaelEischer/separate-large-blobs
...
repository: store oversized blobs in separate pack files
2023-10-17 23:03:52 +02:00
Michael Eischer
cb9cbe55d9
repository: store oversized blobs in separate pack files
...
Store oversized blobs in separate pack files as the blobs is large
enough to warrant its own pack file. This simplifies the garbage
collection of such blobs and keeps the cache smaller, as oversize (tree)
blobs only have to be downloaded if they are actually used.
2023-10-17 22:52:16 +02:00
Michael Eischer
f750aa8dfb
backend/limiter: replace juju/ratelimit with x/time/rate
2023-10-17 22:49:25 +02:00
Michael Eischer
91aef00df3
check: add index loading progress bar
2023-10-01 19:55:29 +02:00
Michael Eischer
3fd0ad7448
repository: list index files only once
2023-10-01 19:53:26 +02:00
arjunajesh
ed65a7dbca
implement progress bar for index loading
2023-10-01 19:52:59 +02:00
Lorenz Bausch
2657217574
Tweak restore summary
2023-09-20 10:59:55 +02:00
Michael Eischer
6e586b64e4
Merge pull request #4450 from MichaelEischer/cleanup-master-index-test
...
master_index: fix inconsistent length blob length in test
2023-09-06 22:55:48 +02:00
greatroar
691c01963b
internal/restic: Return summary from BlobSet.String
...
Fixes #4449 : error messages from prune were too long to fit in scroll
buffers.
2023-08-25 21:41:30 +02:00
Michael Eischer
2bec99dc6f
master_index: fix inconsistent length blob length in test
...
Two blobs with the same hash must always have the same content length.
2023-08-19 20:04:25 +02:00
Michael Eischer
feb6abb7bb
Merge pull request #4394 from greatroar/acl
...
dump: Rewrite Linux ACL handling
2023-08-13 17:01:31 +00:00
greatroar
aaf5254e26
dump: Rewrite Linux ACL handling
...
The old version was taken from an MPL-licensed library. This is a
cleanroom implementation. The code is shorter and it's now explicit that
only Linux ACLs are supported.
2023-08-13 13:37:11 +02:00
Michael Eischer
4a33370072
backend: Fix unreliable TestListCancel/Timeout test
...
The test uses `WithTimeout` to create a context that cancels the List
operation after a given delay. Several backends internally use a derived
child context created using WithCancel.
The cancellation of a context first closes the done channel of the
context (here: the `WithTimeout` context) and _afterwards_ propagates
the cancellation to child contexts (here: the `WithCancel` context).
Therefor if the List implementation uses a child context, then it may
take a moment until that context is also cancelled. Thus give the
context cancellation a moment to propagate.
2023-07-23 11:47:56 +02:00
Michael Eischer
f12bbd9229
restic: check that Node.LinkTarget can handle non-utf8 characters
2023-07-23 00:05:14 +02:00
Michael Eischer
6adb629608
Add support for non-utf8 symlink targets
2023-07-23 00:05:14 +02:00
Michael Eischer
25ff9fa893
Merge pull request #4334 from MichaelEischer/snapshot-subtree-syntax
...
Add support for snapshot:path syntax
2023-07-22 23:59:26 +02:00
Michael Eischer
3f63b53090
lock: Shrink critical section in RefreshStaleLock
...
A cleaning up after a failed lock refresh attempt does not require a
mutex.
2023-07-22 23:29:14 +02:00
Michael Eischer
96eada3d5f
backend/sema: add test for freeze functionality
2023-07-22 23:23:17 +02:00
Michael Eischer
8154f6a77a
Ensure consistent naming for <snapshot>:<subfolder>
syntax
2023-07-22 19:55:57 +02:00
Michael Eischer
090f9d6237
restic: Cleanup and simplify TestCreateSnapshot
2023-07-22 19:55:57 +02:00
Michael Eischer
321cc35cde
restic: add test for FindTreeDirectory
2023-07-22 19:55:57 +02:00
Michael Eischer
a97915642c
restic: Ensure snapshots created by TestCreateSnapshot are valid
...
All nodes in a tree must be sorted by node name. Otherwise functionality
like searching for a specific tree node will break.
2023-07-22 19:55:57 +02:00
Michael Eischer
c64d81063e
test snapshot filtering with snapshot:path syntax
2023-07-22 19:55:57 +02:00
Michael Eischer
85860e6e97
Add support for snapshot:subpath syntax
...
This snapshot specification syntax is supported by the cat, diff, dump,
ls and restore command.
2023-07-22 19:55:57 +02:00
Michael Eischer
5d9b0d894e
lock: add unit test for RefreshStaleLock
2023-07-22 19:55:44 +02:00
Michael Eischer
d4bf7a3cb1
restic: cleanup lock test
2023-07-22 19:55:44 +02:00
Michael Eischer
24c8a33da9
restic: reduce sleeps in lock tests
2023-07-22 19:55:44 +02:00
Michael Eischer
f490288738
lock: freeze backend operations while refreshing stale lock
...
Freeze new backend operations while trying to refresh a stale lock.
2023-07-22 19:55:44 +02:00
Michael Eischer
51718ec561
restic: generic implemention of AsBackend
2023-07-22 19:55:44 +02:00
Michael Eischer
bee3231ed4
lock: try refreshing of stale locks
...
A stale lock may be refreshed if it continues to exist until after a
replacement lock has been created. This ensures that a repository was
not unlocked in the meantime.
2023-07-22 19:55:44 +02:00
Michael Eischer
978ebaac49
rest: use http status code constants
2023-07-14 22:39:38 +02:00
Michael Eischer
307aeb6849
rest: Don't return error if listing non-existent directory
...
When transferring a repository from S3 to, for example, a local disk
then all empty folders will be missing.
When saving files, the missing intermediate folders are created
automatically. Therefore, missing directories can be ignored by the
`List()` operation.
2023-07-14 22:39:38 +02:00
Michael Eischer
e990d3d483
azure: Support authentication using managed / workload identity
...
See https://github.com/Azure/azure-sdk-for-go/tree/sdk/azidentity/v1.3.0/sdk/azidentity
2023-07-09 14:47:03 +02:00
Michael Eischer
9d44682e3e
fuse: fix windows build
2023-07-08 22:40:22 +02:00
Michael Eischer
4d43509423
Merge pull request #4379 from chenxiaolong/symlink_xattrs
...
Add support for extended attributes on symlinks
2023-07-08 16:56:51 +00:00
Michael Eischer
ea9ad77e05
restic: refactor node test
2023-07-08 18:54:56 +02:00
Michael Eischer
cc84884d2e
restic: basic xattr test for files/dirs
2023-07-08 18:49:21 +02:00
Michael Eischer
4a5ae2ba84
restic: test NodeFromFileInfo for symlinks
2023-07-08 18:18:13 +02:00
Michael Eischer
1f1e50f49e
fuse: add test for symlink xattr
2023-07-08 18:02:17 +02:00
Michael Eischer
f3c3b0f377
fuse: deduplicate xattr code
2023-07-08 17:41:45 +02:00
Michael Eischer
bbac74b172
add program version to snapshot
2023-07-07 23:27:10 +02:00
Michael Eischer
a280b7364e
Merge pull request #4395 from greatroar/parsebytes
...
cmd, ui: Move size parsing code and make it more robust
2023-07-07 21:24:47 +00:00
Michael Eischer
e36d17a6f8
azure: tweak documentation for endpoint suffix
2023-07-07 23:10:26 +02:00
arjunajesh
068b115abc
added azure domain parameter
2023-07-07 23:10:26 +02:00
greatroar
41a5bf357f
cmd, ui: Move size parsing code and make it more robust
2023-07-02 21:44:50 +02:00
Andrew Gunnerson
8c02ebb029
Add support for extended attributes on symlinks
...
Linux allows the use of non-`user.` extended attributes on symlinks. One
of the main users of this functionality is SELinux's `security.selinux`
xattr for storing a path's label. By storing symlink xattrs, restic is
now suitable for backing up the root filesystem on Linux distributions
that use SELinux.
This commit adds support for symlink xattrs when backing up data,
restoring data, and mounting snapshots via a fuse mount. All calls to
the xattr library have been updated to the use `L` variants of the
various functions, which always operate on the path given, without
following symlinks.
Fixes : #4375
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
2023-06-19 14:37:31 -04:00
Michael Eischer
74ca82a6f8
Merge pull request #4362 from restic/unified-backend-factory
...
Unified backend initialization
2023-06-17 15:52:37 +02:00
Michael Eischer
93038ed8f4
s3: restore retries for minio tests
2023-06-17 15:25:08 +02:00
Gautam Menghani
8da5a6649b
Preserve oldest snapshot when keep-within* does not collect enough
2023-06-17 15:17:20 +02:00
Michael Eischer
1257c2c075
forget: Add comments to snapshot policy
2023-06-17 15:16:58 +02:00
Gautam Menghani
182b9796e4
Issue #3624 : Preserve oldest snapshot when keep-* values are not satisfied
2023-06-17 15:16:58 +02:00
Michael Eischer
b5511e8e4c
Fix linter warnings
2023-06-17 15:15:58 +02:00
Michael Eischer
50e0d5e6b5
backend: Hardcode backend scheme in Factory
...
Our ParseConfig implementations always expect a specific scheme, thus no
other scheme would work.
2023-06-17 15:15:58 +02:00
Michael Eischer
705ad51bcc
backend: check that StripPassword can be called
2023-06-17 15:15:58 +02:00
Michael Eischer
13a8b5822f
backend: Adjust tests to use the Factory to instantiate the backend
...
This drastically reduces the amount of duplicated test code.
2023-06-17 15:15:58 +02:00
Michael Eischer
3d3bb88745
b2: remove duplicate check for config file during repository creation
...
No other backend implements that check. The check that a repository is
not yet initialized is handled by the Repository later on.
2023-06-17 15:15:58 +02:00
Michael Eischer
3a3cf608f5
b2/s3: Move config validation from ApplyEnvironment to Open/Create
...
Conceptually the backend configuration should be validated when creating
or opening the backend, but not when filling in information from
environment variables into the configuration.
2023-06-17 15:15:58 +02:00
Michael Eischer
19ac12d95b
location: make genericBackendFactory private
2023-06-17 15:15:58 +02:00
Michael Eischer
3325a7c862
location: extract backend specific part of StripPassword
...
The tests for the rest backend now reside there.
2023-06-17 15:15:58 +02:00
Michael Eischer
555be49a79
location: Make ParseConfig-test backend agnostic
...
The backend specific parts of the test are now directly handled by the
respective backend. Duplicate tests were removed.
2023-06-17 15:15:58 +02:00
Michael Eischer
9aa9e0d1ec
local/sftp: move limiter setup into backend
2023-06-17 15:15:58 +02:00
Michael Eischer
7d12c29286
backend: Unify backend construction using factory and registry
...
This unified construction removes most backend-specific code from
global.go. The backend registry will also enable integration tests to
use custom backends if necessary.
2023-06-17 15:15:57 +02:00
Michael Eischer
56836364a4
backend: pass context into every backend constructor
2023-06-17 15:15:57 +02:00
Michael Eischer
4df77e9f26
Merge pull request #4314 from MichaelEischer/restore-json
...
Add JSON support for the restore progress bar
2023-06-16 23:53:17 +02:00
Michael Eischer
191c47d30e
Merge pull request #4353 from MichaelEischer/tune-gc
...
Tune Go garbage collector
2023-06-16 23:24:39 +02:00
Michael Eischer
dd1ef13c1c
Merge pull request #4352 from MichaelEischer/pointerless-index
...
index: optimize go GC performance
2023-06-16 23:23:14 +02:00
Michael Eischer
b2ed42cec4
index: add basic hat test
2023-06-16 23:12:30 +02:00
Michael Eischer
07d1f8047e
restore: More descriptive field names for the JSON output
2023-06-09 12:07:55 +02:00
Michael Eischer
b2b0760eb0
restore: add test for json output
2023-06-09 12:03:33 +02:00
Michael Eischer
d54176ce5d
restore: add basic json progress
2023-06-09 12:03:33 +02:00
Michael Eischer
a9aff885d6
restore: reorganize progress bar code
...
The structure is now much more similar to that of the backup command.
2023-06-09 12:03:33 +02:00
DRON-666
c37d587f81
Use PreallocateFile in local backend
2023-06-09 11:49:06 +03:00
DRON-666
ffc6b3d887
Move PreallocateFile to fs package
2023-06-09 11:49:06 +03:00
Michael Eischer
0b908bb1fb
Address review comments
2023-06-08 20:24:21 +02:00
Matt Armstrong
0372c7ef04
Improve the ETA displayed during backup
...
The ETA restic displays was based on a rate computed across the entire
backup operation. Often restic can progress at uneven rates. In the worst
case, restic progresses over most of the backup at a very high rate and
then finds new data to back up. The displayed ETA is then unrealistic and
never adapts.
Restic now estimates the transfer rate based on a sliding window, with the
goal of adapting to observed changes in rate. To avoid wild changes in the
estimate, several heuristics are used to keep the sliding window wide
enough to be relatively stable.
2023-06-08 20:05:56 +02:00
Michael Eischer
55c21846b1
Revert "index: remove redundant storage of indexmap size"
...
This reverts commit f1c388c623
.
For an uninitialized indexmap the returned size was `-1` which is
unexpected and could cause problems.
2023-06-08 18:08:46 +02:00