Michael Eischer
00d18b7a88
restore: cleanup downloadPack
2024-01-08 20:53:08 +01:00
Michael Eischer
9328f34d43
restore: split downloadPack into smaller methods
2024-01-08 20:52:36 +01:00
Michael Eischer
77434c6e2b
Merge pull request #4474 from ekristen/aws-assume-role
...
Allow AWS Assume Role
2024-01-08 19:07:17 +00:00
Michael Eischer
e4a7eb09ef
Merge pull request #4624 from MichaelEischer/better-restorer-error-reporting
...
Improver restorer error reporting
2024-01-07 11:20:29 +01:00
Michael Eischer
f8b4e932ef
Merge pull request #4620 from MichaelEischer/improve-irregular-file-handling
...
Improve irregular file handling
2024-01-07 11:12:07 +01:00
Michael Eischer
dac3508170
restore: only report errors for blobs that actually failed to load
...
Previously, errors would be reported for all blobs of a packfile that
failed to stream. Now, only the not yet processed blobs are reported.
2024-01-07 10:54:56 +01:00
Michael Eischer
77b1c52673
repository: test that StreamPack only delivers blobs once
2024-01-07 10:54:53 +01:00
Michael Eischer
fe5c337ca2
repository: StreamPack delivers blobs at most once
...
If an error occurred while streaming a pack file, this could result in
passing some of the blobs multiple times to the callback function. This
significantly complicates using StreamPack correctly and is unnecessary.
Retries do not change the content of a blob and thus only deliver the
same result over and over again.
2024-01-07 10:54:49 +01:00
Michael Eischer
0360e540af
Merge pull request #4622 from MichaelEischer/fix-outdated-windows-import
...
termstatus: update import path of golang.org/x/term
2024-01-06 23:24:16 +01:00
Michael Eischer
e6dfefba13
termstatus: update import path of golang.org/x/term
2024-01-06 21:59:26 +01:00
Michael Eischer
02bc73f5eb
s3: minor code cleanups
2024-01-06 21:44:53 +01:00
Michael Eischer
20cf4777cb
s3: check for EnvAWS credentials before Static credentials
...
EnvAWS considers more environment variables, including AWS_SESSION_TOKEN
and thus should be checked first.
2024-01-06 21:43:47 +01:00
Erik Kristensen
5ffb536aae
feat: support AWS assume role
2024-01-06 21:19:58 +01:00
Michael Eischer
33b7c84a7a
deduplicate string formatting of snapshot metadata
...
This removes the spurious ")" bracket at the end and normalizes the
metadata format used by the `ls` command.
2024-01-06 20:20:51 +01:00
Michael Eischer
51419c51d3
archiver: Add filepath to error message if it is not included yet
2024-01-06 19:08:24 +01:00
Michael Eischer
6b79834cc8
archiver: improve error message for irregular files
...
Since Go 1.21, most reparse points are considered as irregular files.
Depending on the underlying driver these can exhibit nearly arbitrary
behavior. When encountering such a file, restic returned an
indecipherable error message: `error: invalid node type ""`.
Add the filepath to the error message and state that the file type is
not supported.
2024-01-06 19:03:11 +01:00
Michael Eischer
0018bb7854
restic: cleanup node type determination
...
os.ModeCharDevice is already included in os.ModeType
2024-01-06 18:43:16 +01:00
Michael Eischer
649a6409ee
rewrite: cleanup tests
2023-12-24 15:36:22 +01:00
Michael Eischer
30e6ed038c
Merge pull request #4586 from Gelma/typos
...
Fix typos
2023-12-23 13:21:50 +00:00
Michael Eischer
53ebe91a50
Move changelog typo fixes to changelog files
2023-12-23 14:05:47 +01:00
Michael Eischer
dbbd31bc3a
CI: update golangci-lint
...
Necessary to properly support Go 1.21.
2023-12-23 13:41:30 +01:00
Andrea Gelmini
241916d55b
Fix typos
2023-12-06 13:11:55 +01:00
Michael Eischer
ee305e6041
backup: rework error reporting for subcommand
2023-10-27 23:58:52 +02:00
Michael Eischer
8bceb8e359
fs: add tests for CommandReader
2023-10-27 23:58:52 +02:00
Michael Eischer
317144c1d6
fs: merge command startup into CommandReader
2023-10-27 23:58:51 +02:00
Michael Eischer
7d879705ad
fs: cleanup CommandReader implementation
2023-10-27 23:58:51 +02:00
Enrico204
37a312e505
restic-from-command: use standard behavior when no output and exit code 0 from command
...
The behavior of the new option should reflect the behavior of normal backups: when the command exit code is zero and there is no output in the stdout, emit a warning but create the snapshot. This commit fixes the integration tests and the ReadCloserCommand struct.
2023-10-27 23:58:51 +02:00
Enrico204
81f8d473df
restic-from-command: abort snapshot on non-zero exit codes
2023-10-27 23:58:51 +02:00
Enrico204
4e5caab114
stdin-from-command: implemented suggestions in #4254
...
The code has been refactored so that the archiver is back to the original code, and the stderr is handled using a go routine to avoid deadlock.
2023-10-27 23:58:51 +02:00
Sebastian Hoß
c133065a9f
Check command result before snapshotting
...
Return with an error containing the stderr of the given command in case it fails. No new snapshot will be created and future prune operations on the repository will remove the unreferenced data.
Signed-off-by: Sebastian Hoß <seb@xn--ho-hia.de>
2023-10-27 23:58:51 +02:00
Sebastian Hoß
25350a9c55
Extend SnapshotOptions w/ command data
...
In order to determine whether to save a snapshot, we need to capture the exit code returned by a command. In order to provide a nice error message, we supply stderr as well.
Signed-off-by: Sebastian Hoß <seb@xn--ho-hia.de>
2023-10-27 23:58:51 +02:00
Michael Eischer
a8fdcf79b7
restorer: Make hardlink index generic
...
This will allow reusing it for the stats command without regressing the
memory usage due to storing an unnecessary file path.
2023-10-27 23:40:42 +02:00
Michael Eischer
45962c2847
Merge pull request #4499 from MichaelEischer/modular-backend-code
...
Split backend code from restic package
2023-10-27 20:19:20 +02:00
Leo R. Lundgren
aafb806a8c
doc: Correct two typos
2023-10-27 18:56:32 +02:00
Martin Michlmayr
41e6a02bcc
Fix typos
2023-10-27 18:56:32 +02:00
Michael Eischer
c7b770eb1f
convert MemorizeList to be repository based
...
Ideally, code that uses a repository shouldn't directly interact with
the underlying backend. Thus, move MemorizeList one layer up.
2023-10-25 23:01:35 +02:00
Michael Eischer
1b8a67fe76
move Backend interface to backend package
2023-10-25 23:00:18 +02:00
Michael Eischer
ceb0774af1
backend: make LoadAll independent of restic package
2023-10-25 22:58:39 +02:00
Michael Eischer
b6d79bdf6f
restic: decouple restic.Handle
2023-10-25 22:54:07 +02:00
Michael Eischer
7881309d63
backend: move backend implementation helpers to util package
...
This removes code that is only used within a backend implementation from
the backend package. The latter now only contains code that also has
external users.
2023-10-25 22:54:07 +02:00
Michael Eischer
8e6fdf5edf
Merge pull request #4520 from awannabeengineer/load-retry-nonexistent
...
retry: Do not retry Load() if file does not exist
2023-10-25 20:42:05 +00:00
Michael Eischer
a28940ea29
check: Suggest usage of restic repair packs
for corrupted blobs
...
For now, the guide is only shown if the blob content does not match its
hash. The main intended usage is to handle data corruption errors when
using maximum compression in restic 0.16.0
2023-10-23 18:36:28 +02:00
Arash Farr
d15ffd9c92
retry: Do not retry Load() if file does not exist
2023-10-22 13:25:32 -05:00
Michael Eischer
7f05af02b9
Merge pull request #4480 from AgathaSorceress/add-rest-auth-env
...
Support reading basic auth credentials for REST server from environment variables
2023-10-21 17:41:08 +00:00
Michael Eischer
ab6defbace
Merge pull request #4519 from adamantike/backend/sftp/add-args-option
...
backend/sftp: Add sftp.args option
2023-10-21 17:37:46 +00:00
Michael Eischer
6a4d6d5da4
sftp: test that Args and Command options cannot be set at the same time
2023-10-21 19:26:39 +02:00
Michael Manganiello
41f70f1f4f
backend/sftp: Add sftp.args option
...
Allow setting custom arguments for the `sftp` backend, by using the
`sftp.args` option. This is similar to the approach already implemented
in the `rclone` backend, to support new arguments without requiring
future code changes for each different SSH argument.
Closes #4241
2023-10-21 19:25:44 +02:00
Michael Eischer
6c7560e537
backend/b2: Switch to official library Backblaze/blazer
...
According to https://github.com/kurin/blazer Readme, the repository is
no longer maintained, and the official one at
https://github.com/Backblaze/blazer took over.
Closes #4148
2023-10-21 19:18:13 +02:00
Agatha V. Lovelace
f1877e721e
feat: support reading REST credentials from env
2023-10-20 13:04:58 +02:00
Michael Eischer
d27cfd10a9
Merge pull request #4487 from lbausch/tweak-restore-summary
...
Tweak restore summary
2023-10-19 20:14:14 +00:00
Michael Eischer
d173d1c532
Merge pull request #4486 from MichaelEischer/separate-large-blobs
...
repository: store oversized blobs in separate pack files
2023-10-17 23:03:52 +02:00
Michael Eischer
cb9cbe55d9
repository: store oversized blobs in separate pack files
...
Store oversized blobs in separate pack files as the blobs is large
enough to warrant its own pack file. This simplifies the garbage
collection of such blobs and keeps the cache smaller, as oversize (tree)
blobs only have to be downloaded if they are actually used.
2023-10-17 22:52:16 +02:00
Michael Eischer
f750aa8dfb
backend/limiter: replace juju/ratelimit with x/time/rate
2023-10-17 22:49:25 +02:00
Michael Eischer
91aef00df3
check: add index loading progress bar
2023-10-01 19:55:29 +02:00
Michael Eischer
3fd0ad7448
repository: list index files only once
2023-10-01 19:53:26 +02:00
arjunajesh
ed65a7dbca
implement progress bar for index loading
2023-10-01 19:52:59 +02:00
Lorenz Bausch
2657217574
Tweak restore summary
2023-09-20 10:59:55 +02:00
Michael Eischer
6e586b64e4
Merge pull request #4450 from MichaelEischer/cleanup-master-index-test
...
master_index: fix inconsistent length blob length in test
2023-09-06 22:55:48 +02:00
greatroar
691c01963b
internal/restic: Return summary from BlobSet.String
...
Fixes #4449 : error messages from prune were too long to fit in scroll
buffers.
2023-08-25 21:41:30 +02:00
Michael Eischer
2bec99dc6f
master_index: fix inconsistent length blob length in test
...
Two blobs with the same hash must always have the same content length.
2023-08-19 20:04:25 +02:00
Michael Eischer
feb6abb7bb
Merge pull request #4394 from greatroar/acl
...
dump: Rewrite Linux ACL handling
2023-08-13 17:01:31 +00:00
greatroar
aaf5254e26
dump: Rewrite Linux ACL handling
...
The old version was taken from an MPL-licensed library. This is a
cleanroom implementation. The code is shorter and it's now explicit that
only Linux ACLs are supported.
2023-08-13 13:37:11 +02:00
Michael Eischer
4a33370072
backend: Fix unreliable TestListCancel/Timeout test
...
The test uses `WithTimeout` to create a context that cancels the List
operation after a given delay. Several backends internally use a derived
child context created using WithCancel.
The cancellation of a context first closes the done channel of the
context (here: the `WithTimeout` context) and _afterwards_ propagates
the cancellation to child contexts (here: the `WithCancel` context).
Therefor if the List implementation uses a child context, then it may
take a moment until that context is also cancelled. Thus give the
context cancellation a moment to propagate.
2023-07-23 11:47:56 +02:00
Michael Eischer
f12bbd9229
restic: check that Node.LinkTarget can handle non-utf8 characters
2023-07-23 00:05:14 +02:00
Michael Eischer
6adb629608
Add support for non-utf8 symlink targets
2023-07-23 00:05:14 +02:00
Michael Eischer
25ff9fa893
Merge pull request #4334 from MichaelEischer/snapshot-subtree-syntax
...
Add support for snapshot:path syntax
2023-07-22 23:59:26 +02:00
Michael Eischer
3f63b53090
lock: Shrink critical section in RefreshStaleLock
...
A cleaning up after a failed lock refresh attempt does not require a
mutex.
2023-07-22 23:29:14 +02:00
Michael Eischer
96eada3d5f
backend/sema: add test for freeze functionality
2023-07-22 23:23:17 +02:00
Michael Eischer
8154f6a77a
Ensure consistent naming for <snapshot>:<subfolder>
syntax
2023-07-22 19:55:57 +02:00
Michael Eischer
090f9d6237
restic: Cleanup and simplify TestCreateSnapshot
2023-07-22 19:55:57 +02:00
Michael Eischer
321cc35cde
restic: add test for FindTreeDirectory
2023-07-22 19:55:57 +02:00
Michael Eischer
a97915642c
restic: Ensure snapshots created by TestCreateSnapshot are valid
...
All nodes in a tree must be sorted by node name. Otherwise functionality
like searching for a specific tree node will break.
2023-07-22 19:55:57 +02:00
Michael Eischer
c64d81063e
test snapshot filtering with snapshot:path syntax
2023-07-22 19:55:57 +02:00
Michael Eischer
85860e6e97
Add support for snapshot:subpath syntax
...
This snapshot specification syntax is supported by the cat, diff, dump,
ls and restore command.
2023-07-22 19:55:57 +02:00
Michael Eischer
5d9b0d894e
lock: add unit test for RefreshStaleLock
2023-07-22 19:55:44 +02:00
Michael Eischer
d4bf7a3cb1
restic: cleanup lock test
2023-07-22 19:55:44 +02:00
Michael Eischer
24c8a33da9
restic: reduce sleeps in lock tests
2023-07-22 19:55:44 +02:00
Michael Eischer
f490288738
lock: freeze backend operations while refreshing stale lock
...
Freeze new backend operations while trying to refresh a stale lock.
2023-07-22 19:55:44 +02:00
Michael Eischer
51718ec561
restic: generic implemention of AsBackend
2023-07-22 19:55:44 +02:00
Michael Eischer
bee3231ed4
lock: try refreshing of stale locks
...
A stale lock may be refreshed if it continues to exist until after a
replacement lock has been created. This ensures that a repository was
not unlocked in the meantime.
2023-07-22 19:55:44 +02:00
Michael Eischer
978ebaac49
rest: use http status code constants
2023-07-14 22:39:38 +02:00
Michael Eischer
307aeb6849
rest: Don't return error if listing non-existent directory
...
When transferring a repository from S3 to, for example, a local disk
then all empty folders will be missing.
When saving files, the missing intermediate folders are created
automatically. Therefore, missing directories can be ignored by the
`List()` operation.
2023-07-14 22:39:38 +02:00
Michael Eischer
e990d3d483
azure: Support authentication using managed / workload identity
...
See https://github.com/Azure/azure-sdk-for-go/tree/sdk/azidentity/v1.3.0/sdk/azidentity
2023-07-09 14:47:03 +02:00
Michael Eischer
9d44682e3e
fuse: fix windows build
2023-07-08 22:40:22 +02:00
Michael Eischer
4d43509423
Merge pull request #4379 from chenxiaolong/symlink_xattrs
...
Add support for extended attributes on symlinks
2023-07-08 16:56:51 +00:00
Michael Eischer
ea9ad77e05
restic: refactor node test
2023-07-08 18:54:56 +02:00
Michael Eischer
cc84884d2e
restic: basic xattr test for files/dirs
2023-07-08 18:49:21 +02:00
Michael Eischer
4a5ae2ba84
restic: test NodeFromFileInfo for symlinks
2023-07-08 18:18:13 +02:00
Michael Eischer
1f1e50f49e
fuse: add test for symlink xattr
2023-07-08 18:02:17 +02:00
Michael Eischer
f3c3b0f377
fuse: deduplicate xattr code
2023-07-08 17:41:45 +02:00
Michael Eischer
bbac74b172
add program version to snapshot
2023-07-07 23:27:10 +02:00
Michael Eischer
a280b7364e
Merge pull request #4395 from greatroar/parsebytes
...
cmd, ui: Move size parsing code and make it more robust
2023-07-07 21:24:47 +00:00
Michael Eischer
e36d17a6f8
azure: tweak documentation for endpoint suffix
2023-07-07 23:10:26 +02:00
arjunajesh
068b115abc
added azure domain parameter
2023-07-07 23:10:26 +02:00
greatroar
41a5bf357f
cmd, ui: Move size parsing code and make it more robust
2023-07-02 21:44:50 +02:00
Andrew Gunnerson
8c02ebb029
Add support for extended attributes on symlinks
...
Linux allows the use of non-`user.` extended attributes on symlinks. One
of the main users of this functionality is SELinux's `security.selinux`
xattr for storing a path's label. By storing symlink xattrs, restic is
now suitable for backing up the root filesystem on Linux distributions
that use SELinux.
This commit adds support for symlink xattrs when backing up data,
restoring data, and mounting snapshots via a fuse mount. All calls to
the xattr library have been updated to the use `L` variants of the
various functions, which always operate on the path given, without
following symlinks.
Fixes : #4375
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
2023-06-19 14:37:31 -04:00
Michael Eischer
74ca82a6f8
Merge pull request #4362 from restic/unified-backend-factory
...
Unified backend initialization
2023-06-17 15:52:37 +02:00
Michael Eischer
93038ed8f4
s3: restore retries for minio tests
2023-06-17 15:25:08 +02:00
Gautam Menghani
8da5a6649b
Preserve oldest snapshot when keep-within* does not collect enough
2023-06-17 15:17:20 +02:00
Michael Eischer
1257c2c075
forget: Add comments to snapshot policy
2023-06-17 15:16:58 +02:00