TrueCloudLab/restic
16
0
Fork 2
Code Issues 1 Pull requests 1 Projects Releases Packages Wiki Activity Actions
7359 commits 9 branches 49 tags 79 MiB
Commit graph

1567 commits

Author SHA1 Message Date
Michael Eischer
b2fcbc21cb lock: rework stale lock refresh to avoid data race 2023-07-22 19:55:44 +02:00
Michael Eischer
05e5e29a8c backup: don't pretend to lock repo for dry run 
The dry run did not actually create lock files, but still ran all other
parts of the lock processing.
 2023-07-22 19:55:44 +02:00
Michael Eischer
f490288738 lock: freeze backend operations while refreshing stale lock 
Freeze new backend operations while trying to refresh a stale lock.
 2023-07-22 19:55:44 +02:00
Michael Eischer
11eb88a2ea lock: cleanup 2023-07-22 19:55:44 +02:00
Michael Eischer
bee3231ed4 lock: try refreshing of stale locks 
A stale lock may be refreshed if it continues to exist until after a
replacement lock has been created. This ensures that a repository was
not unlocked in the meantime.
 2023-07-22 19:55:44 +02:00
Michael Eischer
47206a6579
Merge pull request #4351 from kenny-y-dev/human-readable-ls 
Add --human-readable to ls and find output
 2023-07-16 11:44:41 +00:00
Michael Eischer
89fbd39e59 Don't print stacktrace on invalid cacert option 2023-07-09 14:23:54 +02:00
Michael Eischer
1ce599d2ae Fix handling of empty cacert environment variable 
This resulted in a "empty filename for root certificate supplied" error.
 2023-07-09 14:23:54 +02:00
Michael Eischer
325fa916b5 stats: Add debug mode to collect repository statistics 2023-07-08 23:04:47 +02:00
Michael Eischer
cc3c218baf small cleanups for certificate environment variables 2023-07-08 09:44:59 +02:00
arjunajesh
c1578a2035 certificates can be passed through env vars 2023-07-08 09:44:59 +02:00
Michael Eischer
389f6ee74c backup: add minimal test for program versioni 2023-07-07 23:33:23 +02:00
Michael Eischer
bbac74b172 add program version to snapshot 2023-07-07 23:27:10 +02:00
greatroar
41a5bf357f cmd, ui: Move size parsing code and make it more robust 2023-07-02 21:44:50 +02:00
Michael Eischer
cbf87fbdb3 init: don't include password in debug log 2023-06-17 15:15:58 +02:00
Michael Eischer
50e0d5e6b5 backend: Hardcode backend scheme in Factory 
Our ParseConfig implementations always expect a specific scheme, thus no
other scheme would work.
 2023-06-17 15:15:58 +02:00
Michael Eischer
3a3cf608f5 b2/s3: Move config validation from ApplyEnvironment to Open/Create 
Conceptually the backend configuration should be validated when creating
or opening the backend, but not when filling in information from
environment variables into the configuration.
 2023-06-17 15:15:58 +02:00
Michael Eischer
9aa9e0d1ec local/sftp: move limiter setup into backend 2023-06-17 15:15:58 +02:00
Michael Eischer
7d12c29286 backend: Unify backend construction using factory and registry 
This unified construction removes most backend-specific code from
global.go. The backend registry will also enable integration tests to
use custom backends if necessary.
 2023-06-17 15:15:57 +02:00
Michael Eischer
56836364a4 backend: pass context into every backend constructor 2023-06-17 15:15:57 +02:00
Michael Eischer
4df77e9f26
Merge pull request #4314 from MichaelEischer/restore-json 
Add JSON support for the restore progress bar
 2023-06-16 23:53:17 +02:00
Michael Eischer
191c47d30e
Merge pull request #4353 from MichaelEischer/tune-gc 
Tune Go garbage collector
 2023-06-16 23:24:39 +02:00
Michael Eischer
cf16239058 restore: print output via termStatus 2023-06-09 12:03:33 +02:00
Michael Eischer
d54176ce5d restore: add basic json progress 2023-06-09 12:03:33 +02:00
Michael Eischer
a9aff885d6 restore: reorganize progress bar code 
The structure is now much more similar to that of the backup command.
 2023-06-09 12:03:33 +02:00
Michael Eischer
bb20078641 restore: pass termStatus to restore in tests 2023-06-09 12:03:33 +02:00
Michael Eischer
9464c63550 Make formatNode test timezone independent 
formatNode formats the timestamp according to the current time zone. Pin
the local timezone to UTC to ensure the test works everywhere.
 2023-06-08 19:18:30 +02:00
Michael Eischer
6ebf2dd235 Reduce duplicate code in test for fomatNode 2023-06-08 19:16:16 +02:00
Michael Eischer
5f153109ba Refactor formatNode 2023-06-08 19:12:49 +02:00
Michael Eischer
32a6b66267 backend: add standardized Config.ApplyEnvironment 
This removes the backend specific special cases while parsing the
configuration in `global.go`.
 2023-06-07 22:31:15 +02:00
Michael Eischer
f903db492c backend: let ParseConfig return a Config pointer 
In order to change the backend initialization in `global.go` to be able
to generically call cfg.ApplyEnvironment() for supported backends, the
`interface{}` returned by `ParseConfig` must contain a pointer to the
configuration.

An alternative would be to use reflection to convert the type from
`interface{}(Config)` to `interface{}(*Config)` (from value to pointer
type). However, this would just complicate the type mess further.
 2023-06-07 22:31:15 +02:00
Michael Eischer
2f7b4ceae1 backend: Move environment based configuration into backend 2023-06-07 22:30:35 +02:00
Michael Eischer
49a6a4f5bf
Merge pull request #4356 from MichaelEischer/fix-unlock-hint 
lock: fix missing hint to unlock command if repository is locked
 2023-06-06 21:03:06 +02:00
Michael Eischer
237d00000e lock: fix missing hint to unlock command if repository is locked 2023-06-02 23:16:49 +02:00
Michael Eischer
2fcb3947df prune: trigger GC after prune planning 2023-06-02 21:57:40 +02:00
Michael Eischer
f1b73c9301 Reduce GOGC to 50 
The index used by restic consumes a major part of the total memory
usage. This leads to an unnecessarily large amount of memory that
contains ephemeral objects that are only used for a short time.
 2023-06-02 21:51:50 +02:00
Kenny Y
8812dcd56a
Add --human-readable to ls and find output 
Modifies format module to add options for human readable storage size formatting, using size parsing already in ui/format.
Cmd flag --human-readable added to ls and find commands.
Additional option added to formatNode to support printing size in regular or new human readable format
 2023-06-01 21:18:18 -04:00
Michael Eischer
88a10a368f
Merge pull request #4342 from greatroar/errors 
cmd: Don't check for errno == 0
 2023-05-31 22:17:07 +02:00
greatroar
8f1ca8fabe cmd: Don't check for errno == 0 
Maybe term.Restore's predecessor could return this, but the current
implementation never does.
 2023-05-27 08:52:37 +02:00
Michael Eischer
9747cef338 fix linter warnings 2023-05-26 19:38:44 +02:00
Michael Eischer
8e913e6d3a repair index: always read Quiet flags from GlobalOptions passed as parameter 2023-05-26 19:38:44 +02:00
Michael Eischer
b93459cbb0 repair snapshots: use local copy of globalOptions to open repository 2023-05-26 19:38:44 +02:00
Michael Eischer
692f81ede8 cleanup prune integration test 2023-05-26 19:38:44 +02:00
Michael Eischer
7a268e4aba always access stdout/stderr via globalOptions 2023-05-26 19:38:44 +02:00
Michael Eischer
4b3a0b4104 read JSON/Quiet flag from the passed in globalOptions 2023-05-26 19:38:44 +02:00
Michael Eischer
cebce52c16 test: add helper to capture stdout for integration tests 2023-05-26 19:38:44 +02:00
Michael Eischer
675a49a95b Restructure integration tests 
The tests are now split into individual files for each command. The
separation isn't perfect as many tests make use of multiple commands. In
particular `init`, `backup`, `check` and `list` are used by a larger
number of test cases.

Most tests now reside in files name cmd_<name>_integration_test.go. This
provides a certain indication which commands have significant test
coverage.
 2023-05-26 19:38:43 +02:00
Michael Eischer
e2dba9f5c7 test: cleanup a some check calls 2023-05-26 19:34:57 +02:00
Michael Eischer
06fd6b54d7 test: print log output if testRunCheck fails 2023-05-26 19:34:57 +02:00
Michael Eischer
419e6f26b1 tests: Simplify checks that a specific number of snapshots exists 2023-05-26 19:34:57 +02:00
Michael Eischer
c3212ab6a6 test: use standard logging methods from testing for the test helpers 
Use the logging methods from testing.TB to make use of tb.Helper(). This
allows the tests to log the filename and line number in which the test
helper was called. Previously the test helper was logged which is rarely
useful.
 2023-05-26 19:34:57 +02:00
Michael Eischer
a466e945d9 stats: pass StatsOptions via parameter 2023-05-18 21:17:53 +02:00
Michael Eischer
03b9764bce init/generate: check that no parameters are passed 2023-05-18 21:17:53 +02:00
Michael Eischer
1e33b285c1 lock: remove unused parameter 2023-05-18 21:17:53 +02:00
Michael Eischer
c05f96e6b9 repair index: remove no longer used paramter 2023-05-18 21:17:53 +02:00
Michael Eischer
94752b7ee2 backup: Remove unused parameters from rejectFunc collection 2023-05-18 21:17:53 +02:00
Michael Eischer
472bf5184f Replace lots of unused parameters with _ 
The parameters are required by the implemented function signature or interface.
 2023-05-18 21:17:53 +02:00
Michael Eischer
1514593f22 Remove unused context or testing parameters 2023-05-18 21:17:53 +02:00
Michael Eischer
5e4e268bdc Use _ as parameter name for unused Context 
The context is required by the implemented interface.
 2023-05-18 21:15:45 +02:00
Michael Eischer
319087c056 Remove redundant if ...; err != nil; return pattern 2023-05-18 21:15:45 +02:00
Michael Eischer
e01baeabba Use either test or rtest to refer to internal test helpers 
A single test file should not use both names.
 2023-05-18 21:15:45 +02:00
Michael Eischer
5773b86d02 repository: Push all usage of errors.Fatal out of the package 
As the `Fatal` error type only includes a string, it becomes impossible
to inspect the contained error. This is for a example a problem for the
fuse implementation, which must be able to detect context.Canceled
errors.

Co-authored-by: greatroar <61184462+greatroar@users.noreply.github.com>
 2023-05-18 17:27:41 +02:00
Michael Eischer
ee3c55ea3d
Merge pull request #2876 from aawsome/new-repair-command 
Add repair command
 2023-05-05 23:22:24 +02:00
Michael Eischer
78e5aa6d30 repair snapshots: add basic tests 2023-05-04 23:00:46 +02:00
Michael Eischer
1cb11ad8ad mount: enable debug logging for the flaky TestMount test 
The test case fails from time to time with an Input/Output error while
trying to access the snapshots directory.
 2023-05-01 18:03:17 +02:00
Michael Eischer
5aa37acdaa repair snapshots: cleanup command help 2023-05-01 16:06:17 +02:00
Michael Eischer
7c8dd61e8c repair snapshots: cleanup warnings 2023-05-01 15:22:30 +02:00
Michael Eischer
f6cc10578d repair snapshots: Always sanitize file nodes 
If the node for a file is intact, this is a no-op.
 2023-05-01 15:22:18 +02:00
Michael Eischer
4ce87a7f64 repair snapshots: port to filterAndReplaceSnapshot 
The previous approach of rewriting all snapshots first, then flushing
the repository data and finally removing old snapshots has the downside
that an interrupted command execution leaves behind broken snapshots as
not all new data is already flushed.
 2023-05-01 15:22:03 +02:00
Michael Eischer
e17ee40a31 repair snapshots: Port to use walker.TreeRewriter 2023-05-01 15:20:36 +02:00
Michael Eischer
1bd1f3008d walker: extend TreeRewriter to support snapshot repairing 
This adds support for caching already rewritten trees, handling of load
errors and disabling the check that the serialization doesn't lead to
data loss.
 2023-05-01 15:20:24 +02:00
Michael Eischer
38dac78180 walker: restructure FilterTree into TreeRewriter 
The more generic RewriteNode callback replaces the SelectByName and
PrintExclude functions. The main part of this change is a preparation to
allow using the TreeRewriter for the `repair snapshots` command.
 2023-05-01 15:20:12 +02:00
Michael Eischer
8c4caf09a8 repair snapshots: Do not rename repaired files 
The files in a tree must be sorted in lexical order. However, this
cannot be guaranteed when appending a filename suffix. For two files

file, file.rep

where "file" is broken, this would result in

file.repaired, file.rep

which is no longer sorted.

In addition, adding a filename suffix is also prone to filename
collisions which would require a rather complex search for a
collision-free name in order to work reliably.
 2023-05-01 15:19:36 +02:00
Michael Eischer
375189488c rewrite: prepare for code sharing with rewrite snapshots 2023-05-01 15:19:24 +02:00
Michael Eischer
903651c719 repair snapshots: partially synchronize code with rewrite command 
Simplify CLI options:
* Rename "DeleteSnapshots" to "Forget"
* Replace "AddTag" and "Append" with hardcoded values

Change output and snapshot modifications to be more in line with the
"rewrite" command.
 2023-05-01 15:19:11 +02:00
Michael Eischer
118d599d0a Rename 'rebuild-index' to 'repair index' 
The old name still works, but is deprecated.
 2023-05-01 15:16:44 +02:00
Michael Eischer
db459eda21 move to subcommand 2023-05-01 15:15:48 +02:00
Michael Eischer
a14a63cd29 modernize code 2023-05-01 15:12:09 +02:00
Alexander Weiss
947f0c345e correct typos 2023-05-01 14:56:42 +02:00
Alexander Weiss
d23a2e1925 better error handling and correct nil tree behavior 2023-05-01 14:56:42 +02:00
Alexander Weiss
08ae708b3b make linter happy 2023-05-01 14:56:42 +02:00
Alexander Weiss
5f58797ba7 Add repair command 2023-05-01 14:56:42 +02:00
Michael Eischer
8d971172c4
Merge pull request #4306 from MichaelEischer/document-restic-compression-env 
Document that the compression mode can be set via $RESTIC_COMPRESSION
 2023-04-30 18:26:10 +02:00
Michael Eischer
face5bd7f7 Document that the compression mode can be set via $RESTIC_COMPRESSION 2023-04-30 16:11:53 +02:00
Michael Eischer
1daf928a77
Merge pull request #4305 from MichaelEischer/stracktrace-for-windows 
Print stacktrace in SIGINT handler if RESTIC_DEBUG_STACKTRACE_SIGINT set
 2023-04-30 16:08:58 +02:00
Michael Eischer
c8641f4479 Merge branch 'patch-release' 2023-04-24 20:44:38 +02:00
Alexander Neumann
ac7ac0cb97 Set development version for 0.15.2 2023-04-24 20:28:37 +02:00
Alexander Neumann
be8be3397c Add version for 0.15.2 2023-04-24 20:28:37 +02:00
Michael Eischer
306a29980a Print stacktrace in SIGINT handler if RESTIC_DEBUG_STACKTRACE_SIGINT set 
The builtin mechanism to capture a stacktrace in Go is to send a SIGQUIT
to the running process. However, this mechanism is not avaiable on
Windows. Thus, tweak the SIGINT handler to dump a stacktrace if the
environment variable `RESTIC_DEBUG_STACKTRACE_SIGINT` is set.
 2023-04-23 15:50:40 +02:00
Michael Eischer
22562d2132
Merge pull request #4300 from MichaelEischer/less-flaky-tests 
Increase timeouts for lock refresh tests
 2023-04-23 11:24:59 +02:00
Michael Eischer
51d823348d
Merge pull request #4286 from MichaelEischer/backend-cleanup-logging 
Normalize backend logging and connection limiting
 2023-04-23 11:24:26 +02:00
Michael Eischer
179e11c2ae Increase timeouts for lock refresh tests 
When saving files to the local backend, in some cases the used fsync
calls are slow enough to cause the tests to time out. Thus, increase the
test timeouts as a stopgap measure until we can use the mem backend for
these tests.
 2023-04-22 12:45:59 +02:00
Michael Eischer
ebba233a3a backend/sema: rename constructor to NewBackend 2023-04-22 12:32:57 +02:00
Michael Eischer
1dd873b706
Merge pull request #4293 from MichaelEischer/fix-groups-by 
Fix parent snapshot selection for relative paths
 2023-04-21 22:47:03 +02:00
Michael Eischer
7a60d9e54f
Merge pull request #4288 from MichaelEischer/log-warnings-to-debug-log 
Add warnings via Warnf to the debug log
 2023-04-21 22:39:00 +02:00
Michael Eischer
3001dd8c2b Add test to verify that the backup parent is correctly selected 2023-04-21 22:35:02 +02:00
Michael Eischer
09cddb8927 rewrite: log snapshot saved before removal of the old snapshot 
The snapshot was already saved before removing the old snapshot. Only
the log messages were printed in the wrong order.
 2023-04-17 21:00:45 +02:00
Michael Eischer
913eab3361
Merge pull request #4234 from thndrbrrr/forget-opts-neg1-means-forever-issue-2565 
restic forget --keep-* options will interpret -1 as "forever"
 2023-04-14 23:18:47 +02:00
Michael Eischer
a9c7c12276
Merge pull request #4265 from HeikoSchlittermann/chg/progress-to-stderr 
generate: do not write progress to STDOUT
 2023-04-14 23:18:23 +02:00
Michael Eischer
85eef232e6
Merge pull request #4232 from thndrbrrr/copy-chunker-params-verbose-msg 
init: Add --copy-chunker-params verbose msg
 2023-04-14 23:17:56 +02:00