Commit graph

7041 commits

Author SHA1 Message Date
Michael Eischer
726a1969cd
Merge pull request #2731 from dionorgua/rewrite-snapshot
Implement 'rewrite' command to exclude files from existing snapshots
2022-11-12 20:06:35 +01:00
Michael Eischer
bb0fa76c06 Cleanup exclude pattern collection 2022-11-12 19:55:22 +01:00
Michael Eischer
537cfe2e4c rewrite: Fix check that an exclude pattern was passed
The old check did not consider files containing case insensitive
excludes. The check is now implemented as a function of the
excludePatternOptions struct to improve cohesion.
2022-11-12 19:55:22 +01:00
Leo R. Lundgren
f175da2756 rewrite: Polish documentation 2022-11-12 19:55:22 +01:00
Leo R. Lundgren
f86ef4d3dd rewrite: Polish code and add missing messages 2022-11-12 19:55:22 +01:00
Leo R. Lundgren
c15bedccc0 rewrite: Revert unrelated documentation change 2022-11-12 19:55:22 +01:00
Michael Eischer
f88acd4503 rewrite: Fail if a tree contains an unknown field
In principle, the JSON format of Tree objects is extensible without
requiring a format change. In order to not loose information just play
it safe and reject rewriting trees for which we could loose data.
2022-11-12 19:55:22 +01:00
Michael Eischer
11b8c3a158 rewrite: add documentation 2022-11-12 19:55:22 +01:00
Michael Eischer
ec0c91e233 rewrite: Add tests for further ways to use the command 2022-11-12 19:55:22 +01:00
Michael Eischer
0224e276ec walker: Add tests for FilterTree 2022-11-12 19:55:22 +01:00
Michael Eischer
73f54cc5ea rewrite: rename --inplace to --forget 2022-11-12 19:55:22 +01:00
Michael Eischer
a47d9a1c40 rewrite: use unified snapshot filter options 2022-11-12 19:55:22 +01:00
Michael Eischer
b044649118 rewrite: add minimal test 2022-11-12 19:55:22 +01:00
Michael Eischer
375a3db64d rewrite: non-exclusive lock if snapshots are only added 2022-11-12 19:55:22 +01:00
Michael Eischer
327f418a9c rewrite: cleanup err handling and output 2022-11-12 19:55:22 +01:00
Michael Eischer
ad14d6e4ac rewrite: use SelectByName like in the backup command 2022-11-12 19:55:22 +01:00
Michael Eischer
7ebaf6e899 rewrite: start repository uploader goroutines 2022-11-12 19:55:22 +01:00
Michael Eischer
559acea0d8 unify exclude pattern options 2022-11-12 19:55:22 +01:00
Michael Eischer
4cace1ffe9 unify exclude patterns with backup command 2022-11-12 19:55:22 +01:00
Michael Eischer
2b69a1c53b rewrite: filter all snapshots if none are specified 2022-11-12 19:55:22 +01:00
Michael Eischer
f6339b88af rewrite: extract tree filtering 2022-11-12 19:55:22 +01:00
Michael Eischer
c0f7ba2388 rewrite: simplify dryrun 2022-11-12 19:55:22 +01:00
Michael Eischer
4d6ab83019 rewrite: use treejsonbuilder 2022-11-12 19:55:22 +01:00
Michael Eischer
82592b88b5 rewrite: address most review comments 2022-11-12 19:55:22 +01:00
Michael Eischer
b922774343 rewrite: fix compilation 2022-11-12 19:55:22 +01:00
Dmitry Nezhevenko
dc29709742 Implement 'rewrite' command to exclude files from existing snapshots 2022-11-12 19:55:22 +01:00
Michael Eischer
220eaee76b mount: switch to anacrolix fork of bazil/fuse
The anacrolix fork contains the latest changes from bazil/fuse and
additionally provides support for recent versions of macFUSE.
2022-11-12 19:22:31 +01:00
Michael Eischer
6fa45d0d39
Merge pull request #4011 from greatroar/backup-stdin-password
cmd: Don't read password from stdin for backup --stdin
2022-11-12 19:18:56 +01:00
Michael Eischer
bbd180ae21
Merge pull request #4017 from Rajpratik71/Rajpratik71-patch-1
feat: dependabot workflow automation for updating dependency
2022-11-12 15:48:48 +01:00
Pratik Raj
bef1064b8e
chore: ignore upgrade for 'bazil/fuse' and 'golang.org/x/oauth2' 2022-11-12 19:39:16 +05:30
Michael Eischer
7b4fe7bad5
Merge pull request #4021 from greatroar/mac-fsync
backend/local: Ignore ENOTTY for fsync on Mac
2022-11-11 23:10:37 +01:00
greatroar
348e966daa backend/local: Ignore ENOTTY for fsync on Mac
Fixes #4016.
2022-11-11 22:51:51 +01:00
Michael Eischer
0e5fe4c6ab CI: run golang race checker 2022-11-11 22:15:22 +01:00
Michael Eischer
13fbc96ed3 lock: Synchronize Refresh() and Stale()
The lock test creates a lock and checks that it is not stale. However,
it is possible that the lock is refreshed concurrently, which updates
the lock timestamp. Checking the timestamp in `Stale()` without
synchronization results in a data race. Thus add a lock to prevent
concurrent accesses.
2022-11-11 21:52:53 +01:00
Michael Eischer
e1ba7ab684 lock: Don't copy the lock when checking for process existence
The lock test creates a lock and checks that it is not stale. This also
tests whether the corresponding process still exists. However, it is
possible that the lock is refreshed concurrently, which updates the lock
timestamp. Calling `processExists()` with a value receiver, however,
creates an unsynchronized copy of this field. Thus call the method using
a pointer receiver.
2022-11-11 21:45:55 +01:00
Michael Eischer
dc060356c2 mount: only start next test after mount command cleanup is complete
The test did not wait for the mount command to fully shutdown all
running goroutines. This caused the go race detector to report a data
race related to lock refreshes.

==================
WARNING: DATA RACE
Write at 0x0000021bdfdb by goroutine 667:
  github.com/restic/restic/internal/backend/retry.TestFastRetries()
      /restic/restic/internal/backend/retry/testing.go:7 +0x18f
  github.com/restic/restic/cmd/restic.withTestEnvironment()
      /restic/restic/cmd/restic/integration_helpers_test.go:175 +0x183
  github.com/restic/restic/cmd/restic.TestMountSameTimestamps()
      /restic/restic/cmd/restic/integration_fuse_test.go:202 +0xac
  testing.tRunner()
      /usr/lib/go/src/testing/testing.go:1446 +0x216
  testing.(*T).Run.func1()
      /usr/lib/go/src/testing/testing.go:1493 +0x47

Previous read at 0x0000021bdfdb by goroutine 609:
  github.com/restic/restic/internal/backend/retry.(*Backend).retry()
      /restic/restic/internal/backend/retry/backend_retry.go:72 +0x9e
  github.com/restic/restic/internal/backend/retry.(*Backend).Remove()
      /restic/restic/internal/backend/retry/backend_retry.go:149 +0x17d
  github.com/restic/restic/internal/cache.(*Backend).Remove()
      /restic/restic/internal/cache/backend.go:38 +0x11d
  github.com/restic/restic/internal/restic.(*Lock).Unlock()
      /restic/restic/internal/restic/lock.go:190 +0x249
  github.com/restic/restic/cmd/restic.refreshLocks.func1()
      /restic/restic/cmd/restic/lock.go:86 +0xae
  runtime.deferreturn()
      /usr/lib/go/src/runtime/panic.go:476 +0x32
  github.com/restic/restic/cmd/restic.lockRepository.func2()
      /restic/restic/cmd/restic/lock.go:61 +0x71

[...]

Goroutine 609 (finished) created at:
  github.com/restic/restic/cmd/restic.lockRepository()
      /restic/restic/cmd/restic/lock.go:61 +0x488
  github.com/restic/restic/cmd/restic.lockRepo()
      /restic/restic/cmd/restic/lock.go:25 +0x219
  github.com/restic/restic/cmd/restic.runMount()
      /restic/restic/cmd/restic/cmd_mount.go:126 +0x1f8
  github.com/restic/restic/cmd/restic.testRunMount()
      /restic/restic/cmd/restic/integration_fuse_test.go:61 +0x1ce
  github.com/restic/restic/cmd/restic.checkSnapshots.func1()
      /restic/restic/cmd/restic/integration_fuse_test.go:90 +0x124
==================
2022-11-11 21:43:01 +01:00
Michael Eischer
32c9667990
Merge pull request #4019 from MichaelEischer/fix-file-saver-race
archiver: Fix race condition resulting in files containing null IDs
2022-11-11 20:52:33 +01:00
Michael Eischer
d268552a0a
Merge pull request #4014 from MichaelEischer/fix-debug-examine
debug: fix crash in `debug examine --reupload-blobs`
2022-11-10 20:37:32 +01:00
Michael Eischer
5756c96c9f archiver: Fix race condition resulting in files containing null IDs
In some rare cases files could be created which contain null IDs (all
zero) in their content list. This was caused by a race condition between
growing the `Content` slice and inserting the blob IDs into it. In some
cases the blob ID was written to the old slice, which a short time
afterwards was replaced with a larger copy, that did not yet contain the
blob ID.
2022-11-10 20:19:37 +01:00
Pratik Raj
df614fff26
feat: dependabot workflow automation for updating dependency
Signed-off-by: Pratik Raj <Rajpratik71@gmail.com>
2022-11-10 16:02:03 +05:30
Michael Eischer
11a4bb051e debug: fix crash in debug examine --reupload-blobs 2022-11-09 22:13:17 +01:00
Michael Eischer
5f9ac2b165
Merge pull request #4010 from MichaelEischer/file-saver-sanity-check
archiver: Check that saved file does not have null IDs in content
2022-11-08 23:07:32 +01:00
Michael Eischer
b1d1202b1d archiver: Check that saved file does not have null IDs in content
Null IDs in the file content indicate that something went wrong. Thus
fails before saving the affected file.
2022-11-08 22:57:41 +01:00
greatroar
5dceadeb72 cmd: Don't read password from stdin for backup --stdin 2022-11-06 14:55:57 +01:00
Michael Eischer
1ccab95bc4 b2: Support file hiding instead of deleting them permanently
Automatically fall back to hiding files if not authorized to permanently
delete files. This allows using restic with an append-only application
key with B2.  Thus, an attacker cannot directly delete backups with the
API key used by restic.

To use this feature create an application key without the deleteFiles
capability. It is recommended to restrict the key to just one bucket.
For example using the b2 command line tool:

    b2 create-key --bucket <bucketName> <keyName> listBuckets,readFiles,writeFiles,listFiles

Suggested-by: Daniel Gröber <dxld@darkboxed.org>
2022-11-05 20:10:45 +01:00
Michael Eischer
24a2e5cab9
Merge pull request #4008 from MichaelEischer/tweak-lock-refresh-test
lock: Tweak timeouts for lock refresh test
2022-11-05 10:53:13 +01:00
Michael Eischer
403390479c
Merge pull request #3997 from greatroar/fuse-hash
fuse: Better check for whether snapshots changed
2022-11-05 10:52:11 +01:00
Michael Eischer
d29abc1a31
Merge pull request #4007 from MichaelEischer/hide-compression-level-for-v1-repo
Only print compression level starting from repository version 2
2022-11-05 10:33:25 +01:00
greatroar
c091e43b33 fuse: Better check for whether snapshots changed
We previously checked whether the set of snapshots might have changed
based only on their number, which fails when as many snapshots are
forgotten as are added. Check for the SHA-256 of their id's instead.
2022-11-05 09:32:45 +01:00
Michael Eischer
aaac63da8d lock: Tweak timeouts for lock refresh test
For some reason the test fails from time to time. Increase the timeouts
to hopefully avoid this issue.
2022-11-04 22:48:18 +01:00