Commit graph

1522 commits

Author SHA1 Message Date
Michael Eischer
648edeca40 retry: Do not retry Stat() if file does not exist
In non test/debug code, Stat() is used exclusively to check whether a
file exists. Thus, do not retry if a file is reported as not existing.
2022-12-03 11:42:48 +01:00
Michael Eischer
40ac678252 backend: remove Test method
The Test method was only used in exactly one place, namely when trying
to create a new repository it was used to check whether a config file
already exists.

Use a combination of Stat() and IsNotExist() instead.
2022-12-03 11:28:10 +01:00
Michael Eischer
57d8eedb88
Merge pull request #4020 from greatroar/fuse-inode
fuse: Better inode generation
2022-12-02 22:28:15 +01:00
Michael Eischer
ca1803cacb
Merge pull request #4063 from MichaelEischer/replace-ioutil-usage
Replace ioutil usage
2022-12-02 21:49:40 +01:00
Michael Eischer
fa20a78bb6
Merge pull request #4056 from greatroar/cleanup
backend, fs, options: Minor cleanup
2022-12-02 19:44:54 +01:00
Michael Eischer
ff7ef5007e Replace most usages of ioutil with the underlying function
The ioutil functions are deprecated since Go 1.17 and only wrap another
library function. Thus directly call the underlying function.

This commit only mechanically replaces the function calls.
2022-12-02 19:36:43 +01:00
greatroar
65612d797c backend, options: Prefer strings.Cut to SplitN
Also realigned the various "split into host🪣prefix"
implementations.
2022-12-02 19:19:14 +01:00
greatroar
60aa87bbab fs: Remove explicit type check in extendedStat
Without comma-ok, the runtime inserts the same check with a similar
enough panic message:

    interface conversion: interface {} is nil, not *syscall.Stat_t
2022-11-27 19:58:06 +01:00
greatroar
e5d597fd22 bloblru: Upgrade to hashicorp/golang-lru/v2
The new genericized LRU cache no longer needs to have the IDs separately
allocated:

name   old time/op    new time/op    delta
Add-8     494ns ± 2%     388ns ± 2%  -21.46%  (p=0.000 n=10+9)

name   old alloc/op   new alloc/op   delta
Add-8      176B ± 0%      152B ± 0%  -13.64%  (p=0.000 n=10+10)

name   old allocs/op  new allocs/op  delta
Add-8      5.00 ± 0%      3.00 ± 0%  -40.00%  (p=0.000 n=10+10)
2022-11-27 17:18:13 +01:00
greatroar
c9c7671c58 fuse: Clean up inode generation 2022-11-27 13:53:42 +01:00
Michael Eischer
530f129a39 rest: remove workaround for content-length handling bug 2022-11-27 13:18:44 +01:00
Michael Eischer
a1eb923876 remove no longer necessary conditional compiles 2022-11-27 13:18:44 +01:00
Michael Eischer
9113b2620f
Merge pull request #4024 from MichaelEischer/macos-fuse
mount: switch to anacrolix fork of bazil/fuse
2022-11-26 12:15:14 +01:00
Michael Eischer
f115d64634
Merge pull request #4022 from MichaelEischer/race-checker
CI: Run the golang race checker
2022-11-26 12:13:50 +01:00
greatroar
189e0fe5a9 fuse: Better inode generation
Hard links to the same file now get the same inode within the FUSE
mount. Also, inode generation is faster and, more importantly, no longer
allocates.

Benchmarked on Linux/amd64. Old means the benchmark with

        sink = fs.GenerateDynamicInode(1, sub.node.Name)

instead of calling inodeFromNode. Results:

name                   old time/op    new time/op    delta
Inode/no_hard_links-8     137ns ± 4%      34ns ± 1%   -75.20%  (p=0.000 n=10+10)
Inode/hard_link-8        33.6ns ± 1%     9.5ns ± 0%   -71.82%  (p=0.000 n=9+8)

name                   old alloc/op   new alloc/op   delta
Inode/no_hard_links-8     48.0B ± 0%      0.0B       -100.00%  (p=0.000 n=10+10)
Inode/hard_link-8         0.00B          0.00B           ~     (all equal)

name                   old allocs/op  new allocs/op  delta
Inode/no_hard_links-8      1.00 ± 0%      0.00       -100.00%  (p=0.000 n=10+10)
Inode/hard_link-8          0.00           0.00           ~     (all equal)
2022-11-16 08:35:01 +01:00
Michael Eischer
32ffcd86a2
Merge pull request #3993 from MichaelEischer/backup-json-full-snapshot-id
backup: print full snapshot id in JSON summary
2022-11-12 20:42:35 +01:00
Michael Eischer
4b5234924b
Merge pull request #2875 from fgma/issue2699
issue2699: restore symlinks on windows when run as admin user
2022-11-12 20:06:45 +01:00
Michael Eischer
f88acd4503 rewrite: Fail if a tree contains an unknown field
In principle, the JSON format of Tree objects is extensible without
requiring a format change. In order to not loose information just play
it safe and reject rewriting trees for which we could loose data.
2022-11-12 19:55:22 +01:00
Michael Eischer
0224e276ec walker: Add tests for FilterTree 2022-11-12 19:55:22 +01:00
Michael Eischer
ad14d6e4ac rewrite: use SelectByName like in the backup command 2022-11-12 19:55:22 +01:00
Michael Eischer
f6339b88af rewrite: extract tree filtering 2022-11-12 19:55:22 +01:00
Michael Eischer
220eaee76b mount: switch to anacrolix fork of bazil/fuse
The anacrolix fork contains the latest changes from bazil/fuse and
additionally provides support for recent versions of macFUSE.
2022-11-12 19:22:31 +01:00
Michael Eischer
7b4fe7bad5
Merge pull request #4021 from greatroar/mac-fsync
backend/local: Ignore ENOTTY for fsync on Mac
2022-11-11 23:10:37 +01:00
greatroar
348e966daa backend/local: Ignore ENOTTY for fsync on Mac
Fixes #4016.
2022-11-11 22:51:51 +01:00
Michael Eischer
13fbc96ed3 lock: Synchronize Refresh() and Stale()
The lock test creates a lock and checks that it is not stale. However,
it is possible that the lock is refreshed concurrently, which updates
the lock timestamp. Checking the timestamp in `Stale()` without
synchronization results in a data race. Thus add a lock to prevent
concurrent accesses.
2022-11-11 21:52:53 +01:00
Michael Eischer
e1ba7ab684 lock: Don't copy the lock when checking for process existence
The lock test creates a lock and checks that it is not stale. This also
tests whether the corresponding process still exists. However, it is
possible that the lock is refreshed concurrently, which updates the lock
timestamp. Calling `processExists()` with a value receiver, however,
creates an unsynchronized copy of this field. Thus call the method using
a pointer receiver.
2022-11-11 21:45:55 +01:00
Michael Eischer
5756c96c9f archiver: Fix race condition resulting in files containing null IDs
In some rare cases files could be created which contain null IDs (all
zero) in their content list. This was caused by a race condition between
growing the `Content` slice and inserting the blob IDs into it. In some
cases the blob ID was written to the old slice, which a short time
afterwards was replaced with a larger copy, that did not yet contain the
blob ID.
2022-11-10 20:19:37 +01:00
Michael Eischer
b1d1202b1d archiver: Check that saved file does not have null IDs in content
Null IDs in the file content indicate that something went wrong. Thus
fails before saving the affected file.
2022-11-08 22:57:41 +01:00
Michael Eischer
403390479c
Merge pull request #3997 from greatroar/fuse-hash
fuse: Better check for whether snapshots changed
2022-11-05 10:52:11 +01:00
greatroar
c091e43b33 fuse: Better check for whether snapshots changed
We previously checked whether the set of snapshots might have changed
based only on their number, which fails when as many snapshots are
forgotten as are added. Check for the SHA-256 of their id's instead.
2022-11-05 09:32:45 +01:00
Alexander Neumann
8dd95b710e
Merge pull request #3992 from MichaelEischer/err-on-invalid-compression
Return error if RESTIC_COMPRESSION env variable is invalid
2022-11-04 19:41:34 +01:00
Michael Eischer
9354262b1b backup: fix stuck status bar
The status bar got stuck once the first error was reported, the scanner
completed or some file was backed up. Either case sets a flag that the
scanner has started.

This flag is used to hide the progress bar until the flag is set. Due to
an inverted condition, the opposite happened and the status stopped
refreshing once the flag was set.

In addition, the scannerStarted flag was not set when the scanner just
reported progress information.
2022-11-02 21:31:13 +01:00
Michael Eischer
06141ce1f4 backup: print full snapshot id in JSON summary 2022-10-31 19:03:42 +01:00
Michael Eischer
59a90943bb
Merge pull request #3983 from greatroar/formatting
Centralize and fix formatting of bytes, percentages, durations
2022-10-31 18:52:24 +01:00
greatroar
5ab3e6276a ui: Fix FormatBytes at exactly 1024 time a unit
1024 would be displayed as "1024 bytes" instead of "1.000 KiB", etc.
2022-10-31 18:39:28 +01:00
Michael Eischer
8fe159cc5a enable ysmlink tests for windows 2022-10-30 18:43:04 +01:00
Michael Eischer
144257f8bd restore symlink timestamps on windows 2022-10-30 11:04:04 +01:00
Michael Eischer
c0f34af9db backup: hide files from status which are read completely but not saved
As the FileSaver is asynchronously waiting for all blobs of a file to be
stored, the number of active files is higher than the number of files
from which restic is reading concurrently. Thus to not confuse users,
only display files in the status from which restic is currently reading.
2022-10-30 10:29:12 +01:00
Michael Eischer
b52a8ff05c ui: Properly clear lines no longer used for status
Previously, the old status text remained until it was overwritten.
2022-10-30 10:29:12 +01:00
Michael Eischer
b4de902596 archiver: Asynchronously complete FutureFile
After reading and chunking all data in a file, the FutureFile still has
to wait until the FutureBlobs are completed. This was done synchronously
which results in blocking the file saver and prevents the next file from
being read.

By replacing the FutureBlob with a callback, it becomes possible to
complete the FutureFile asynchronously.
2022-10-30 10:29:11 +01:00
Michael Eischer
c7ace314f6
Merge pull request #3989 from greatroar/eachbypack
More compact data structure for Index.EachByPack
2022-10-30 00:02:55 +02:00
greatroar
0e8893dae9 index: Compact data structure for Index.EachByPack 2022-10-29 23:09:17 +02:00
greatroar
137f0bc944 repository: Fix benchmarkSaveAndEncrypt 2022-10-29 23:09:17 +02:00
Michael Eischer
01f0db4e56 return error if RESTIC_COMPRESSION env variable is invalid 2022-10-29 22:03:39 +02:00
Michael Eischer
7c87fb941c
Merge pull request #3986 from greatroar/counter
ui/progress: Load both values in a single Lock/Unlock
2022-10-29 21:50:55 +02:00
Michael Eischer
3b0bb02a68
Merge pull request #3977 from greatroar/progress
ui/backup: Replace channels with a mutex
2022-10-29 21:33:04 +02:00
Michael Eischer
0d260cfd82 enable symlink test on windows 2022-10-29 21:26:34 +02:00
fgma
8e5eb1090c issue2699: restore symlinks on windows when run as admin user 2022-10-29 21:19:33 +02:00
greatroar
2dafda9164 ui/progress: Load both values in a single Lock/Unlock
We always need both values, except in a test, so we don't need to lock
twice and risk scheduling in between.

Also, removed the resetting in Done. This copied a mutex, which isn't
allowed. Static analyzers tend to trip over that.
2022-10-25 07:55:24 +02:00
greatroar
006380199e cmd, ui: Deduplicate formatting utilities 2022-10-23 13:40:07 +02:00