Commit graph

1127 commits

Author SHA1 Message Date
MichaelEischer
fb4c5af5c4
Merge pull request #3642 from gco/master
Fix test failures on Solaris
2022-02-12 22:07:14 +01:00
MichaelEischer
cc90f2ba6b
Merge pull request #2816 from greatroar/noatime
Set O_NOATIME flag on Linux
2022-02-07 21:38:31 +01:00
Michael Eischer
6b17a7110c backup: Set O_NOATIME in the right place
The archiver uses FS.OpenFile, where FS is an instance of the FS
interface. This is different from fs.OpenFile, which uses the OpenFile
method provided by the fs package.
2022-02-06 15:00:37 +01:00
greatroar
7080fed7ae Set O_NOATIME flag on Linux
Citing Kerrisk, The Linux Programming Interface:

    The O_NOATIME flag is intended for use by indexing and backup
    programs. Its use can significantly reduce the amount of disk
    activity, because repeated disk seeks back and forth across the
    disk are not required to read the contents of a file and to update
    the last access time in the file’s i-node[.]

restic used to do this, but the functionality was removed along with the
fadvise call in #670.
2022-02-06 15:00:34 +01:00
Michael Eischer
8ae4d86a84 rename snapshot timestamp filter variable 2022-02-05 22:42:38 +01:00
Greg
2e9180638e Fix test failures on Solaris
Add exceptions for symlinks, sticky bits, and device nodes in the same places where the BSDSs and/or Darwin have them.
2022-01-25 18:05:56 -08:00
Aneesh Agrawal
058dfc20da Avoid choosing parent snapshot newer than time of current snapshot
Currently, `restic backup` (if a `--parent` is not provided)
will choose the most recent matching snapshot as the parent snapshot.
This makes sense in the usual case,
where we tag the snapshot-being-created with the current time.

However, this doesn't make sense if the user has passed `--time`
and is currently creating a snapshot older than the latest snapshot.
Instead, choose the most recent snapshot
which is not newer than the snapshot-being-created's timestamp,
to avoid any time travel.

Impetus for this change:
I'm using restic for the first time!
I have a number of existing BTRFS snapshots
I am backing up via restic to serve as my initial set of backups.
I initially `restic backup`'d the most recent snapshot to test,
then started backing up each of the other snapshots.
I noticed in `restic cat snapshot <id>` output
that all the remaining snapshots have the most recent as the parent.
2022-01-23 23:55:00 -05:00
rawtaz
7f6fc78f95
Merge pull request #3544 from restic/fix-b2-delete-retry
b2: Successful delete if file does not exist
2022-01-13 23:28:30 +01:00
Charlie Jiang
a5b0e0bef4 fix: rclone receiving SIGINT prematurely on Windows causing restic hang forever
Co-authored-by: greatroar <61184462+greatroar@users.noreply.github.com>
2021-12-28 13:14:46 +08:00
Michael Eischer
50053a85d3 rest: Adjust http2 missing eof test to golang >= 1.17.3, >= 1.16.10
The missing eof with http2 when a response included a content-length
header but no data, has been fixed in golang 1.17.3/1.16.10. Therefore
just drop the canary test and schedule it for removal once go 1.18 is
required as minimum version by restic.
2021-11-13 21:57:30 +01:00
MichaelEischer
f1cfb97237
Merge pull request #3514 from phcreery/rclone_timeout
rclone: extend timeout from 60s to 240s
2021-11-07 18:10:34 +01:00
MichaelEischer
cb81ee9396
Merge pull request #3474 from kitone/fix-issue-3382
Honor RESTIC_CACHE_DIR environment variable
2021-11-07 17:57:54 +01:00
Michael Eischer
b0e64deb27 rclone: Fix timeout calculation 2021-11-07 17:49:33 +01:00
phcreery
43d173b042 rclone: add timeout option and documentation 2021-11-07 17:49:21 +01:00
Michael Eischer
15d7313387 local: Fix fd leak when encountering files directly inside data/ 2021-11-06 19:44:57 +01:00
MichaelEischer
78c7dd53ef
Merge pull request #3526 from greatroar/dump-refactor
Refactor internal/dump + concurrent load/write
2021-11-05 22:38:39 +01:00
kitone
1b23675f21 cache --cleanup should handle directories created by restic check.
Because there is no guarantee that a cleanup of these directories will occur
after the "restic check", we extend the behavior to detect and manage these
specific cache directories and allow their cleanup too.
2021-11-04 15:10:38 +01:00
greatroar
c71729dfc4 Refactor internal/dump + concurrent load/write
Package internal/dump has been reworked so its API consists of a single
type Dumper that handles tar and zip formats. Tree loading and node
writing happen concurrently.
2021-11-01 23:01:55 +01:00
Alexander Neumann
fb5d9345a7
Merge pull request #3510 from MichaelEischer/fix-archiver-early-on-abort
archiver: Fix TestArchiverAbortEarlyOnError test
2021-10-16 15:37:41 +02:00
kitone
95eb859b54 Honor RESTIC_CACHE_DIR environment variable
Fix #3382: restic check doesn't obey the RESTIC_CACHE_DIR environment variable
2021-10-10 16:00:02 +02:00
Michael Eischer
257740b0cc b2: Successful delete if file does not exist
When deleting a file, B2 sometimes returns a "500 Service Unavailable"
error but nevertheless correctly deletes the file. Due to retries in
the B2 library blazer, we sometimes also see a "400 File not present"
error. The retries of restic for the delete request then fail with
"404 File with such name does not exist.".

As we have to rely on request retries in a distributed system to handle
temporary errors, also consider a delete request to be successful if the
file is reported as not existing. This should be safe as B2 claims to
provide a strongly consistent bucket listing and thus a missing file
shouldn't mysteriously show up again later on.
2021-10-09 23:51:12 +02:00
MichaelEischer
f2a3b3b4a1
Merge pull request #3537 from greatroar/dump-lru
Check cap instead of len in bloblru
2021-10-09 20:00:27 +02:00
Garry McNulty
708d7a2574 s3: Add warning if key ID or secret is empty
Also add debug message if no credential types are available.

Closes #2388
2021-10-06 23:13:40 +01:00
greatroar
634a9c162d Check cap instead of len in bloblru
restic dump uses bloblru.Cache to keep buffers alive, but also reuses
evicted buffers. That means large buffers may be used to store small
blobs, causing the cache to think it's using less memory than it
actually does.
2021-10-03 09:34:17 +02:00
greatroar
c892c0bab9 internal/restic: Don't allocate in Tree.Insert
name         old time/op    new time/op    delta
BuildTree-8    34.6µs ± 4%     7.0µs ± 3%  -79.68%  (p=0.000 n=18+19)

name         old alloc/op   new alloc/op   delta
BuildTree-8    34.0kB ± 0%     0.9kB ± 0%  -97.37%  (p=0.000 n=20+20)

name         old allocs/op  new allocs/op  delta
BuildTree-8       108 ± 0%         1 ± 0%  -99.07%  (p=0.000 n=20+20)
2021-09-26 18:08:48 +02:00
MichaelEischer
78dac2fd48
Merge pull request #3523 from greatroar/swift-v2
Upgrade ncw/swift to v2
2021-09-24 21:41:36 +02:00
Michael Eischer
5ea8bba1a1 swift: restore context err check for list() 2021-09-24 21:19:46 +02:00
MichaelEischer
a5e103a212
Merge pull request #3522 from greatroar/dump-lru
Use LRU cache in restic dump
2021-09-24 20:33:58 +02:00
greatroar
e7ec0453b1 Upgrade ncw/swift to v2 2021-09-24 19:08:37 +02:00
greatroar
fe04d024c7 Use LRU cache in restic dump 2021-09-24 15:45:08 +02:00
Uli Martens
718966a81a Move Blobcache into dedicated internal package 2021-09-24 15:45:00 +02:00
greatroar
4f33eca634 Remove unused Writer arg to internal/dump.writeDump 2021-09-24 15:40:42 +02:00
Sam Lucidi
897d8e662c Add --insecure-tls flag to disable SSL cert verification
Signed-off-by: Sam Lucidi <slucidi@redhat.com>
2021-09-21 10:52:40 -04:00
MichaelEischer
1827b16ade
Merge pull request #3519 from greatroar/maphash
Replace siphash by hash/maphash
2021-09-19 19:46:03 +02:00
greatroar
8d2996eaaa Replace siphash by hash/maphash
In Go 1.17.1, maphash has become quite a bit faster than siphash, so we
can drop one third-party dependency. maphash is just an interface to the
standard Go map's hash function, which we already trust for other use
cases.

Benchmark results on linux/amd64, -benchtime=3s:

name                                             old time/op    new time/op    delta
IndexHasUnknown-8                                  50.6ns ±10%    41.0ns ±19%  -18.92%  (p=0.000 n=9+10)
IndexHasKnown-8                                    52.6ns ±12%    41.5ns ±12%  -21.13%  (p=0.000 n=9+10)
IndexMapHash-8                                     3.64µs ± 1%    2.00µs ± 0%  -45.09%  (p=0.000 n=10+9)
IndexAlloc-8                                        700ms ± 1%     601ms ± 6%  -14.18%  (p=0.000 n=8+10)
IndexAllocParallel-8                                205ms ± 5%     192ms ± 8%   -6.18%  (p=0.043 n=10+10)
MasterIndexAlloc-8                                  319ms ± 1%     279ms ± 5%  -12.58%  (p=0.000 n=10+10)
MasterIndexLookupSingleIndex-8                      156ns ± 8%     147ns ± 6%   -5.46%  (p=0.023 n=10+10)
MasterIndexLookupMultipleIndex-8                    150ns ± 7%     142ns ± 8%   -5.69%  (p=0.007 n=10+10)
MasterIndexLookupSingleIndexUnknown-8              74.4ns ± 6%    72.0ns ± 9%     ~     (p=0.175 n=10+9)
MasterIndexLookupMultipleIndexUnknown-8            67.4ns ± 9%    65.5ns ± 7%     ~     (p=0.340 n=9+9)
MasterIndexLookupParallel/known,indices=25-8        461ns ± 2%     445ns ± 2%   -3.49%  (p=0.000 n=10+10)
MasterIndexLookupParallel/unknown,indices=25-8      408ns ±11%     378ns ± 5%   -7.22%  (p=0.035 n=10+9)
MasterIndexLookupParallel/known,indices=50-8        479ns ± 1%     437ns ± 4%   -8.82%  (p=0.000 n=10+10)
MasterIndexLookupParallel/unknown,indices=50-8      406ns ± 8%     343ns ±15%  -15.44%  (p=0.001 n=10+10)
MasterIndexLookupParallel/known,indices=100-8       480ns ± 1%     455ns ± 5%   -5.15%  (p=0.000 n=8+10)
MasterIndexLookupParallel/unknown,indices=100-8     391ns ±18%     382ns ± 8%     ~     (p=0.315 n=10+10)
MasterIndexLookupBlobSize-8                        71.0ns ± 8%    57.2ns ±11%  -19.36%  (p=0.000 n=9+10)
PackerManager-8                                     254ms ± 1%     254ms ± 1%     ~     (p=0.285 n=15+15)

name                                             old speed      new speed      delta
IndexMapHash-8                                   1.12GB/s ± 1%  2.05GB/s ± 0%  +82.13%  (p=0.000 n=10+9)
PackerManager-8                                   208MB/s ± 1%   207MB/s ± 1%     ~     (p=0.281 n=15+15)

name                                             old alloc/op   new alloc/op   delta
IndexMapHash-8                                      0.00B          0.00B          ~     (all equal)
IndexAlloc-8                                        400MB ± 0%     400MB ± 0%     ~     (p=1.000 n=9+10)
IndexAllocParallel-8                                401MB ± 0%     401MB ± 0%   +0.00%  (p=0.000 n=10+10)
MasterIndexAlloc-8                                  258MB ± 0%     262MB ± 0%   +1.42%  (p=0.000 n=9+10)
PackerManager-8                                    73.1kB ± 0%    73.1kB ± 0%     ~     (p=0.382 n=13+13)

name                                             old allocs/op  new allocs/op  delta
IndexMapHash-8                                       0.00           0.00          ~     (all equal)
IndexAlloc-8                                         907k ± 0%      907k ± 0%   -0.00%  (p=0.000 n=10+10)
IndexAllocParallel-8                                 907k ± 0%      907k ± 0%   +0.00%  (p=0.009 n=10+10)
MasterIndexAlloc-8                                   327k ± 0%      317k ± 0%   -3.06%  (p=0.000 n=10+10)
PackerManager-8                                       744 ± 0%       744 ± 0%     ~     (all equal)
2021-09-19 16:05:18 +02:00
MichaelEischer
58efe21eca
Merge pull request #3264 from amozoss/upstream-master
Refactor backup progress
2021-09-19 14:54:42 +02:00
Michael Eischer
921e328b56 restore: Fix linting error 2021-09-19 14:41:07 +02:00
Michael Eischer
2cdc0719af restorer: Sanitize verify errors 2021-09-19 14:01:26 +02:00
greatroar
bdcdfaf6b4 restore --verify: buffer reuse consistency and comment 2021-09-19 13:11:27 +02:00
greatroar
2b94742ca5 Replace no-op closures in restorer by nil check 2021-09-19 13:11:16 +02:00
greatroar
d357744104 Handle canceled context in restore --verify properly 2021-09-19 13:11:05 +02:00
greatroar
d4225ec803 Simplify buffer growing in Restorer.verifyFile
Suggested-by: Igor Fedorenko <igor@ifedorenko.com>
2021-09-19 13:10:55 +02:00
greatroar
de8521ae56 Report number of successes from Restorer.VerifyFiles
Previously, the number of attempts was reported.
2021-09-19 13:10:44 +02:00
greatroar
bb066cf7d3 Concurrent Restorer.VerifyFiles
Time to verify a 2GB snapshot down from 9.726s to 4.645s (-52%).
2021-09-19 13:10:41 +02:00
greatroar
556424d61b Reuse buffer in Restorer.VerifyFiles
Time to verify a 2GB snapshot down from 11.568s to 9.726s (-16%).
2021-09-19 12:55:31 +02:00
greatroar
973fa921cb Test and document Restorer.VerifyFiles 2021-09-19 12:52:11 +02:00
Michael Eischer
e0d615c264 archiver: Fix TestArchiverAbortEarlyOnError test
This can be caused when the test has uploaded four blobs, then queues
two blobs for upload which are delayed. Then a seventh file can be
opened which lead to a test failure.
2021-09-12 22:17:17 +02:00
Michael Eischer
fbb0e6499a ui: consolidate backup ui in ui/backup package 2021-09-12 16:20:15 +02:00
Michael Eischer
cccb0d4064 ui: assert that ProgressPrinter interface is implemented 2021-09-12 15:26:40 +02:00
Michael Eischer
d62bfed65d ui: move SetDryRun to ProgressReporter 2021-09-12 15:25:58 +02:00