TrueCloudLab/restic
16
0
Fork 2
Code Issues 1 Pull requests Projects Releases Packages Wiki Activity Actions
4740 commits 9 branches 49 tags 79 MiB
Commit graph

133 commits

Author SHA1 Message Date
Michael Eischer
f6f11400c2 restic: Remove unused variable value 2020-09-05 10:07:16 +02:00
Michael Eischer
e2dc5034d3 restic/file: Use FileType for all File constants 
The data type of a variable is only carried on when using iota.
 2020-09-05 10:07:16 +02:00
Michael Eischer
37113282ca Merge a few variable declaration and initializations 2020-09-05 10:05:34 +02:00
Michael Eischer
337725c354 Use sort.Strings 2020-09-05 10:05:34 +02:00
aawsome
0fed6a8dfc
Use "pack file" instead of "data file" (#2885) 
- changed variable names, especially changed DataFile into PackFile
- changed in some comments
- always use "pack file" in docu
 2020-08-16 11:16:38 +02:00
Michael Eischer
af66a62c04 FindUsedBlobs: Test that seen blobs are skipped 
This also copies the TreeLoader interface from internal/walker to allow
stubbing the repository in the call to `FindUsedBlobs`.
 2020-08-01 12:29:16 +02:00
Michael Eischer
9ea1a78bd4 FindUsedBlobs: Check for seen blobs before loading trees 
The only effective change in behavior is that that toplevel nodes can
also be skipped.
 2020-08-01 12:29:16 +02:00
Michael Eischer
184103647a FindUsedBlobs: merge seen into blobs BlobSet 
The seen BlobSet always contained a subset of the entries in blobs.
Thus use blobs instead and avoid the memory overhead of the second set.

Suggested-by: Alexander Weiss <alex@weissfam.de>
 2020-08-01 12:29:16 +02:00
Michael Eischer
c847aace35 Rename Index interface to MasterIndex 
The interface is now only implemented by repository.MasterIndex.
 2020-07-25 21:19:46 +02:00
Alexander Weiss
9d1fb94c6c make Lookup() return all blobs 
+ simplify syntax
 2020-07-25 21:18:34 +02:00
rawtaz
5d8d70542f
Merge pull request #2852 from MichaelEischer/drop-go-1.11 
Drop support for Go version 1.11
 2020-07-25 01:17:36 +02:00
Michael Eischer
7c23381a2b Drop support for Go version 1.11 2020-07-24 18:52:39 +02:00
MichaelEischer
34181b13a2
Merge pull request #2328 from MichaelEischer/no-repeated-checks 
Fix duplicate tree checks within `restic check`
 2020-07-22 22:08:02 +02:00
Michael Eischer
ddf0b8cd0b checker: Properly distinguish between data and tree blobs 
If a data blob and a tree blob with the same ID (= same content) exist,
then the checker did not report a data or tree blob as unused when the
blob of the other type was still in use.
 2020-07-20 22:58:39 +02:00
greatroar
7bda28f31f Chaining hash table for repository.Index 
These are faster to construct but slower to access. The allocation rate
is halved, the peak memory usage almost halved compared to standard map.

Benchmark results on linux/amd64, -benchtime=3s -count=20:

name                                            old time/op    new time/op    delta
PackerManager-8                                    178ms ± 0%     178ms ± 0%     ~     (p=0.231 n=20+20)
DecodeIndex-8                                      4.54s ± 0%     4.30s ± 0%   -5.20%  (p=0.000 n=18+17)
DecodeIndexParallel-8                              4.54s ± 0%     4.30s ± 0%   -5.22%  (p=0.000 n=19+18)
IndexHasUnknown-8                                 44.4ns ± 5%    50.5ns ±11%  +13.82%  (p=0.000 n=19+17)
IndexHasKnown-8                                   48.3ns ± 0%    51.5ns ±12%   +6.68%  (p=0.001 n=16+20)
IndexAlloc-8                                       758ms ± 1%     616ms ± 1%  -18.69%  (p=0.000 n=19+19)
IndexAllocParallel-8                               234ms ± 3%     204ms ± 2%  -12.60%  (p=0.000 n=20+18)
MasterIndexLookupSingleIndex-8                     122ns ± 0%     145ns ± 9%  +18.44%  (p=0.000 n=14+20)
MasterIndexLookupMultipleIndex-8                   369ns ± 2%     429ns ± 8%  +16.27%  (p=0.000 n=20+20)
MasterIndexLookupSingleIndexUnknown-8             68.4ns ± 5%    74.9ns ±13%   +9.47%  (p=0.000 n=20+20)
MasterIndexLookupMultipleIndexUnknown-8            315ns ± 3%     369ns ±11%  +17.14%  (p=0.000 n=20+20)
MasterIndexLookupParallel/known,indices=5-8        743ns ± 1%     816ns ± 2%   +9.87%  (p=0.000 n=17+17)
MasterIndexLookupParallel/unknown,indices=5-8      238ns ± 1%     260ns ± 2%   +9.14%  (p=0.000 n=19+20)
MasterIndexLookupParallel/known,indices=10-8      1.01µs ± 3%    1.11µs ± 2%   +9.79%  (p=0.000 n=19+20)
MasterIndexLookupParallel/unknown,indices=10-8     222ns ± 0%     269ns ± 2%  +20.83%  (p=0.000 n=16+20)
MasterIndexLookupParallel/known,indices=20-8      1.06µs ± 2%    1.19µs ± 2%  +12.95%  (p=0.000 n=19+18)
MasterIndexLookupParallel/unknown,indices=20-8     413ns ± 1%     530ns ± 1%  +28.19%  (p=0.000 n=18+20)
SaveAndEncrypt-8                                  30.2ms ± 1%    30.4ms ± 0%   +0.71%  (p=0.000 n=19+19)
LoadTree-8                                         540µs ± 1%     576µs ± 1%   +6.73%  (p=0.000 n=20+20)
LoadBlob-8                                        5.64ms ± 0%    5.64ms ± 0%     ~     (p=0.883 n=18+17)
LoadAndDecrypt-8                                  5.93ms ± 0%    5.95ms ± 1%     ~     (p=0.247 n=20+19)
LoadIndex-8                                       25.1ms ± 0%    24.5ms ± 1%   -2.54%  (p=0.000 n=18+17)

name                                            old speed      new speed      delta
PackerManager-8                                  296MB/s ± 0%   296MB/s ± 0%     ~     (p=0.229 n=20+20)
SaveAndEncrypt-8                                 139MB/s ± 1%   138MB/s ± 0%   -0.71%  (p=0.000 n=19+19)
LoadBlob-8                                       177MB/s ± 0%   177MB/s ± 0%     ~     (p=0.890 n=18+17)
LoadAndDecrypt-8                                 169MB/s ± 0%   168MB/s ± 1%     ~     (p=0.227 n=20+19)

name                                            old alloc/op   new alloc/op   delta
PackerManager-8                                   91.8kB ± 0%    91.8kB ± 0%     ~     (p=0.772 n=12+19)
IndexAlloc-8                                       786MB ± 0%     400MB ± 0%  -49.04%  (p=0.000 n=20+18)
IndexAllocParallel-8                               786MB ± 0%     401MB ± 0%  -49.04%  (p=0.000 n=19+15)
SaveAndEncrypt-8                                  21.0MB ± 0%    21.0MB ± 0%   +0.00%  (p=0.000 n=19+19)

name                                            old allocs/op  new allocs/op  delta
PackerManager-8                                    1.41k ± 0%     1.41k ± 0%     ~     (all equal)
IndexAlloc-8                                        977k ± 0%      907k ± 0%   -7.18%  (p=0.000 n=20+20)
IndexAllocParallel-8                                977k ± 0%      907k ± 0%   -7.17%  (p=0.000 n=19+15)
SaveAndEncrypt-8                                    73.0 ± 0%      73.0 ± 0%     ~     (all equal)
 2020-07-19 13:58:22 +02:00
MichaelEischer
dd7b4f54f5
Merge pull request #2709 from greatroar/minio-sha256 
Use Minio's optimized SHA-256
 2020-06-12 23:32:58 +02:00
Alexander Weiss
91906911b0 Fix non-intuitive repository behavior 
- The SaveBlob method now checks for duplicates.
- Moves handling of pending blobs to MasterIndex.
  -> also cleans up pending index entries when they are saved in the index
  -> when using SaveBlob no need to care about index any longer
- Always check for full index and save it when storing packs.
  -> removes the need of an index uploader
  -> also removes the verbose "uploaded intermediate index" messages
- The Flush method now also saves the index
- Fix race condition when checking and saving full/non-finalized indexes
 2020-06-11 13:05:23 +02:00
greatroar
42a3db05b0 Use Minio's optimized SHA-256 
internal/repository benchmarks on an Intel i7-3770k:

name               old speed      new speed       delta
PackerManager-8     209MB/s ± 1%    291MB/s ± 1%  +38.94%  (p=0.008 n=5+5)
SaveAndEncrypt-8    112MB/s ± 1%    135MB/s ± 1%  +20.25%  (p=0.008 n=5+5)
 2020-04-28 07:57:18 +02:00
greatroar
be5a0ff59f Centralize buffer allocation and size checking in Repository.LoadBlob 
Benchmark results for internal/repository:

name              old time/op    new time/op    delta
LoadTree-8           479µs ± 2%     478µs ± 1%   ~     (p=0.780 n=10+9)
LoadBlob-8          11.6ms ± 2%    11.6ms ± 1%   ~     (p=0.631 n=10+10)
LoadAndDecrypt-8    13.2ms ± 2%    13.3ms ± 3%   ~     (p=0.631 n=10+10)

name              old alloc/op   new alloc/op   delta
LoadTree-8          41.2kB ± 0%    41.2kB ± 0%   ~     (all equal)
LoadBlob-8          2.28kB ± 0%    2.28kB ± 0%   ~     (all equal)
LoadAndDecrypt-8    2.10MB ± 0%    2.10MB ± 0%   ~     (all equal)

name              old allocs/op  new allocs/op  delta
LoadTree-8             652 ± 0%       652 ± 0%   ~     (all equal)
LoadBlob-8            24.0 ± 0%      24.0 ± 0%   ~     (all equal)
LoadAndDecrypt-8      30.0 ± 0%      30.0 ± 0%   ~     (all equal)

name              old speed      new speed      delta
LoadBlob-8        86.2MB/s ± 2%  86.4MB/s ± 1%   ~     (p=0.594 n=10+10)
LoadAndDecrypt-8  75.7MB/s ± 2%  75.4MB/s ± 3%   ~     (p=0.617 n=10+10)
 2020-04-23 10:04:20 +02:00
greatroar
2e31120f89 Remove unused argument to restic.fakeFile 2020-04-18 17:40:13 +02:00
MichaelEischer
b7c0d4d8bf
Merge pull request #2644 from greatroar/signal-notify-buffered 
Make all signal.Notify channels buffered
 2020-04-18 11:31:02 +02:00
rawtaz
9efbe98879
Merge pull request #2623 from alrs/internal-restic-err-before-close 
internal/restic: close os.File after checking for error
 2020-03-18 22:23:20 +01:00
rawtaz
7d9300efca
Merge pull request #2637 from greatroar/unused 
Remove Go 1.5 compatibility code
 2020-03-18 22:22:15 +01:00
greatroar
47d4d5bf1b Make all signal.Notify channels buffered 2020-03-12 20:59:39 +01:00
greatroar
4de12bf593 Remove restic.RandReader 
math/rand.Rand has implemented Reader since Go 1.6. The repacking tests
are not deterministic, but they weren't before, either.
 2020-03-09 10:00:28 +01:00
Lars Lehtonen
4a2156d3f0
internal/restic: close os.File after checking for error 2020-03-05 16:22:46 -08:00
rawtaz
58bd165253
Merge pull request #2581 from aawsome/multiple-hostnames 
Allow multiple hostnames tags
 2020-02-27 08:35:23 +01:00
Alexander Weiss
9a9101d144 Support specifying multiple host flags for various commands 
The `dump`, `find`, `forget`, `ls`, `mount`, `restore`, `snapshots`,
`stats` and `tag` commands will now take into account multiple
`--host` and `-H` flags.
 2020-02-26 22:17:59 +01:00
Lars Lehtonen
16b321b140
internal/restic: fix dropped test error 2020-02-12 13:32:45 -08:00
Ward Vandewege
69a0d0ee90
fix typos 2019-12-21 21:00:28 -05:00
rawtaz
da4473aba6
Merge pull request #2391 from kastenhq/refresh-lock-time 
Update Lock.Time in lock.Refresh()
 2019-11-26 21:44:29 +01:00
Leo R. Lundgren
4d2aa18273 Accept both singular and plural for --group-by (#2330) 2019-11-20 19:25:48 +01:00
Eric Hamilton
1596d06f8e Update Lock.Time in lock.Refresh() 2019-09-04 11:38:35 -07:00
Alexandr Bruyako
38ea7ed4f6 remove unused code 2019-07-01 00:24:45 +03:00
Alexander Neumann
355db0bc29 windows: Use LastWriteTime for ctime and mtime 
Windows does not have a concept of a `change time` in the sense as Unix
has it: the field `CreationTime` of the `Win32FileAttributeData` struct
is not updated when attributes or content is changed. So from now on
we're using the `LastWriteTime` as the `change time` on Windows.
 2019-05-05 14:02:11 +02:00
Alexander Neumann
939f3e972c node: Make sure year of all timestamps is valid 
Sometimes restic gets bogus timestamps which cannot be converted to
JSON, because the stdlib JSON encoder returns an error if the year is
not within [0, 9999]. We now make sure that we at least record _some_
timestamp and cap the year either to 0000 or 9999. Before, restic would
refuse to save the file at all, so this improves the status quo.

This fixes #2174 and #1173
 2019-04-24 16:39:36 +02:00
Alexander Neumann
18ec27a0da Merge pull request #2243 from treymo/ls-length-2104 
backend find: Check prefix length < snapshot IDs when searching
 2019-04-23 09:41:19 +02:00
Jan Niklas Richter
733519d895 Refactor duplicate code for grouping snapshots 
This commit is a followup to the addition of the --group-by flag for the
snapshots command. Adding the grouping code there introduced duplicated
code (the forget command also does grouping). This commit refactors
boths sides to only use shared code.
 2019-04-22 14:57:53 +02:00
Trey Moore
5ac9c1157a Prevent "slice bounds out of range" error if prefix is longer than snapshot IDs. 
This includes tests as well as other tests for the backend_find.Find function. Issue #2104.
 2019-04-16 20:26:20 -07:00
Alexander Neumann
d51e9d1b98 Add []byte to repo.LoadAndDecrypt and utils.LoadAll 
This commit changes the signatures for repository.LoadAndDecrypt and
utils.LoadAll to allow passing in a []byte as the buffer to use. This
buffer is enlarged as needed, and returned back to the caller for
further use.

In later commits, this allows reducing allocations by reusing a buffer
for multiple calls, e.g. in a worker function.
 2019-04-13 13:38:39 +02:00
Alexander Neumann
203d775190 restic: Make JSON unmarshal for ID more efficient 
This commit reduces several allocations in UnmarshalJSON() by decoding
the hex string directly in a single step.
 2019-04-13 13:38:39 +02:00
Alexander Neumann
c1058005c3 Fix build on FreeBSD for Go < 1.12 2019-03-16 20:52:37 +01:00
Dmitri Goutnik
1d8b21cdad Fix build on FreeBSD with Go 1.12 2019-03-06 09:41:49 -05:00
plumbeo
71891b340c Support time ranges expressed in hours in snapshot retention policies 
Make restic forget --keep-within accept time ranges measured in hours and choose
accordingly which snapshots to keep and which to forget. Add relative tests.
 2018-11-26 14:27:42 +01:00
plumbeo
7486bfea5b Extend custom Duration granularity to hours and add tests 2018-11-14 17:27:02 +01:00
Alexander Neumann
3ae2a79bdf restic: Use local rand source instead of global one 2018-10-28 14:01:30 +01:00
Igor Fedorenko
da57302fca restore: Removed legacy restore implementation 
Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>
 2018-10-14 17:37:30 +02:00
Jakob Unterwurzacher
e42d2d1da8 restore: suppress lchown errors when not running as root 
Like "cp -a" and "rsync -a" do, only report lchown errors if we run
as root.

Like cp from GNU coreutils does, we check Geteuid() to determine if
we are running as root
( http://git.savannah.gnu.org/cgit/coreutils.git/tree/src/copy.c#n3012 ).

On Windows, lchown errors are always reported.

Fixes https://github.com/restic/restic/issues/1766
 2018-09-09 15:39:10 +02:00
Alexander Neumann
9fd3796d93 forget: Display reasons why snapshots are kept 
This change displays the reasons for keeping a snapshot in the table,
unless `--compact` is specified.
 2018-08-20 21:47:51 +02:00
Matthew Holt
6f5b0f3622 stat: Improve error message for bad snapshot ID (fixes #1933) 2018-08-08 16:49:36 -06:00