TrueCloudLab/restic
16
0
Fork 3
Code Issues 2 Pull requests 2 Projects Releases Packages Wiki Activity Actions
7309 commits 9 branches 49 tags 79 MiB
Commit graph

354 commits

Author SHA1 Message Date
Alexander Weiss
ce5d630681 Add MasterIndex.PackSize() 2020-11-21 22:13:54 +01:00
Alexander Weiss
c3ddde9e7d Return hdrSize in ListPack 2020-11-21 22:13:54 +01:00
greatroar
c45f8ee075 Replace restic.statT interface by concrete types 
name                old time/op    new time/op    delta
NodeFillUser-8        1.81µs ± 9%    1.50µs ± 5%  -17.07%  (p=0.000 n=19+20)
NodeFromFileInfo-8    1.76µs ± 4%    1.49µs ± 6%  -15.63%  (p=0.000 n=20+19)

name                old alloc/op   new alloc/op   delta
NodeFillUser-8          496B ± 0%      352B ± 0%  -29.03%  (p=0.000 n=20+20)
NodeFromFileInfo-8      496B ± 0%      352B ± 0%  -29.03%  (p=0.000 n=20+20)

name                old allocs/op  new allocs/op  delta
NodeFillUser-8          3.00 ± 0%      2.00 ± 0%  -33.33%  (p=0.000 n=20+20)
NodeFromFileInfo-8      3.00 ± 0%      2.00 ± 0%  -33.33%  (p=0.000 n=20+20)
 2020-11-14 23:23:26 +01:00
greatroar
63e32c44c0 Improve error reporting from restic diff 
Instead of a stacktrace, restic diff 111 222 now reports:

	Fatal: no matching ID found for prefix "111"
 2020-11-11 16:40:40 +01:00
MichaelEischer
407843c5f9
Merge pull request #3034 from hoyho/gh_master 
bugfix: omit ENOTDATA for extended attributes
 2020-11-09 22:38:49 +01:00
hoyho
9df52327cc bugfix: omit ENOTDATA for extended attributes 
Signed-off-by: hoyho <luohaihao@gmail.com>
 2020-11-10 00:20:34 +08:00
greatroar
ddca699cd2 Replace restic.Progress with new progress.Counter 
This fixes two race conditions while cleaning up the code.
 2020-11-09 12:12:35 +01:00
Alexander Weiss
38cc4393f6 Add Masterindex.Save(); Add Index.Packs() 2020-11-06 20:23:30 +01:00
Alexander Neumann
5144141321
Merge pull request #2718 from aawsome/new-cleanup-command 
Reimplementation of prune
 2020-11-05 10:12:19 +01:00
greatroar
c2f3eee5af Don't separately allocate sync.{Mutex,Once} if not necessary 
Separate allocation of synchronization devices suggests they're shared
between objects, but they're not.
 2020-11-04 09:55:43 +01:00
Alexander Weiss
b2f5381737 Make realistic forget --prune --dryrun 2020-11-03 16:42:21 +01:00
Alexander Weiss
ce7d613749 Add Blob.Handle() 2020-11-03 16:42:21 +01:00
MichaelEischer
41a45ae908
Merge pull request #3016 from greatroar/uid-gid-lookup 
Simplify/optimize cached UID/GID lookups
 2020-10-12 23:24:55 +02:00
greatroar
f0cd16e5ea Cache uint32-typed ids in lookup{Username,Group} 
NodeFillUser-8        1.92µs ± 5%    1.75µs ± 3%   -8.89%  (p=0.000 n=10+10)
NodeFromFileInfo-8    1.89µs ± 7%    1.76µs ± 4%   -6.69%  (p=0.001 n=10+10)

name                old alloc/op   new alloc/op   delta
NodeFillUser-8          504B ± 0%      496B ± 0%   -1.59%  (p=0.000 n=10+10)
NodeFromFileInfo-8      504B ± 0%      496B ± 0%   -1.59%  (p=0.000 n=10+10)

name                old allocs/op  new allocs/op  delta
NodeFillUser-8          5.00 ± 0%      3.00 ± 0%  -40.00%  (p=0.000 n=10+10)
NodeFromFileInfo-8      5.00 ± 0%      3.00 ± 0%  -40.00%  (p=0.000 n=10+10)
 2020-10-12 15:08:15 +02:00
greatroar
a03fe4562e Remove unused error return in lookup{Username,Group} 2020-10-12 14:48:23 +02:00
Michael Eischer
e638b46a13 Embed context into ReaderAt 
The io.Reader interface does not support contexts, such that it is
necessary to embed the context into the backendReaderAt struct. This has
the problem that a reader might suddenly stop working when it's
contained context is canceled. However, this is now problem here as the
reader instances never escape the calling function.
 2020-10-09 22:39:07 +02:00
Michael Eischer
4b0fcaed45 unlock: use proper context for locks cleanup 
The list operation used by RemoveStaleLocks or RemoveAllLocks will
already be canceled by the passed in context. Therefore we can also just
cancel the remove operation as the unlock command won't process all lock
files anyways.
 2020-10-09 22:37:56 +02:00
Michael Eischer
c458e114d4 pass context to Find / FindSnapshot 
This allows proper interruption of restic while it searches for
snapshots or key files.
 2020-10-09 22:37:56 +02:00
Leo R. Lundgren
028c9a5343 backup: Correct keep policy text 
Makes the following corrections to the "Applying Policy:" output:

- keep the last 1 snapshots snapshots => keep 1 latest snapshots
- keep the last 1 snapshots, 3 hourly, 5 yearly snapshots => keep 1 latest, 3 hourly, 5 yearly snapshots
 2020-09-28 14:26:53 +02:00
greatroar
23fcbb275a Remove restic.Cache interface 
It was used in one code path, which then asserted its concrete type as
*cache.Cache.

Privatised some of the interface methods.
 2020-09-18 10:48:13 +02:00
Michael Eischer
b77e933d80 Convert underscore variable names to camelCase 2020-09-05 10:07:16 +02:00
Michael Eischer
f6f11400c2 restic: Remove unused variable value 2020-09-05 10:07:16 +02:00
Michael Eischer
e2dc5034d3 restic/file: Use FileType for all File constants 
The data type of a variable is only carried on when using iota.
 2020-09-05 10:07:16 +02:00
Michael Eischer
37113282ca Merge a few variable declaration and initializations 2020-09-05 10:05:34 +02:00
Michael Eischer
337725c354 Use sort.Strings 2020-09-05 10:05:34 +02:00
aawsome
0fed6a8dfc
Use "pack file" instead of "data file" (#2885) 
- changed variable names, especially changed DataFile into PackFile
- changed in some comments
- always use "pack file" in docu
 2020-08-16 11:16:38 +02:00
Michael Eischer
af66a62c04 FindUsedBlobs: Test that seen blobs are skipped 
This also copies the TreeLoader interface from internal/walker to allow
stubbing the repository in the call to `FindUsedBlobs`.
 2020-08-01 12:29:16 +02:00
Michael Eischer
9ea1a78bd4 FindUsedBlobs: Check for seen blobs before loading trees 
The only effective change in behavior is that that toplevel nodes can
also be skipped.
 2020-08-01 12:29:16 +02:00
Michael Eischer
184103647a FindUsedBlobs: merge seen into blobs BlobSet 
The seen BlobSet always contained a subset of the entries in blobs.
Thus use blobs instead and avoid the memory overhead of the second set.

Suggested-by: Alexander Weiss <alex@weissfam.de>
 2020-08-01 12:29:16 +02:00
Michael Eischer
c847aace35 Rename Index interface to MasterIndex 
The interface is now only implemented by repository.MasterIndex.
 2020-07-25 21:19:46 +02:00
Alexander Weiss
9d1fb94c6c make Lookup() return all blobs 
+ simplify syntax
 2020-07-25 21:18:34 +02:00
rawtaz
5d8d70542f
Merge pull request #2852 from MichaelEischer/drop-go-1.11 
Drop support for Go version 1.11
 2020-07-25 01:17:36 +02:00
Michael Eischer
7c23381a2b Drop support for Go version 1.11 2020-07-24 18:52:39 +02:00
MichaelEischer
34181b13a2
Merge pull request #2328 from MichaelEischer/no-repeated-checks 
Fix duplicate tree checks within `restic check`
 2020-07-22 22:08:02 +02:00
Michael Eischer
ddf0b8cd0b checker: Properly distinguish between data and tree blobs 
If a data blob and a tree blob with the same ID (= same content) exist,
then the checker did not report a data or tree blob as unused when the
blob of the other type was still in use.
 2020-07-20 22:58:39 +02:00
greatroar
7bda28f31f Chaining hash table for repository.Index 
These are faster to construct but slower to access. The allocation rate
is halved, the peak memory usage almost halved compared to standard map.

Benchmark results on linux/amd64, -benchtime=3s -count=20:

name                                            old time/op    new time/op    delta
PackerManager-8                                    178ms ± 0%     178ms ± 0%     ~     (p=0.231 n=20+20)
DecodeIndex-8                                      4.54s ± 0%     4.30s ± 0%   -5.20%  (p=0.000 n=18+17)
DecodeIndexParallel-8                              4.54s ± 0%     4.30s ± 0%   -5.22%  (p=0.000 n=19+18)
IndexHasUnknown-8                                 44.4ns ± 5%    50.5ns ±11%  +13.82%  (p=0.000 n=19+17)
IndexHasKnown-8                                   48.3ns ± 0%    51.5ns ±12%   +6.68%  (p=0.001 n=16+20)
IndexAlloc-8                                       758ms ± 1%     616ms ± 1%  -18.69%  (p=0.000 n=19+19)
IndexAllocParallel-8                               234ms ± 3%     204ms ± 2%  -12.60%  (p=0.000 n=20+18)
MasterIndexLookupSingleIndex-8                     122ns ± 0%     145ns ± 9%  +18.44%  (p=0.000 n=14+20)
MasterIndexLookupMultipleIndex-8                   369ns ± 2%     429ns ± 8%  +16.27%  (p=0.000 n=20+20)
MasterIndexLookupSingleIndexUnknown-8             68.4ns ± 5%    74.9ns ±13%   +9.47%  (p=0.000 n=20+20)
MasterIndexLookupMultipleIndexUnknown-8            315ns ± 3%     369ns ±11%  +17.14%  (p=0.000 n=20+20)
MasterIndexLookupParallel/known,indices=5-8        743ns ± 1%     816ns ± 2%   +9.87%  (p=0.000 n=17+17)
MasterIndexLookupParallel/unknown,indices=5-8      238ns ± 1%     260ns ± 2%   +9.14%  (p=0.000 n=19+20)
MasterIndexLookupParallel/known,indices=10-8      1.01µs ± 3%    1.11µs ± 2%   +9.79%  (p=0.000 n=19+20)
MasterIndexLookupParallel/unknown,indices=10-8     222ns ± 0%     269ns ± 2%  +20.83%  (p=0.000 n=16+20)
MasterIndexLookupParallel/known,indices=20-8      1.06µs ± 2%    1.19µs ± 2%  +12.95%  (p=0.000 n=19+18)
MasterIndexLookupParallel/unknown,indices=20-8     413ns ± 1%     530ns ± 1%  +28.19%  (p=0.000 n=18+20)
SaveAndEncrypt-8                                  30.2ms ± 1%    30.4ms ± 0%   +0.71%  (p=0.000 n=19+19)
LoadTree-8                                         540µs ± 1%     576µs ± 1%   +6.73%  (p=0.000 n=20+20)
LoadBlob-8                                        5.64ms ± 0%    5.64ms ± 0%     ~     (p=0.883 n=18+17)
LoadAndDecrypt-8                                  5.93ms ± 0%    5.95ms ± 1%     ~     (p=0.247 n=20+19)
LoadIndex-8                                       25.1ms ± 0%    24.5ms ± 1%   -2.54%  (p=0.000 n=18+17)

name                                            old speed      new speed      delta
PackerManager-8                                  296MB/s ± 0%   296MB/s ± 0%     ~     (p=0.229 n=20+20)
SaveAndEncrypt-8                                 139MB/s ± 1%   138MB/s ± 0%   -0.71%  (p=0.000 n=19+19)
LoadBlob-8                                       177MB/s ± 0%   177MB/s ± 0%     ~     (p=0.890 n=18+17)
LoadAndDecrypt-8                                 169MB/s ± 0%   168MB/s ± 1%     ~     (p=0.227 n=20+19)

name                                            old alloc/op   new alloc/op   delta
PackerManager-8                                   91.8kB ± 0%    91.8kB ± 0%     ~     (p=0.772 n=12+19)
IndexAlloc-8                                       786MB ± 0%     400MB ± 0%  -49.04%  (p=0.000 n=20+18)
IndexAllocParallel-8                               786MB ± 0%     401MB ± 0%  -49.04%  (p=0.000 n=19+15)
SaveAndEncrypt-8                                  21.0MB ± 0%    21.0MB ± 0%   +0.00%  (p=0.000 n=19+19)

name                                            old allocs/op  new allocs/op  delta
PackerManager-8                                    1.41k ± 0%     1.41k ± 0%     ~     (all equal)
IndexAlloc-8                                        977k ± 0%      907k ± 0%   -7.18%  (p=0.000 n=20+20)
IndexAllocParallel-8                                977k ± 0%      907k ± 0%   -7.17%  (p=0.000 n=19+15)
SaveAndEncrypt-8                                    73.0 ± 0%      73.0 ± 0%     ~     (all equal)
 2020-07-19 13:58:22 +02:00
MichaelEischer
dd7b4f54f5
Merge pull request #2709 from greatroar/minio-sha256 
Use Minio's optimized SHA-256
 2020-06-12 23:32:58 +02:00
Alexander Weiss
91906911b0 Fix non-intuitive repository behavior 
- The SaveBlob method now checks for duplicates.
- Moves handling of pending blobs to MasterIndex.
  -> also cleans up pending index entries when they are saved in the index
  -> when using SaveBlob no need to care about index any longer
- Always check for full index and save it when storing packs.
  -> removes the need of an index uploader
  -> also removes the verbose "uploaded intermediate index" messages
- The Flush method now also saves the index
- Fix race condition when checking and saving full/non-finalized indexes
 2020-06-11 13:05:23 +02:00
greatroar
42a3db05b0 Use Minio's optimized SHA-256 
internal/repository benchmarks on an Intel i7-3770k:

name               old speed      new speed       delta
PackerManager-8     209MB/s ± 1%    291MB/s ± 1%  +38.94%  (p=0.008 n=5+5)
SaveAndEncrypt-8    112MB/s ± 1%    135MB/s ± 1%  +20.25%  (p=0.008 n=5+5)
 2020-04-28 07:57:18 +02:00
greatroar
be5a0ff59f Centralize buffer allocation and size checking in Repository.LoadBlob 
Benchmark results for internal/repository:

name              old time/op    new time/op    delta
LoadTree-8           479µs ± 2%     478µs ± 1%   ~     (p=0.780 n=10+9)
LoadBlob-8          11.6ms ± 2%    11.6ms ± 1%   ~     (p=0.631 n=10+10)
LoadAndDecrypt-8    13.2ms ± 2%    13.3ms ± 3%   ~     (p=0.631 n=10+10)

name              old alloc/op   new alloc/op   delta
LoadTree-8          41.2kB ± 0%    41.2kB ± 0%   ~     (all equal)
LoadBlob-8          2.28kB ± 0%    2.28kB ± 0%   ~     (all equal)
LoadAndDecrypt-8    2.10MB ± 0%    2.10MB ± 0%   ~     (all equal)

name              old allocs/op  new allocs/op  delta
LoadTree-8             652 ± 0%       652 ± 0%   ~     (all equal)
LoadBlob-8            24.0 ± 0%      24.0 ± 0%   ~     (all equal)
LoadAndDecrypt-8      30.0 ± 0%      30.0 ± 0%   ~     (all equal)

name              old speed      new speed      delta
LoadBlob-8        86.2MB/s ± 2%  86.4MB/s ± 1%   ~     (p=0.594 n=10+10)
LoadAndDecrypt-8  75.7MB/s ± 2%  75.4MB/s ± 3%   ~     (p=0.617 n=10+10)
 2020-04-23 10:04:20 +02:00
greatroar
2e31120f89 Remove unused argument to restic.fakeFile 2020-04-18 17:40:13 +02:00
MichaelEischer
b7c0d4d8bf
Merge pull request #2644 from greatroar/signal-notify-buffered 
Make all signal.Notify channels buffered
 2020-04-18 11:31:02 +02:00
rawtaz
9efbe98879
Merge pull request #2623 from alrs/internal-restic-err-before-close 
internal/restic: close os.File after checking for error
 2020-03-18 22:23:20 +01:00
rawtaz
7d9300efca
Merge pull request #2637 from greatroar/unused 
Remove Go 1.5 compatibility code
 2020-03-18 22:22:15 +01:00
greatroar
47d4d5bf1b Make all signal.Notify channels buffered 2020-03-12 20:59:39 +01:00
greatroar
4de12bf593 Remove restic.RandReader 
math/rand.Rand has implemented Reader since Go 1.6. The repacking tests
are not deterministic, but they weren't before, either.
 2020-03-09 10:00:28 +01:00
Lars Lehtonen
4a2156d3f0
internal/restic: close os.File after checking for error 2020-03-05 16:22:46 -08:00
rawtaz
58bd165253
Merge pull request #2581 from aawsome/multiple-hostnames 
Allow multiple hostnames tags
 2020-02-27 08:35:23 +01:00
Alexander Weiss
9a9101d144 Support specifying multiple host flags for various commands 
The `dump`, `find`, `forget`, `ls`, `mount`, `restore`, `snapshots`,
`stats` and `tag` commands will now take into account multiple
`--host` and `-H` flags.
 2020-02-26 22:17:59 +01:00
Lars Lehtonen
16b321b140
internal/restic: fix dropped test error 2020-02-12 13:32:45 -08:00
Ward Vandewege
69a0d0ee90
fix typos 2019-12-21 21:00:28 -05:00
rawtaz
da4473aba6
Merge pull request #2391 from kastenhq/refresh-lock-time 
Update Lock.Time in lock.Refresh()
 2019-11-26 21:44:29 +01:00
Leo R. Lundgren
4d2aa18273 Accept both singular and plural for --group-by (#2330) 2019-11-20 19:25:48 +01:00
Eric Hamilton
1596d06f8e Update Lock.Time in lock.Refresh() 2019-09-04 11:38:35 -07:00
Alexandr Bruyako
38ea7ed4f6 remove unused code 2019-07-01 00:24:45 +03:00
Alexander Neumann
355db0bc29 windows: Use LastWriteTime for ctime and mtime 
Windows does not have a concept of a `change time` in the sense as Unix
has it: the field `CreationTime` of the `Win32FileAttributeData` struct
is not updated when attributes or content is changed. So from now on
we're using the `LastWriteTime` as the `change time` on Windows.
 2019-05-05 14:02:11 +02:00
Alexander Neumann
939f3e972c node: Make sure year of all timestamps is valid 
Sometimes restic gets bogus timestamps which cannot be converted to
JSON, because the stdlib JSON encoder returns an error if the year is
not within [0, 9999]. We now make sure that we at least record _some_
timestamp and cap the year either to 0000 or 9999. Before, restic would
refuse to save the file at all, so this improves the status quo.

This fixes #2174 and #1173
 2019-04-24 16:39:36 +02:00
Alexander Neumann
18ec27a0da Merge pull request #2243 from treymo/ls-length-2104 
backend find: Check prefix length < snapshot IDs when searching
 2019-04-23 09:41:19 +02:00
Jan Niklas Richter
733519d895 Refactor duplicate code for grouping snapshots 
This commit is a followup to the addition of the --group-by flag for the
snapshots command. Adding the grouping code there introduced duplicated
code (the forget command also does grouping). This commit refactors
boths sides to only use shared code.
 2019-04-22 14:57:53 +02:00
Trey Moore
5ac9c1157a Prevent "slice bounds out of range" error if prefix is longer than snapshot IDs. 
This includes tests as well as other tests for the backend_find.Find function. Issue #2104.
 2019-04-16 20:26:20 -07:00
Alexander Neumann
d51e9d1b98 Add []byte to repo.LoadAndDecrypt and utils.LoadAll 
This commit changes the signatures for repository.LoadAndDecrypt and
utils.LoadAll to allow passing in a []byte as the buffer to use. This
buffer is enlarged as needed, and returned back to the caller for
further use.

In later commits, this allows reducing allocations by reusing a buffer
for multiple calls, e.g. in a worker function.
 2019-04-13 13:38:39 +02:00
Alexander Neumann
203d775190 restic: Make JSON unmarshal for ID more efficient 
This commit reduces several allocations in UnmarshalJSON() by decoding
the hex string directly in a single step.
 2019-04-13 13:38:39 +02:00
Alexander Neumann
c1058005c3 Fix build on FreeBSD for Go < 1.12 2019-03-16 20:52:37 +01:00
Dmitri Goutnik
1d8b21cdad Fix build on FreeBSD with Go 1.12 2019-03-06 09:41:49 -05:00
plumbeo
71891b340c Support time ranges expressed in hours in snapshot retention policies 
Make restic forget --keep-within accept time ranges measured in hours and choose
accordingly which snapshots to keep and which to forget. Add relative tests.
 2018-11-26 14:27:42 +01:00
plumbeo
7486bfea5b Extend custom Duration granularity to hours and add tests 2018-11-14 17:27:02 +01:00
Alexander Neumann
3ae2a79bdf restic: Use local rand source instead of global one 2018-10-28 14:01:30 +01:00
Igor Fedorenko
da57302fca restore: Removed legacy restore implementation 
Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>
 2018-10-14 17:37:30 +02:00
Jakob Unterwurzacher
e42d2d1da8 restore: suppress lchown errors when not running as root 
Like "cp -a" and "rsync -a" do, only report lchown errors if we run
as root.

Like cp from GNU coreutils does, we check Geteuid() to determine if
we are running as root
( http://git.savannah.gnu.org/cgit/coreutils.git/tree/src/copy.c#n3012 ).

On Windows, lchown errors are always reported.

Fixes https://github.com/restic/restic/issues/1766
 2018-09-09 15:39:10 +02:00
Alexander Neumann
9fd3796d93 forget: Display reasons why snapshots are kept 
This change displays the reasons for keeping a snapshot in the table,
unless `--compact` is specified.
 2018-08-20 21:47:51 +02:00
Matthew Holt
6f5b0f3622 stat: Improve error message for bad snapshot ID (fixes #1933) 2018-08-08 16:49:36 -06:00
Alexander Neumann
abb18a830c Fix test 2018-07-14 11:51:34 +02:00
Alexander Neumann
7ddf91b65c Merge pull request #1772 from ifedorenko/restore-verify 
Restore verify
 2018-06-23 14:50:29 +02:00
Iku Iwasa
f080142137 Support NetBSD without fuse 2018-06-16 15:55:04 +09:00
Igor Fedorenko
5fa6dc53cb Refactor: introduced restorer tree visitor 
Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>
 2018-06-11 07:57:22 -04:00
Igor Fedorenko
26be094f28 Refactor: moved restorer to separate package 
Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>
 2018-06-11 07:57:22 -04:00
Alexander Neumann
060d8b57e0 Restructure TestApplyPolicy 2018-05-13 12:48:42 +02:00
Alexander Neumann
cc627e832b Add custom Duration type 2018-05-13 12:48:42 +02:00
Alexander Neumann
5a0f0e3faa Add support for keeping a range of snapshots 2018-05-13 12:48:42 +02:00
Matthew Holt
b52f2aa9a4 forget: Add policy to keep snapshots before a date 2018-05-13 12:48:10 +02:00
Alexander Neumann
64d98945a6 backup: Clean target paths before finding parent 
This resolves an issue described in the forum where restic could not
find a parent snapshot if the target path ends with a slash:
https://forum.restic.net/t/new-archiver-code-please-test/623/23
 2018-04-30 22:03:11 +02:00
Alexander Neumann
76b616451f Remove unneeded code 2018-04-23 21:40:33 +02:00
Alexander Neumann
b6f98bdb02 node: Fill minimal info 2018-04-22 11:37:05 +02:00
Alexander Neumann
6a7c23d2ae tree: Add convenience functions 2018-04-22 11:37:05 +02:00
Alexander Neumann
cc847a3d6d tree: Improve error for pre-existing node 2018-04-22 11:37:05 +02:00
Alexander Neumann
baebf45e2e FindLatestSnapshot: Use absolute paths 2018-04-22 11:37:05 +02:00
Alexander Neumann
fa4f438bc1 snapshot: Do not modify slice of paths 2018-04-22 11:37:05 +02:00
Alexander Neumann
4e0b2a8e3a snapshot: correct error handling for filepath.Abs 2018-04-22 11:37:05 +02:00
Alexander Neumann
a472868e06 fs: Add TestChdir() 2018-04-22 11:37:05 +02:00
Alexander Neumann
09bd924710 Do not restore sockets, correct error handling 
Closes #1730
 2018-04-20 13:53:11 +02:00
Alexander Neumann
e68a7fea8a check: Allow filling the cache during check 
Closes #1665
 2018-04-01 13:59:27 +02:00
Alexander Neumann
d3f9c8b362 forget: Print policy 2018-03-30 10:24:26 +02:00
Alexander Neumann
b9ada91054 Reduce data for TestCreateSnapshot 2018-03-11 21:42:39 +01:00
Alexander Neumann
c67a8452f7 Disable polynomial check for chunker for tests 2018-03-11 20:59:40 +01:00
Alexander Neumann
d8dcbc89d1 lock: Ignore invalid lock file 
This commit fixes a bug introduced in
e9ea268847: When an invalid lock is
encountered (e.g. if the file is empty), the code used to ignore that,
but now returns the error.

Now, invalid files are ignored for the normal lock check, and removed
when `restic unlock --remove-all` is run.

Closes #1652
 2018-03-05 20:19:57 +01:00
Alexander Neumann
be0a5b7f06 Merge pull request #1649 from jasperla/solaris 
Minimal set of patches to get restic working on Solaris
 2018-03-05 20:00:17 +01:00
Jasper Lievisse Adriaanse
96311d1a2b Add support for illumos/Solaris 
This does come without xattr/fuse support at this point.

NB: not hooking up the integration tests as restic won't compile without
    cgo with Go < 1.10.
 2018-03-04 20:11:29 +00:00
Alexander Neumann
929afc63d5 Use int64 for the length in the RewindReader 2018-03-04 10:40:42 +01:00
Alexander Neumann
99f7fd74e3 backend: Improve Save() 
As mentioned in issue [#1560](https://github.com/restic/restic/pull/1560#issuecomment-364689346)
this changes the signature for `backend.Save()`. It now takes a
parameter of interface type `RewindReader`, so that the backend
implementations or our `RetryBackend` middleware can reset the reader to
the beginning and then retry an upload operation.

The `RewindReader` interface also provides a `Length()` method, which is
used in the backend to get the size of the data to be saved. This
removes several ugly hacks we had to do to pull the size back out of the
`io.Reader` passed to `Save()` before. In the `s3` and `rest` backend
this is actively used.
 2018-03-03 15:49:44 +01:00
Alexander Neumann
93210614f4 backend/retry: return worker function error and abort 
This is a bug fix: Before, when the worker function fn in List() of the
RetryBackend returned an error, the operation is retried with the next
file. This is not consistent with the documentation, the intention was
that when fn returns an error, this is passed on to the caller and the
List() operation is aborted. Only errors happening on the underlying
backend are retried.

The error leads to restic ignoring exclusive locks that are present in
the repo, so it may happen that a new backup is written which references
data that is going to be removed by a concurrently running `prune`
operation.

The bug was reported by a user here:
https://forum.restic.net/t/restic-backup-returns-0-exit-code-when-already-locked/484
 2018-02-24 13:26:13 +01:00
Alexander Neumann
be02008025 Merge pull request #1611 from qbit/master 
On OpenBSD only root can set sticky bit
 2018-02-18 12:57:54 +01:00
Igor Fedorenko
d58ae43317 Reworked Backend.Load API to retry errors during ongoing download 
Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>
 2018-02-16 21:12:14 -05:00
Aaron Bieber
99d88ad297
Disable the 'testSticky' test on OpenBSD. Only root can set sticky. 2018-02-11 07:46:31 -07:00
Igor Fedorenko
aa333f4d49 Implement RetryBackend.List() 
Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>
 2018-01-29 22:14:12 -05:00
Alexander Neumann
663c57ab4d debug: Remove manual Str() call Log() 2018-01-25 20:49:41 +01:00
Alexander Neumann
9c55e8d69c Merge pull request #1549 from MJDSys/more_index_lookup_avoids 
More optimizations to avoid calling Index.Lookup()
 2018-01-24 20:53:30 +01:00
Igor Fedorenko
0084e42cb6 Optimize Repository.ListPack() 
Use pack file size returned by Backend.List() to avoid extra per-pack
Backend.Stat() requests

Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>
 2018-01-23 22:39:51 -05:00
Matthew Dawson
df2c03a6a4
repository/master_index: Optimize Index.Lookup() 
When looking up a blob in the master index, with several
indexes present in the master index, a significant amount of time
is spent generating errors for each failed lookup.  However, these
errors are often used to check if a blob is present, but the contents
are not inspected making the overhead of the error not useful.

Instead, change Index.Lookup (and Index.LookupSize) to instead return
a boolean denoting if the blob was found instead of an error.  Also change
all the calls to these functions to handle the new function signature.

benchmark                                            old ns/op     new ns/op     delta
BenchmarkMasterIndexLookupSingleIndex-6              820           897           +9.39%
BenchmarkMasterIndexLookupMultipleIndex-6            12821         2001          -84.39%
BenchmarkMasterIndexLookupSingleIndexUnknown-6       5378          492           -90.85%
BenchmarkMasterIndexLookupMultipleIndexUnknown-6     17026         1649          -90.31%

benchmark                                            old allocs     new allocs     delta
BenchmarkMasterIndexLookupSingleIndex-6              9              9              +0.00%
BenchmarkMasterIndexLookupMultipleIndex-6            59             19             -67.80%
BenchmarkMasterIndexLookupSingleIndexUnknown-6       22             6              -72.73%
BenchmarkMasterIndexLookupMultipleIndexUnknown-6     72             16             -77.78%

benchmark                                            old bytes     new bytes     delta
BenchmarkMasterIndexLookupSingleIndex-6              160           160           +0.00%
BenchmarkMasterIndexLookupMultipleIndex-6            3200          240           -92.50%
BenchmarkMasterIndexLookupSingleIndexUnknown-6       1232          48            -96.10%
BenchmarkMasterIndexLookupMultipleIndexUnknown-6     4272          128           -97.00%
 2018-01-23 22:25:56 -05:00
Alexander Neumann
685ce719ad Add comments for List() 2018-01-23 23:14:05 +01:00
Alexander Neumann
b0c6e53241 Fix calls to repo/backend.List() everywhere 2018-01-21 21:15:09 +01:00
Alexander Neumann
52230b8f07 backend: Rework List() 
For a discussion see #1567
 2018-01-21 18:35:37 +01:00
Alexander Neumann
c31a5e7e5c Add argument to Skipf() 2018-01-17 23:14:37 +01:00
Alexander Neumann
b8620429e2 restorer: Set directory mode as last step 
Closes #1512
 2018-01-07 15:13:24 +01:00
Alexander Neumann
aab414b368 Add test case for #1512 2018-01-07 14:50:47 +01:00
Alexander Neumann
6c2b2a58ad backend: Retry deletes 2017-12-22 22:41:28 +01:00
Alexander Neumann
3a5e040b7e Merge pull request #1439 from armhold/propagate-context 
replace ad-hoc context.TODO() with gopts.ctx
 2017-12-08 20:27:36 +01:00
Alexander Neumann
82ded35706 node: Correctly compare times 2017-12-03 18:33:35 +01:00
Alexander Neumann
88607fc625 Correctly format all node types 2017-12-03 17:38:55 +01:00
George Armhold
20d78ab0d9 add context to repo.Flush() call in test code 
gh-1434
 2017-12-03 07:32:50 -05:00
George Armhold
d886cb5c27 replace ad-hoc context.TODO() with gopts.ctx, so that cancellation 
can properly trickle down from cmd_*.

gh-1434
 2017-12-03 07:22:14 -05:00
Alexander Neumann
0f5e38609f restore: Fix restore to relative path 2017-11-26 18:36:48 +01:00
Alexander Neumann
27d29b9853 restorer: Make sure node names are clean 2017-11-26 15:28:45 +01:00
Alexander Neumann
8a171731ba restorer: Add tests for invalid node names 2017-11-26 15:28:37 +01:00
Alexander Neumann
8ce5d35543 Merge pull request #1390 from armhold/lint 
small cleanup
 2017-10-28 09:54:35 +02:00
George Armhold
eea96f652d go fmt 2017-10-26 16:22:10 -04:00
George Armhold
f5fa602482 detect and return error from file Close() in Node.createFileAt() 
gh-1385
 2017-10-26 13:53:31 -04:00
Konrad Wojas
5b96885c6d Control progress rate with RESTIC_PROGRESS_FPS env 
Add a RESTIC_PROGRESS_FPS environment variable to limit the interval
at which the progress indicator updates (allowed values: 1-60).

The default rate of 60 FPS can cause high terminal CPU load on some
systems, like iTerm2 on macOS with font anti-aliasing enabled.

Usage:

    RESTIC_PROGRESS_FPS=1 restic ...
    RESTIC_PROGRESS_FPS=60 restic ...
 2017-10-26 14:46:56 +08:00
George Armhold
bcdebfb84e small cleanup: 
- be explicit when discarding returned errors from .Close(), etc.
- remove named return values from funcs when naked return not used
- fix some "err" shadowing when redeclaration not needed
 2017-10-25 12:03:55 -04:00
Alexander Neumann
e56370eb5b Remove Deleter interface 2017-10-14 16:04:29 +02:00
Alexander Neumann
0e722efb09 backend: Add Delete() to restic.Backend interface 2017-10-14 15:56:25 +02:00
Alexander Neumann
1dcfd64028 Merge pull request #1325 from antonlindstrom/lookup-group 
Add group name in fillUser
 2017-10-04 21:32:46 +02:00
Anton Lindstrom
26df48b2aa Add group name in fillUser 
This does a lookup of the group name from the GID and adds it to the
Node.
 2017-10-04 20:11:54 +02:00
Herbert
3473c3f7b6 Remove all dot-imports 2017-10-02 15:06:39 +02:00
Alexander Neumann
2e72b57f2f Correct debug message 2017-09-25 14:35:37 +02:00
Alexander Neumann
176bfa6529 backend: Improve ReadAt 2017-09-24 23:11:23 +02:00
Alexander Neumann
610b676444 Automatically exclude current restic cache 2017-09-24 21:54:53 +02:00
Alexander Neumann
9be24a1c9f Add cache 
This commits adds rudimentary support for a cache directory, enabled by
default. The cache directory is created if it does not exist. The cache
is used if there's anything in it, newly created snapshot and index
files are written to the cache automatically.
 2017-09-24 21:54:53 +02:00
Alexander Neumann
3f6e11d26e Allow sorting nodes in trees 2017-09-22 12:37:05 +02:00
Alexander Neumann
a60e751217 Use .Equal() instead of == for time.Time 
Closes #1238
 2017-09-15 20:57:35 +02:00
Alexander Neumann
36eaa22ed0 Merge pull request #1205 from mungomat/backup_time 
Backup time
 2017-09-11 19:01:41 +02:00
Alexander Neumann
62df316356 Merge pull request #1194 from mungomat/bugfix_bucker_always 
fix: bucker
 2017-09-11 19:01:23 +02:00
Alexander Neumann
9fa909ccd6 Update golden files 2017-09-11 17:52:22 +02:00
Tobias Klein
9924c311c9 added test cases 2017-09-10 12:23:28 +02:00
Tobias Klein
a677f1139a removed unnacessary line 2017-09-10 10:41:07 +02:00
Tobias Klein
087c3fe1dc tests updated 2017-09-09 13:26:35 +02:00
Tobias Klein
43ff971dfd new sub-option for backup: time 
New option to specify the timestamp for a backup
 2017-09-09 13:26:35 +02:00
Alexander Neumann
fa2ee78a5c Merge pull request #1044 from lloeki/982-improve-restore 
Improve restore
 2017-09-04 21:51:12 +02:00
Tobias Klein
2d73a273af saving a variable 2017-09-03 17:09:55 +02:00
Tobias Klein
761af08889 fix: bucker 
bucker "always" does not return a unique id in case of exact same timestamps
 2017-09-03 17:09:55 +02:00
Greg V
3fbdd12b04 Handle SIGINFO on all supported platforms 
Not just darwin
 2017-09-02 22:06:31 +03:00
Loic Nageleisen
f880ff21aa Fixing restore with excluded 
An exclude filter is basically a 'wildcard but foo', so even if a
childMayMatch, other children of a dir may not, therefore childMayMatch
does not matter, but we should not go down unless the dir is selected
for restore.
 2017-08-16 15:25:02 +02:00
Loic Nageleisen
4a36993c19 Smarter filter when children won't match 
This improves restore performance by several orders of magniture by not
going through the whole tree recursively when we can anticipate that no
match will ever occur.
 2017-08-16 15:25:02 +02:00
Emil Hessman
c2ff7150aa internal: check error before deferring file Close() 
If there is an error, file will be `nil`. We should check the returned error before deferring file `Close()`.
 2017-08-13 19:28:13 +02:00
Alexander Neumann
23c903074c Move restic package to internal/restic 2017-07-24 17:43:32 +02:00