KurlesHS/restic
1
0
Fork 0
forked from TrueCloudLab/restic
Code Pull requests Activity
4839 commits 10 branches 49 tags 79 MiB
Commit graph

297 commits

Author SHA1 Message Date
Alexander Neumann
f9f6124558 Update chunker 2020-07-20 22:07:02 +02:00
greatroar
7bda28f31f Chaining hash table for repository.Index 
These are faster to construct but slower to access. The allocation rate
is halved, the peak memory usage almost halved compared to standard map.

Benchmark results on linux/amd64, -benchtime=3s -count=20:

name                                            old time/op    new time/op    delta
PackerManager-8                                    178ms ± 0%     178ms ± 0%     ~     (p=0.231 n=20+20)
DecodeIndex-8                                      4.54s ± 0%     4.30s ± 0%   -5.20%  (p=0.000 n=18+17)
DecodeIndexParallel-8                              4.54s ± 0%     4.30s ± 0%   -5.22%  (p=0.000 n=19+18)
IndexHasUnknown-8                                 44.4ns ± 5%    50.5ns ±11%  +13.82%  (p=0.000 n=19+17)
IndexHasKnown-8                                   48.3ns ± 0%    51.5ns ±12%   +6.68%  (p=0.001 n=16+20)
IndexAlloc-8                                       758ms ± 1%     616ms ± 1%  -18.69%  (p=0.000 n=19+19)
IndexAllocParallel-8                               234ms ± 3%     204ms ± 2%  -12.60%  (p=0.000 n=20+18)
MasterIndexLookupSingleIndex-8                     122ns ± 0%     145ns ± 9%  +18.44%  (p=0.000 n=14+20)
MasterIndexLookupMultipleIndex-8                   369ns ± 2%     429ns ± 8%  +16.27%  (p=0.000 n=20+20)
MasterIndexLookupSingleIndexUnknown-8             68.4ns ± 5%    74.9ns ±13%   +9.47%  (p=0.000 n=20+20)
MasterIndexLookupMultipleIndexUnknown-8            315ns ± 3%     369ns ±11%  +17.14%  (p=0.000 n=20+20)
MasterIndexLookupParallel/known,indices=5-8        743ns ± 1%     816ns ± 2%   +9.87%  (p=0.000 n=17+17)
MasterIndexLookupParallel/unknown,indices=5-8      238ns ± 1%     260ns ± 2%   +9.14%  (p=0.000 n=19+20)
MasterIndexLookupParallel/known,indices=10-8      1.01µs ± 3%    1.11µs ± 2%   +9.79%  (p=0.000 n=19+20)
MasterIndexLookupParallel/unknown,indices=10-8     222ns ± 0%     269ns ± 2%  +20.83%  (p=0.000 n=16+20)
MasterIndexLookupParallel/known,indices=20-8      1.06µs ± 2%    1.19µs ± 2%  +12.95%  (p=0.000 n=19+18)
MasterIndexLookupParallel/unknown,indices=20-8     413ns ± 1%     530ns ± 1%  +28.19%  (p=0.000 n=18+20)
SaveAndEncrypt-8                                  30.2ms ± 1%    30.4ms ± 0%   +0.71%  (p=0.000 n=19+19)
LoadTree-8                                         540µs ± 1%     576µs ± 1%   +6.73%  (p=0.000 n=20+20)
LoadBlob-8                                        5.64ms ± 0%    5.64ms ± 0%     ~     (p=0.883 n=18+17)
LoadAndDecrypt-8                                  5.93ms ± 0%    5.95ms ± 1%     ~     (p=0.247 n=20+19)
LoadIndex-8                                       25.1ms ± 0%    24.5ms ± 1%   -2.54%  (p=0.000 n=18+17)

name                                            old speed      new speed      delta
PackerManager-8                                  296MB/s ± 0%   296MB/s ± 0%     ~     (p=0.229 n=20+20)
SaveAndEncrypt-8                                 139MB/s ± 1%   138MB/s ± 0%   -0.71%  (p=0.000 n=19+19)
LoadBlob-8                                       177MB/s ± 0%   177MB/s ± 0%     ~     (p=0.890 n=18+17)
LoadAndDecrypt-8                                 169MB/s ± 0%   168MB/s ± 1%     ~     (p=0.227 n=20+19)

name                                            old alloc/op   new alloc/op   delta
PackerManager-8                                   91.8kB ± 0%    91.8kB ± 0%     ~     (p=0.772 n=12+19)
IndexAlloc-8                                       786MB ± 0%     400MB ± 0%  -49.04%  (p=0.000 n=20+18)
IndexAllocParallel-8                               786MB ± 0%     401MB ± 0%  -49.04%  (p=0.000 n=19+15)
SaveAndEncrypt-8                                  21.0MB ± 0%    21.0MB ± 0%   +0.00%  (p=0.000 n=19+19)

name                                            old allocs/op  new allocs/op  delta
PackerManager-8                                    1.41k ± 0%     1.41k ± 0%     ~     (all equal)
IndexAlloc-8                                        977k ± 0%      907k ± 0%   -7.18%  (p=0.000 n=20+20)
IndexAllocParallel-8                                977k ± 0%      907k ± 0%   -7.17%  (p=0.000 n=19+15)
SaveAndEncrypt-8                                    73.0 ± 0%      73.0 ± 0%     ~     (all equal)
 2020-07-19 13:58:22 +02:00
greatroar
4cf1c8e8da Changelog entry for subquadratic reading in mount 2020-07-12 18:27:16 +02:00
Alexander Weiss
b5543cff5d Loop over index files for 'list blobs' 
=> reduces memory consumption a lot!
 2020-06-14 17:25:34 +02:00
MichaelEischer
74bc7141c1
Merge pull request #2781 from aawsome/reduce-index-memory 
Reduce index memory
 2020-06-14 10:58:28 +02:00
Alexander Weiss
7419844885 add changelog, benchmark, memory calculation 2020-06-14 07:50:15 +02:00
rawtaz
1d66bb9e62
Merge pull request #2784 from MichaelEischer/minio-sha256-changelog 
Add changelog for #2709
 2020-06-13 18:33:54 +02:00
Michael Eischer
0b2c31b05b Add changelog for #2709 2020-06-13 18:13:36 +02:00
Alexander Weiss
70347e95d5 disable index uploads for prune command 
+ modifications of changelog
 2020-06-12 09:24:38 +02:00
Alexander Weiss
91906911b0 Fix non-intuitive repository behavior 
- The SaveBlob method now checks for duplicates.
- Moves handling of pending blobs to MasterIndex.
  -> also cleans up pending index entries when they are saved in the index
  -> when using SaveBlob no need to care about index any longer
- Always check for full index and save it when storing packs.
  -> removes the need of an index uploader
  -> also removes the verbose "uploaded intermediate index" messages
- The Flush method now also saves the index
- Fix race condition when checking and saving full/non-finalized indexes
 2020-06-11 13:05:23 +02:00
Martin Michlmayr
5cc1760fdf
Fix typos 2020-05-16 14:05:26 +08:00
MichaelEischer
9790d8ce1c
Merge pull request #2668 from MichaelEischer/fix-stats-blobs-crash 
stats: Fix crash in blobs-per-file mode on missing blob
 2020-04-18 17:21:26 +02:00
Michael Eischer
4a400f94bb Add changelog 2020-04-12 22:27:09 +02:00
Michael Eischer
0ce81d88b6 stats: Fix crash in blobs-per-file mode on missing blob 
In a damaged repository with a missing blob, the error message tried to
dereference the subtreeID field of the current node, which is a file
however. Said field is set to nil for a file thus causing a segfault
when dereferenced.

Fix this by using the actual parentTreeID.
 2020-03-27 22:17:54 +01:00
Peter Schultz
90fc639a67 Allow specifying user and host when adding keys 
The username and hostname for new keys can be specified with the new
--user and --host flags, respectively. The flags are used only by the
`key add` command and are otherwise ignored.

This allows adding keys with for a desired user and host without having
to run restic as that particular user on that particular host, making
automated key management easier.

Co-authored-by: James TD Smith <ahktenzero@mohorovi.cc>
 2020-03-23 13:11:10 +01:00
rawtaz
cec7d581f3
Merge pull request #2610 from restic/remove-vendor 
Remove vendored dependencies
 2020-03-01 20:22:45 +01:00
Alexander Neumann
0c03a80fc4
Merge pull request #2592 from greatroar/sftp-ipv6 
Support IPv6 in SFTP backend
 2020-03-01 19:38:44 +01:00
Alexander Neumann
b1c77172c2 Add entry to changelog 2020-03-01 19:32:35 +01:00
greatroar
60e4a88f17 Changelog entry for SFTP w/ IPv6 addresses 2020-03-01 11:04:06 +01:00
greatroar
5cd0bce452 Honor RESTIC_CACHE_DIR on Mac and Windows 2020-02-28 15:44:32 +01:00
rawtaz
58bd165253
Merge pull request #2581 from aawsome/multiple-hostnames 
Allow multiple hostnames tags
 2020-02-27 08:35:23 +01:00
Alexander Weiss
9a9101d144 Support specifying multiple host flags for various commands 
The `dump`, `find`, `forget`, `ls`, `mount`, `restore`, `snapshots`,
`stats` and `tag` commands will now take into account multiple
`--host` and `-H` flags.
 2020-02-26 22:17:59 +01:00
Alexander Neumann
a40ac37550 Add changelog file 2020-02-26 20:55:12 +01:00
Igor Fedorenko
f17ffa0283 restorer: Allow writing target file blobs out of order 
Much simpler implementation that guarantees each required pack
is downloaded only once (and hence does not need to manage
pack cache). Also improves large file restore performance.

Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>
 2020-02-26 16:14:45 +01:00
David Potter
71900248ab Correct #2531 (stats restore-size calculates hard links incorrectly) 2020-02-15 19:34:22 +01:00
David Potter
23055aaadf Correct #2537 (cmd_stats file counting issue) 2020-02-15 19:34:16 +01:00
curiousleo
27d6e5e186 Add changelog file. 2020-02-15 01:32:43 +01:00
Michael Eischer
294ca55967 Add changelog entry 2020-02-13 21:14:20 +01:00
rawtaz
680a14afa1
Merge pull request #2530 from restic/fix-sftp-mkdirall 
sftp: Use MkdirAll provided by the client
 2020-02-13 01:12:59 +01:00
rawtaz
bf199e5ca7
Minor speling corection. 2020-02-13 00:47:09 +01:00
Alexander Neumann
b58dfda212 Add entry to changelog 2020-02-12 21:29:26 +01:00
Daniel Hoffend
e7cdf2acbb fix backup --json total_bytes_processed output 
Closes #2429
 2020-02-09 02:02:45 +01:00
Alexander Neumann
6e9778ae0e Add changelog file 2020-01-06 21:46:22 +01:00
Alexander Neumann
6ec5dc8016 Fix template for new version of calens 
The `wrap` function has been renamed to `wrapIndent`
 2020-01-01 11:58:21 +01:00
streambinder
c56cbfe95b changelog: add pull-2423 2019-12-19 13:15:37 +01:00
Lorenz Bausch
d818b7618b
Display respective snapshot date when using restic find 2019-12-17 07:51:12 +01:00
Matt Holt
1c3812a6f6
Merge pull request #2342 from rpsene/master 
Add support for ppc64le
 2019-12-16 11:50:52 -07:00
Rafael Peria de Sene
ec91b80f09 Add support for ppc64le 
This commit adds support for cross-compiling Restic for ppc64le
as part of the release process.

Add a new file:   changelog/unreleased/issue-2277
Modifies:   helpers/build-release-binaries/main.go
Modifies:   run_integration_tests.go
 2019-12-15 22:52:13 -03:00
Eric Hamilton
d21a13f1ee Create issue-2390 2019-12-10 23:20:18 +01:00
Alexander Neumann
fe0361ffcb Move changelog entry for #2179 
For background on how this happened see
https://forum.restic.net/t/psa-attention-anyone-not-using-a-recent-master-ie-0-9-5-version-restic-will-not-detect-changed-files-if-mtime-is-reset-by-the-application-eg-ms-excel-and-xls-xlsx-files-on-macos/1772/8?u=fd0
 2019-11-24 15:15:19 +01:00
rawtaz
952469473f
Merge pull request #2486 from rawtaz/changelog-gitkeep 
changelog: Add .gitkeep to persist unreleased/ folder
 2019-11-24 14:59:39 +01:00
rawtaz
02108f202e
Merge pull request #2487 from rawtaz/2469-fix-diff-stats 
diff: Fix wrong bytes reported in diff stats (#2469)
 2019-11-23 12:15:28 +01:00
Leo R. Lundgren
c0a5530950 changelog: Fix typo in TEMPLATE. 2019-11-22 17:32:56 +01:00
Leo R. Lundgren
77ef92b95c diff: Fix wrong bytes reported in diff stats (#2469) 2019-11-22 17:26:01 +01:00
Leo R. Lundgren
69f75bbf70 changelog: Add .gitkeep to persist the unreleased/ folder. 2019-11-22 17:14:17 +01:00
Alexander Neumann
29b7b17491 Prepare changelog files for 0.9.6 2019-11-22 16:18:33 +01:00
Alexander Neumann
fb95426f64 Rename environment variable to AWS_DEFAULT_REGION 
This seems to be the correct name, at least the AWS cli uses it:
https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html
 2019-11-22 15:24:42 +01:00
Alexander Neumann
4cadc89ad3 Add documentation and changelog 2019-11-22 15:16:59 +01:00
rawtaz
929d2b8df3
Merge pull request #2479 from rawtaz/sing-pulitzer-group 
Accept both singular and plural for --group-by (#2330)
 2019-11-20 21:03:05 +01:00
Leo R. Lundgren
4f0682d730 Correct name of changelog file for #2321 2019-11-20 21:01:10 +01:00
Leo R. Lundgren
4d2aa18273 Accept both singular and plural for --group-by (#2330) 2019-11-20 19:25:48 +01:00
rawtaz
26d1f9f4ba
Merge pull request #2307 from gary-kim/fix-2306-password-retries 
Allow multiple retries for interactive password input
 2019-11-20 18:30:20 +01:00
Alexander Neumann
3f94f63967 Add entry to changelog 2019-06-30 21:38:10 +02:00
Fred Akalin
3ca424050f Add upper bound for t in --read-data-subset=n/t (#2304) 
* Add upper bound for t in --read-data-subset=n/t

* Add changelog entry
 2019-06-29 14:34:53 -04:00
Gary Kim
fea2464d4d
Allow multiple retries for interactive password input 
Restic used to quit if the repository password was typed incorrectly once.
Restic will now ask the user again for the repository password if typed incorrectly.
The user will now get three tries to input the correct password before restic quits.
 2019-06-13 20:11:02 +08:00
Garry McNulty
8066195e6e fs: Handle absolute pathname for --stdin-filename 
Return valid directory info from Lstat() for parent directories of the
specified filename. Previously only "/" and "." were valid directories.

Also set directory mode as this is checked by archiver.

Closes #2063
 2019-05-08 15:21:27 +02:00
Alexander Neumann
919dd2ac84 Merge pull request #2252 from restic/fix-2249 
Read fresh metadata for unmodified files
 2019-04-25 09:15:50 +02:00
Alexander Neumann
870bc5108e Merge pull request #2253 from restic/fix-2174 
Make sure timestamps are valid
 2019-04-25 09:14:43 +02:00
Alexander Neumann
a6481b3707 Merge pull request #2212 from cbane/check-ctime 
Examine file ctime when checking if files have changed.
 2019-04-25 08:58:27 +02:00
Courtney Bane
00b527fb09 Update changelog text, and add pull request link. 2019-04-24 20:54:15 -05:00
Alexander Neumann
939f3e972c node: Make sure year of all timestamps is valid 
Sometimes restic gets bogus timestamps which cannot be converted to
JSON, because the stdlib JSON encoder returns an error if the year is
not within [0, 9999]. We now make sure that we at least record _some_
timestamp and cap the year either to 0000 or 9999. Before, restic would
refuse to save the file at all, so this improves the status quo.

This fixes #2174 and #1173
 2019-04-24 16:39:36 +02:00
Alexander Neumann
4f45b14f25 Add changelog file 2019-04-24 15:17:25 +02:00
Courtney Bane
b8c2544dcb Examine file ctime when checking if files have changed. 2019-04-23 21:54:35 -05:00
Alexander Neumann
1ac560181b Prepare changelog for 0.9.5 2019-04-23 13:19:23 +02:00
Alexander Neumann
8cab0c121d Merge pull request #2221 from classmarkets/2203-password-from-stdin 
Fix reading password from stdin
 2019-04-22 20:59:59 +02:00
Alexander Neumann
5979414bcd Merge pull request #2124 from Kidswiss/tar 
Ability to dump folders to tar via stdout
 2019-04-22 20:58:59 +02:00
Simon Beck
cc8b690b52 Restore whole folder to sdtout as tar 
With this change it is possible to dump a folder to stdout as a tar. The

It can be used just like the normal dump command:

`./restic dump fa97e6e1 "/data/test/" > test.tar`

Where `/data/test/` is a a folder instead of a file.
 2019-04-22 20:44:33 +02:00
Jan Niklas Richter
3d5a0c799b Add changelog of group-by option for snapshots command 2019-04-22 14:37:58 +02:00
Peter Schultz
5715517e29 Fix reading password from stdin 
Reading the password from non-terminal stdin used io.ReadFull with a
byte slice of length 1000.

We are now using a Scanner to read one line of input, independent of its
length.

Additionally, if stdin is not a terminal, the password is read only
once instead of twice (in an effort to detect typos).

Fixes #2203

Signed-off-by: Peter Schultz <peter.schultz@classmarkets.com>
 2019-04-15 10:05:49 +02:00
Alexander Neumann
2c6ba5d9ac Merge pull request #2231 from cdhowie/issue-2224 
find command: don't abort on tree load errors
 2019-04-14 19:31:06 +02:00
Alexander Neumann
1c87d01bad Merge pull request #2220 from BenoitKnecht/s3-storage-class 
Add config option to set S3 storage class
 2019-04-13 14:54:32 +02:00
Alexander Neumann
78a3ffcfb9 Merge pull request #2185 from d3zd3z/json-forget 
Json forget
 2019-04-13 14:23:32 +02:00
Chris Howie
36f22a0feb find command: don't abort on tree load errors 2019-03-29 22:58:12 -04:00
Benoît Knecht
be23313072 changelog: Add entry for s3.storage-class option 2019-03-26 16:39:27 +01:00
Alexander Neumann
95434cff16 Merge pull request 2156 from kayrus/swift-app-cred 
Swift: introduce application credential auth support
 2019-03-16 12:17:17 +01:00
Heiko Bornholdt
d138b38f28 Add pull-2205 to changelog 2019-03-10 21:37:01 +01:00
David Brown
449c049ce9 Add changelog for #2184 2019-02-23 09:37:38 -07:00
Alexander Neumann
9f436d80e1 Add changelog file 2019-02-23 11:16:08 +01:00
kayrus
6ebcfe7c18 Swift: introduce application credential auth support 2019-02-14 14:19:05 +01:00
Alexander Neumann
56e5467096 Merge pull request 2032 from j6s/feature.case-insensitive-exclude 
Add options for case insensitive includes & excludes
 2019-02-10 12:40:33 +01:00
Alexander Neumann
5ee932a124 Add changelog for #2139 2019-02-10 12:35:41 +01:00
Alexander Neumann
fed25714a4 Merge pull request 2139 from restic/fail-zero-bytes-stdin 
Return error when reading zero byte from stdin
 2019-02-10 12:34:05 +01:00
Alexander Neumann
97aafc1eec Add changelog file for #1944 2019-02-10 12:31:18 +01:00
Alexander Neumann
77ab10d401 Create changelog dir for 0.9.4 2019-01-06 21:39:56 +01:00
Alexander Neumann
2996c110f1 Merge pull request 2017 from kylewlacy/fuse_default_permissions_option 
mount: Add default-permissions flag to set FUSE option
 2019-01-06 21:19:27 +01:00
Alexander Neumann
4609b5c24d Merge pull request 2094 from juergenhoetzel/password-command 
Add support for reading password from external command
 2019-01-06 21:14:52 +01:00
Alexander Neumann
830511460a mount: Turn on DefaultPermissions for --allow-other` 
This commit changes the logic slightly: checking the permissions in the
fuse mount when nobody else besides the current user can access the fuse
mount does not sense. The current user has access to the repo files in
addition to the password, so they can access all data regardless of what
the fuse mount does.

Enabling `--allow-root` allows the root user to access the files in the
fuse mount, for this user no permission checks will be done anyway.

The code now enables `DefaultPermissions` automatically when
`--allow-other` is set, it can be disabled with
`--no-default-permissions` to restore the old behavior.
 2019-01-06 20:55:49 +01:00
Alexander Neumann
0dc3648416 Add documentation for --password-command 2019-01-06 20:41:38 +01:00
Alexander Neumann
1f246c5309 Merge pull request 2090 from plumbeo/within-hours 
Increase the granularity of the "keep within" snapshot retention policy
 2019-01-06 15:04:55 +01:00
Alexander Neumann
6f69ae1b8d Add docs, changelog file 2019-01-06 14:52:20 +01:00
Kyle Lacy
d4ff5b6bf4 Add changelog entry about "DefaultPermissions" change 2018-11-26 21:16:01 -08:00
plumbeo
3edc723bf0 Update 'restic forget --keep-within' documentation and add changelog entry 2018-11-26 14:27:48 +01:00
Alexander Neumann
56af0ce370 Add entry to changelog 2018-11-25 19:07:42 +01:00
Alexander Neumann
2434ab2106 Merge pull request 2098 from cdhowie/key-hinting 
Add key hinting (#2097)
 2018-11-25 17:52:54 +01:00
Chris Howie
1688713400 Add key hinting (#2097) 2018-11-25 09:13:18 -05:00
Johannes Hertenstein
879f6e0c81 Add changelog file for insensitive includes & excludes 2018-11-25 13:32:16 +01:00
Alexander Neumann
6dc7cca597 Merge pull request 2086 from frebib/fix-2085 
Allow --files-from multiple times
 2018-11-25 13:21:29 +01:00
Alexander Neumann
d32c7c2aba Rework changelog for #2085 2018-11-25 12:55:03 +01:00
Joe Groocock
09e9b74cbd Allow --files-from multiple times 
Fixes https://github.com/restic/restic/issues/2085
 2018-11-25 12:55:03 +01:00
Alexander Neumann
d53595e43c Merge pull request 2088 from garrmcnu/stats-latest-host 
stats: Add shorthand flag `-H` for host name filter
 2018-11-24 19:05:53 +01:00