1576 commits

Author	SHA1	Message	Date
Michael Eischer	55bea6e7a6	filter: Fix crash for '**' pattern	2021-05-14 23:50:31 +02:00
MichaelEischer	6d8ceefd67	Merge pull request #3373 from greatroar/simplify-limiter ... Simplify internal/limiter	2021-05-14 20:27:31 +02:00
MichaelEischer	64b00d28b1	Merge pull request #3345 from greatroar/sftp-enospc ... Check for ENOSPC and remove broken files in SFTP	2021-05-13 20:09:38 +02:00
greatroar	ae170e2b38	Simplify internal/limiter	2021-04-24 11:54:43 +02:00
Michael Eischer	7cb8ea69ba	Add test to mintty pipe detection	2021-04-11 20:02:09 +02:00
Michael Eischer	80564a9bc9	Properly detect mintty output redirection ... mintty on windows always uses pipes to connect stdout between processes and for the terminal output. The previous implementation always assumed that stdout connected to a pipe means that stdout is displayed on a mintty terminal. However, this detection breaks when using pipes to connect processes and for powershell which uses pipes when redirecting to a file. Now the pipe filename is queried and matched against the pattern used by msys / cygwin when connected to the terminal. In all other cases assume that a pipe is just a regular pipe.	2021-04-11 20:02:09 +02:00
Michael Eischer	5e6af77b7a	Unify interactive terminal detection code ... Previously the progress bar / status update interval used stdoutIsTerminal to determine whether it is possible to update the progress bar or not. However, its implementation differed from the detection within the backup command which included additional checks to detect the presence of mintty on Windows. mintty behaves like a terminal but uses pipes for communication. This adds stdoutCanUpdateStatus() which calls the same terminal detection code used by backup. This ensures that all commands consistently switch between interactive and non-interactive terminal mode. stdoutIsTerminal() now also returns true whenever stdoutCanUpdateStatus() does so. This is required to properly handle the special case of mintty.	2021-04-11 20:02:09 +02:00
MichaelEischer	cc254dfefe	Merge pull request #3362 from greatroar/darwin-preallocate ... Use FcntlFstore to preallocate on Mac	2021-04-10 22:47:56 +02:00
greatroar	23531be272	Use FcntlFstore to preallocate on Mac	2021-04-10 16:54:07 +02:00
rawtaz	74c0607c92	Merge pull request #3319 from MichaelEischer/skip-prealloc-test ... restorer: Skip preallocate test if not supported by the filesystem	2021-04-07 18:59:06 +02:00
greatroar	dc88ca79b6	Handle lack of space and remove broken files in SFTP backend	2021-03-27 15:02:14 +01:00
Alexander Neumann	88a23521dd	Merge pull request #3327 from MichaelEischer/fix-s3-sanity-check ... s3: Fix sanity check	2021-03-11 13:13:46 +01:00
greatroar	187a77fb27	Upgrade pkg/sftp to 1.13 and simplify SFTP.IsNotExist	2021-03-10 21:05:24 +01:00
Michael Eischer	2a9f0f19b6	s3: Fix sanity check ... The sanity check shouldn't replace the error message if there is already one.	2021-03-08 20:23:57 +01:00
Michael Eischer	f70aca6f6f	restorer: Skip preallocate test if not supported by the filesystem	2021-03-04 20:33:46 +01:00
Michael Eischer	a293fd9aef	local: Ignore files in intermediate folders ... For example the data folder of a repository might contain hidden files which caused list operations to fail.	2021-02-27 13:47:55 +01:00
Alexander Neumann	9985368d46	Merge pull request #3255 from MichaelEischer/restorer-check-error ... restorer: Check dropped error	2021-02-03 16:10:46 +01:00
Alexander Neumann	2dd592a06c	Merge pull request #3254 from lorenz/allow-http2 ... Allow HTTP/2	2021-02-02 21:08:20 +01:00
Michael Eischer	47faf69230	restorer: Check dropped error	2021-01-31 18:06:28 +01:00
Alexander Weiss	5e22ae10f1	Add error handling for fileRestorer	2021-01-31 14:22:57 +01:00
Alexander Weiss	573221aa40	Add failing test for fileRestorer	2021-01-31 13:40:42 +01:00
Lorenz Brun	5427119205	Allow HTTP/2	2021-01-31 02:44:30 +01:00
Michael Eischer	e0867c9682	backend: try to cleanup test leftovers	2021-01-30 21:23:20 +01:00
Michael Eischer	f740b2fb23	mem: check upload length before storing upload	2021-01-30 21:23:20 +01:00
Alexander Neumann	0e5f2fff71	Merge pull request #3243 from restic/fix-scanner-overlap ... backup: Fix total size for overlapping targets	2021-01-30 21:17:21 +01:00
Alexander Neumann	04ca69cc78	Address issues reported by golint	2021-01-30 20:45:57 +01:00
Alexander Neumann	f867e65bcd	Fix issues reported by staticcheck	2021-01-30 20:43:53 +01:00
Alexander Neumann	0858fbf6aa	Add more error handling	2021-01-30 20:19:47 +01:00
Alexander Neumann	aef3658a5f	Address review comments	2021-01-30 20:02:37 +01:00
Alexander Neumann	200f09522d	Add more error checks	2021-01-30 20:02:37 +01:00
Alexander Neumann	3c753c071c	errcheck: More error handling	2021-01-30 20:02:37 +01:00
Alexander Neumann	16313bfcc9	errcheck: Add error check for MergeFinalIndexes()	2021-01-30 20:02:37 +01:00
Alexander Neumann	75f53955ee	errcheck: Add error checks ... Most added checks are straight forward.	2021-01-30 20:02:37 +01:00
Michael Eischer	a53778cd83	rest: handle dropped error in save operation	2021-01-30 19:25:04 +01:00
Michael Eischer	8a486eafed	gs: Don't drop error when finishing upload ... The error returned when finishing the upload of an object was dropped. This could cause silent upload failures and thus data loss in certain cases. When a MD5 hash for the uploaded blob is specified, a wrong hash/damaged upload would return its error via the Close() whose error was dropped.	2021-01-30 13:31:32 +01:00
Alexander Neumann	cdd704920d	azure: Pass data length to Azure libray ... The azureAdapter was used directly without a pointer, but the Len() method was only defined with a pointer receiver (which means Len() is not present on a azureAdapter{}, only on a pointer to it).	2021-01-29 21:08:41 +01:00
Alexander Neumann	bbdf18c4a2	Merge pull request #3176 from MichaelEischer/backend-content-length ... Pass upload size to backends and sanity check it	2021-01-29 20:33:44 +01:00
Michael Eischer	1f583b3d8e	backend: test that incomplete uploads fail	2021-01-29 13:51:53 +01:00
Michael Eischer	c73316a111	backends: add sanity check for the uploaded file size ... Bugs in the error handling while uploading a file to the backend could cause incomplete files, e.g. https://github.com/golang/go/issues/42400 which could affect the local backend. Proactively add sanity checks which will treat an upload as failed if the reported upload size does not match the actual file size.	2021-01-29 13:51:51 +01:00
Michael Eischer	4526d5d197	swift: explicitly pass upload size to library ... This allows properly setting the content-length which could help the server-side to detect incomplete uploads.	2021-01-29 13:50:46 +01:00
Michael Eischer	dca9b6f5db	azure: explicitly pass upload size ... Previously the fallback from the azure library was to read the whole blob into memory and use that to determine the upload size.	2021-01-29 13:50:46 +01:00
Alexander Neumann	5c617859ab	backup/scanner: Fix total size for overlapping targets ... Before, the scanner would could files twice if they were included in the list of backup targets twice, e.g. `restic backup foo foo/bar` would could the file `foo/bar` twice. This commit uses the tree structure from the archiver to run the scanner, so both parts see the same files.	2021-01-29 11:31:36 +01:00
Alexander Neumann	81211750ba	archiver/tree: Introduce functions Leaf() and NodeNames()	2021-01-29 11:11:28 +01:00
greatroar	6bd8a2faaa	backup: Add --ignore-ctime option and document change detection	2021-01-28 23:42:10 +01:00
MichaelEischer	43cb26010a	Merge pull request #3242 from greatroar/fprintln ... internal/ui/termstatus: Use Fprintln to get a newline	2021-01-28 20:34:06 +01:00
Alexander Neumann	4d40c70214	Merge pull request #3211 from MichaelEischer/sftp-speedup ... Speed-up caching/pack download via SFTP	2021-01-28 14:16:55 +01:00
Alexander Neumann	6aa7e9f9c6	Merge pull request #3174 from MichaelEischer/parallelize-lock-loading ... Parallelize lock file loading	2021-01-28 13:52:12 +01:00
Alexander Neumann	bdfedf1f5b	Merge pull request #3173 from MichaelEischer/unify-index-loading ... Unify index loading	2021-01-28 13:50:42 +01:00
greatroar	b9cfe6f68a	internal/ui/termstatus: Use Fprintln to get a newline	2021-01-28 13:30:10 +01:00
Michael Eischer	68608a89ad	restic: add comment about StreamTrees shutdown	2021-01-28 11:10:50 +01:00
Michael Eischer	ddb7697d29	restic: Test progress reporting of StreamTrees	2021-01-28 11:10:50 +01:00
Michael Eischer	313ad0e32f	progress/counter: Fix test for final report call	2021-01-28 11:10:50 +01:00
Michael Eischer	e2b0072441	check: add progress bar to the tree structure check	2021-01-28 11:10:50 +01:00
Michael Eischer	505f8a2229	progress/counter: Support updating the progress bar maximum	2021-01-28 11:10:47 +01:00
Michael Eischer	eda8c67616	restic: let FindUsedBlobs handle multiple snapshots at once	2021-01-28 11:08:43 +01:00
Michael Eischer	258ce0c1e5	parallel: report progress for StreamTrees ... This assigns an id to each tree root and then keeps track of how many tree loads (i.e. trees referenced for the first time) are pending per tree root. Once a tree root and its subtrees were fully processed there are no more pending tree loads and the tree root is reported as processed.	2021-01-28 11:08:43 +01:00
Michael Eischer	f2a1b125cb	restic: Actually parallelize FindUsedBlobs	2021-01-28 11:08:43 +01:00
Michael Eischer	6e03f80ca2	check: Split the parallelized tree loader into a reusable component ... The actual code change is minimal	2021-01-28 11:08:43 +01:00
Michael Eischer	1d7bb01a6b	check: Cleanup tree loading and switch to use errgroup ... The helper methods are now wired up in the Structure method.	2021-01-28 11:08:43 +01:00
Alexander Neumann	a4689eb3b9	Merge pull request #3199 from MichaelEischer/non-interactive-counter ... Don't print progress on non-interactive terminals	2021-01-28 10:53:38 +01:00
Alexander Neumann	c5a66e9181	ui: Simlify channel receive	2021-01-28 10:42:02 +01:00
Michael Eischer	678e75e1c2	sftp: enforce use of optimized upload method ... ReadFrom was already used by Save before, this just ensures that this won't accidentally change in the future.	2021-01-03 22:23:53 +01:00
Michael Eischer	6b5b29dbee	limiter: add unit tests	2021-01-03 22:23:53 +01:00
Michael Eischer	f35f2c48cd	limiter: support WriteTo in LimitBackend for read rate limiting	2021-01-03 22:23:53 +01:00
Michael Eischer	bcb852a8d0	hashing: support WriteTo in the reader	2021-01-03 22:23:53 +01:00
Michael Eischer	69d5b4c36b	restorer: lower-case variable name	2021-01-03 13:55:59 +01:00
Alexander Weiss	34a33565c8	Fix loadBlob in filerestorer	2021-01-01 08:06:04 +01:00
Alexander Weiss	7409225fa8	Add filerestorer test where only parts of pack are used	2021-01-01 07:24:46 +01:00
Alexander Weiss	07b3f65a6f	filesrestorer: Re-use buffer	2021-01-01 07:24:46 +01:00
Alexander Weiss	3e0acf1395	restore: Don't save (part of) pack in memory	2021-01-01 07:24:46 +01:00
Alexander Neumann	8b84c96d9d	Merge pull request #3204 from MichaelEischer/archiver-tomb-race ... archiver: fix race condition during worker startup	2020-12-30 20:04:20 +01:00
Michael Eischer	debc4a3a99	archiver: fix race condition during worker startup ... When the tomb is created with a canceled context, then the workers started via `t.Go` exist nearly immediately. Once for the first time all started goroutines have been stopped, it is not allowed to issue further calls to `t.Go`. This is a problem when the started goroutines exit immediately, as for example the first goroutine might already have stopped before starting the second one, which is not allowed as once the first goroutines has stopped no goroutines were running. To fix this race condition the startup and main task of the archiver now also run within a `t.Go` function. This also allows unifying the error handling as it is no longer necessary to distinguish between errors returned by the workers or the saveTree processing. The tomb now just returns the first error encountered, which should also be the most descriptive one.	2020-12-30 17:31:22 +01:00
MichaelEischer	31e56f1ad5	Merge pull request #3197 from SkYNewZ/fix/3183 ... Fix tag handling for multiple tag lists	2020-12-29 18:44:38 +01:00
Alexander Neumann	b2efa0af39	Merge pull request #3164 from MichaelEischer/improve-context-cancel ... Improve context cancel handling in archiver and backends	2020-12-29 17:03:42 +01:00
Michael Eischer	cff4955a48	ui: remove dead struct member	2020-12-29 16:32:18 +01:00
Michael Eischer	92da5168e1	ui: force backup progress update on signal	2020-12-29 16:32:18 +01:00
Michael Eischer	34afc93ddc	ui/progress: extract signal handling into own package	2020-12-29 16:32:18 +01:00
Michael Eischer	023eea6463	ui: don't shorten non-interactive progress output	2020-12-29 16:32:18 +01:00
Michael Eischer	684600cf42	ui: update status for the backup command on non-interactive terminals ... Allow the backup command to print status on non-interactive terminals. The output is disabled by setting a MinUpdatePause == 0.	2020-12-29 16:03:43 +01:00
Michael Eischer	13ce981794	ui: cleanup backup status shutdown	2020-12-29 16:03:43 +01:00
Michael Eischer	c2ef049f1b	ui/progress: don't print progress on non-interactive terminals ... This reverts to the old behavior of not printing progress updates on non-interactive terminals. It was accidentally changed in #3058.	2020-12-29 16:03:43 +01:00
Quentin Lemaire	334d8ce724	feat(tags): Create Flatten() method	2020-12-29 10:59:46 +01:00
greatroar	3b09ae9074	AIX port	2020-12-29 01:35:01 +01:00
greatroar	18531e3d6f	Portability fixes to internal/restic ... syscall.Mknod is not available on AIX.	2020-12-29 01:35:01 +01:00
Michael Eischer	d0ca8fb0b8	backend: test that a canceled context prevents RetryBackend operations	2020-12-28 21:06:47 +01:00
Michael Eischer	08b7f2b58d	archiver: test that context canceled error is not dropped	2020-12-28 21:06:47 +01:00
Michael Eischer	e483b63c40	retrybackend: Fail operations when context is already canceled ... Depending on the used backend, operations started with a canceled context may fail or not. For example the local backend still works in large parts when called with a canceled context. Backends transfering data via http don't work. It is also not possible to retry failed operations in that state as the RetryBackend will abort with a 'context canceled' error. Ensure uniform behavior of all backends by checking for a canceled context by checking for a canceled context as a first step in the RetryBackend. This ensures uniform behavior across all backends, as backends are always wrapped in a RetryBackend.	2020-12-28 21:06:47 +01:00
Michael Eischer	fc60b560ba	archiver: Let saveTree report a canceled context as an error ... If the context was canceled then saveTree might receive a treeID or not depending on the timing. This could cause saveTree to incorrectly return a nil treeID as valid. Fix this always returning an error when the context was canceled in the meantime.	2020-12-28 21:06:47 +01:00
Michael Eischer	736e964317	archiver: Don't loose error if background context is canceled ... A canceled background context lets the blob/tree/fileSavers exit without reporting an error. The error handling previously replaced a 'context canceled' error received by the main backup method with the error reported by the savers. However, in case of a canceled background context that error is nil, causing restic to loose the error and save a snapshot with a nil tree.	2020-12-28 21:06:47 +01:00
DRON-666	332b1896d1	Some options fixes ... Add tests for bool type. Fix subtle bug in TestOptionsApplyInvalid. Fix options list formatting.	2020-12-23 23:26:04 +03:00
MichaelEischer	cb6b0f6255	Merge pull request #3181 from tWido/feature-no-retry-permission ... Don't retry when "Permission denied" occurs in local backend	2020-12-23 20:24:29 +01:00
MichaelEischer	1e73aac610	Merge pull request #3179 from aawsome/check-filesize ... check: Remove filesize counter	2020-12-23 20:22:02 +01:00
Alexander Weiss	2a1add7538	check: remove file size counter	2020-12-23 02:34:31 +01:00
tWido	7dab113035	Don't retry when "Permission denied" occurs in local backend	2020-12-22 23:41:12 +01:00
Michael Eischer	de99207046	repository: tweak comment for packs method	2020-12-22 23:01:58 +01:00
Alexander Weiss	68b74e359e	Count packs directly in RebuildIndexFiles	2020-12-22 23:01:58 +01:00
Michael Eischer	b9f5d3fe13	repository: Add test for ForAllIndexes	2020-12-22 22:36:18 +01:00
Michael Eischer	a12c5f1d37	repository: move otherwise unused LoadIndex to tests	2020-12-22 22:36:18 +01:00
Michael Eischer	24474a36f4	repository: deduplicate index loading implementation	2020-12-22 22:36:18 +01:00
Michael Eischer	96904f8972	check: extract parallel index loading	2020-12-22 22:36:18 +01:00
Alexander Neumann	6c514adb8a	ui/progress: Use mutex instead of atomic ... The counter value needs to be aligned to 64 bit in memory for the atomic functions to work on some platform (such as 32 bit ARM). The atomic package says in its documentation: > These functions require great care to be used correctly. Except for > special, low-level applications, synchronization is better done with > channels or the facilities of the sync package. This commit replaces the atomic functions with a simple sync.Mutex, so we don't have to care about alignment.	2020-12-22 21:03:27 +01:00
DRON-666	83b10dbb12	Deduplicate dumper closing logic	2020-12-19 02:42:46 +03:00
DRON-666	e136dd8696	Deduplicate test code	2020-12-19 02:06:54 +03:00
DRON-666	33adb58817	Minor fixes and linter suggestions	2020-12-19 02:04:17 +03:00
DRON-666	89ab6d557e	dump: Add zip dumper	2020-12-19 01:09:47 +03:00
DRON-666	0256f95994	dump: Split tar and walk logic	2020-12-19 01:09:47 +03:00
Michael Eischer	34b6130a0e	restic: parallelize lock file loading	2020-12-18 20:46:16 +01:00
greatroar	66d904c905	Make invalid handles permanent errors	2020-12-17 12:47:53 +01:00
greatroar	746dbda413	Mark "ssh exited" errors in SFTP as permanent	2020-12-17 12:43:09 +01:00
greatroar	f7784bddb3	Don't retry when "no space left on device" in local backend ... Also adds relevant documentation to the restic.Backend interface.	2020-12-17 12:43:09 +01:00
Alexander Neumann	e96677cafb	Merge pull request #3158 from MichaelEischer/support-swift-auth-id-variables ... swift: Add support for id based keystone v3 auth parameters	2020-12-12 16:27:38 +01:00
Michael Eischer	1d69341e88	swift: Add support for id based keystone v3 auth parameters ... This adds support for the following environment variables, which were previously missing: OS_USER_ID User ID for keystone v3 authentication OS_USER_DOMAIN_ID User domain ID for keystone v3 authentication OS_PROJECT_DOMAIN_ID Project domain ID for keystone v3 authentication OS_TRUST_ID Trust ID for keystone v3 authentication	2020-12-11 19:22:34 +01:00
Alexander Neumann	36c5d39c2c	Fix issues reported by semgrep	2020-12-11 09:41:59 +01:00
Alexander Neumann	7facc8ccc1	Merge pull request #2505 from aawsome/fix-repo-configfile ... Fix repo configfile	2020-12-07 07:52:37 +01:00
Alexander Neumann	ba31c6fdaa	Merge pull request #3150 from MichaelEischer/fix-windows-redir-output ... termstatus: Fix canUpdateStatus detection for redirected output on windows	2020-12-06 21:17:27 +01:00
Alexander Neumann	b58799d83a	Merge pull request #3152 from MichaelEischer/fix-backup-background-hang ... backup: Fix shutdown hang when running in the background on linux	2020-12-06 21:16:00 +01:00
Alexander Neumann	0d5b764f90	Merge pull request #3130 from aawsome/snapshots-parallel ... Make loading snapshots parallel	2020-12-06 21:08:18 +01:00
Michael Eischer	b48f579530	termstatus: Fix canUpdateStatus detection for redirected output ... The canUpdateStatus check was simplified in #2608, but it accidentally flipped the condition. The correct check is as follows: If the output is a pipe then restic probably runs in mintty/cygwin. In that case it's possible to update the output status. In all other cases it isn't. This commit inverts to condition again to offer the previous and correct behavior.	2020-12-06 19:02:42 +01:00
Michael Eischer	401ef92c5f	backup: Fix shutdown hang when running in the background ... On shutdown the backup commands waits for the terminal output goroutine to stop. However while running in the background the goroutine ignored the canceled context.	2020-12-06 18:53:41 +01:00
Alexander Weiss	e329623771	Remove LoadAllSnapshots	2020-12-06 05:22:27 +01:00
Alexander Weiss	26f85779be	Parallelize ForAllSnapshots	2020-12-06 05:09:58 +01:00
Alexander Weiss	5b9ee56335	Add ForAllSnapshots	2020-12-06 05:04:21 +01:00
Michael Eischer	d05c88a5d6	fuse: Properly set uid/gid for directories ... In #2584 this was changed to use the uid/gid of the root node. This would be okay for the top-level directory of a snapshot, however, this change also applied to normal directories within a snapshot. This change reverts the problematic part and adds a test that directory attributes are represented correctly.	2020-11-30 23:41:49 +01:00
MichaelEischer	f2959127b6	Merge pull request #3065 from greatroar/local-subdirs ... Don't recurse in local backend's List if not required	2020-11-29 19:03:59 +01:00
Alexander Weiss	aa7a5f19c2	Use BlobHandle in index methods	2020-11-22 20:41:12 +01:00
Alexander Weiss	e3013271a6	Harmonize naming	2020-11-22 20:41:12 +01:00
Alexander Weiss	92bd448691	Make BlobHandle substruct of Blob	2020-11-22 20:41:10 +01:00
Alexander Weiss	a851c53cbe	Use PackSize in checker	2020-11-21 22:13:54 +01:00
Alexander Weiss	ce5d630681	Add MasterIndex.PackSize()	2020-11-21 22:13:54 +01:00
Alexander Weiss	c3ddde9e7d	Return hdrSize in ListPack	2020-11-21 22:13:54 +01:00
greatroar	8e213e82fc	backend/local: replace fs.Walk with custom walker ... This code is more strict in what it expects to find in the backend: depending on the layout, either a directory full of files or a directory full of such directories.	2020-11-19 16:46:42 +01:00
Alexander Neumann	75eff92b56	Merge pull request #3107 from eleith/do-not-require-bucket-permissions-for-init ... do not require gs bucket permissions to init repository	2020-11-18 16:53:45 +01:00
eleith	a24e986b2b	do not require gs bucket permissions to init repository ... a gs service account may only have object permissions on an existing bucket but no bucket create/get permissions. these service accounts currently are blocked from initialization a restic repository because restic can not determine if the bucket exists. this PR updates the logic to assume the bucket exists when the bucket attribute request results in a permissions denied error. this way, restic can still initialize a repository if the service account does have object permissions fixes: https://github.com/restic/restic/issues/3100	2020-11-18 06:14:11 -08:00
rawtaz	1b6e8c888f	Merge pull request #3102 from tofran/remove-rclone-drive-use-tras-default-param ... Remove `--drive-use-trash=false` from rclone param	2020-11-17 11:21:20 +01:00
Michael Eischer	1f43cac12d	check: Only track data blobs when unused blobs should be reported ... This improves the memory usage of check a lot as it now only has to track tree blobs when run using the default parameters.	2020-11-15 18:43:07 +01:00
Michael Eischer	6da66c15d8	check: Simplify referenced blob tracking ... The result is identical as long as the context in not canceled. However, in that case the result is incomplete anyways.	2020-11-15 18:42:55 +01:00
Michael Eischer	3500f9490c	check: Simplify blob status tracking ... UnusedBlobs now directly reads the list of existing blobs from the repository index. This removes the need for the blobStatusExists flag, which in turn allows converting the blobRefs map into a BlobSet.	2020-11-15 18:42:42 +01:00
Michael Eischer	b8c7543a55	check: Merge 'size could not be found' and 'not found in index' errors ... By construction these two errors always show up in pairs: 'size could not be found' is printed when the blob is not found in the repository index. That blob is also part of the `blobs` array. Later on, check iterates over that array and checks whether the blob is marked as existing. Which cannot be the case as that mark is generated by iterating over the repository index. The merged warning no longer reports the blob index within a file. That information could also be derived by printing the affected tree using `cat` and searching for the blob.	2020-11-15 18:41:50 +01:00
Alexander Neumann	3c0c0c132b	Merge pull request #3006 from aawsome/new-rebuild-index ... Reimplement rebuild-index and remove /internal/index	2020-11-15 17:48:43 +01:00
Alexander Neumann	9968220652	Merge pull request #2850 from greatroar/packer-malloc ... Decrease allocation rate in internal/pack	2020-11-15 17:19:49 +01:00
greatroar	ab2b7d7f9a	Decrease allocation rate in internal/pack ... internal/repository benchmark results: name old time/op new time/op delta PackerManager-8 179ms ± 1% 181ms ± 1% +0.78% (p=0.009 n=10+10) name old speed new speed delta PackerManager-8 294MB/s ± 1% 292MB/s ± 1% -0.77% (p=0.009 n=10+10) name old alloc/op new alloc/op delta PackerManager-8 91.3kB ± 0% 72.2kB ± 0% -20.92% (p=0.000 n=9+7) name old allocs/op new allocs/op delta PackerManager-8 1.38k ± 0% 0.76k ± 0% -45.20% (p=0.000 n=10+7)	2020-11-15 16:51:47 +01:00
greatroar	9a8a2cae4c	Move pack testing logic to test file	2020-11-15 16:45:49 +01:00
Alexander Weiss	9607cad267	Remove internal/index	2020-11-15 07:05:09 +01:00
Alexander Weiss	30b6a0878a	Reimplement rebuild-index	2020-11-15 07:05:09 +01:00
Alexander Weiss	187c8fb259	Parallelize MasterIndex.Save()	2020-11-15 07:05:09 +01:00
Alexander Weiss	1ec628ddf5	Add extraObsolete to MasterIndex.Save	2020-11-15 07:05:09 +01:00
Alexander Weiss	5898cb341f	Use CreateIndexFromPacks() in test	2020-11-15 07:05:05 +01:00
Alexander Weiss	43732bb885	Add CreateIndexFromPacks()	2020-11-15 07:04:51 +01:00
greatroar	c45f8ee075	Replace restic.statT interface by concrete types ... name old time/op new time/op delta NodeFillUser-8 1.81µs ± 9% 1.50µs ± 5% -17.07% (p=0.000 n=19+20) NodeFromFileInfo-8 1.76µs ± 4% 1.49µs ± 6% -15.63% (p=0.000 n=20+19) name old alloc/op new alloc/op delta NodeFillUser-8 496B ± 0% 352B ± 0% -29.03% (p=0.000 n=20+20) NodeFromFileInfo-8 496B ± 0% 352B ± 0% -29.03% (p=0.000 n=20+20) name old allocs/op new allocs/op delta NodeFillUser-8 3.00 ± 0% 2.00 ± 0% -33.33% (p=0.000 n=20+20) NodeFromFileInfo-8 3.00 ± 0% 2.00 ± 0% -33.33% (p=0.000 n=20+20)	2020-11-14 23:23:26 +01:00
MichaelEischer	3601a9b6cd	Merge pull request #2690 from SkYNewZ/master ... Fix #2688: Handle comma-separated list tags when using backup command	2020-11-14 23:19:42 +01:00
MichaelEischer	333c5a19d4	Merge pull request #3082 from aawsome/check-sizes ... Check: check sizes of packs from index and packheader	2020-11-14 22:37:42 +01:00
Quentin Lemaire	ae441d3134	fix(backup): Switch tags cobra type to handle comma-separated list	2020-11-14 15:48:56 +00:00
Alexander Weiss	17bb77b1f9	check: Also check blob length and offset	2020-11-14 00:42:49 +01:00
Alexander Weiss	80dcfca191	check: Check sizes computed from index and pack header	2020-11-14 00:42:49 +01:00
tofran	94a154c7ca	Remove --drive-use-trash=false from rclone param ... Google drive trash retention policy changed making this no longer a good default a go Issue #3095	2020-11-13 22:58:48 +00:00
Alexander Neumann	04d1983800	Merge pull request #3090 from fgma/vss-fix-386 ... vss: fix DeleteSnapshots() and GetSnapshotProperties() on 386	2020-11-13 21:30:34 +01:00
Alexander Neumann	8ad9f88993	helpers: Improve error message	2020-11-12 20:38:31 +01:00
fgma	f9223cd827	vss: fix DeleteSnapshots() and GetSnapshotProperties() on 386	2020-11-12 19:31:00 +01:00
rawtaz	dfb9326b1b	Merge pull request #3085 from LordGaav/s3-list-objects-v1-flag ... Extended option to select V1 API for ListObjects on S3 backend	2020-11-11 20:52:44 +01:00
Alexander Neumann	4a0b7328ec	s3: Remove dots for config description	2020-11-11 20:20:35 +01:00
greatroar	63e32c44c0	Improve error reporting from restic diff ... Instead of a stacktrace, restic diff 111 222 now reports: Fatal: no matching ID found for prefix "111"	2020-11-11 16:40:40 +01:00
Nick Douma	829959390a	Provide UseV1 parameter to minio.ListObjectsOptions based on s3.list-objects-v1	2020-11-11 11:54:38 +01:00
Nick Douma	ccd55d529d	Add s3.list-objects-v1 extended option and default to false	2020-11-11 11:54:36 +01:00
Nick Douma	4ddcc17135	Add support for boolean extended options	2020-11-11 11:54:27 +01:00
MichaelEischer	407843c5f9	Merge pull request #3034 from hoyho/gh_master ... bugfix: omit ENOTDATA for extended attributes	2020-11-09 22:38:49 +01:00
MichaelEischer	46d31ab86d	Merge pull request #3058 from greatroar/counter ... Replace restic.Progress with new progress.Counter (fixes two race conditions)	2020-11-09 22:19:09 +01:00
Alexander Neumann	c986823d3f	Merge pull request #3048 from aawsome/check-pack-index ... check: check index for packs that are read	2020-11-09 20:12:32 +01:00
Alexander Weiss	239931578c	check: check index for packs that are read	2020-11-09 17:28:14 +01:00
hoyho	9df52327cc	bugfix: omit ENOTDATA for extended attributes ... Signed-off-by: hoyho <luohaihao@gmail.com>	2020-11-10 00:20:34 +08:00
greatroar	21b787a4d1	Stop Counters where they're constructed and started	2020-11-09 13:03:31 +01:00
greatroar	ddca699cd2	Replace restic.Progress with new progress.Counter ... This fixes two race conditions while cleaning up the code.	2020-11-09 12:12:35 +01:00
Alexander Neumann	3bf43d7951	Merge pull request #2475 from restic/use-github-actions ... Use GitHub Actions for CI	2020-11-08 17:30:12 +01:00
Alexander Neumann	5cf42884c8	Run tests on GitHub Actions	2020-11-08 16:04:57 +01:00
Alexander Weiss	826cfa0533	fix context in archiver tests	2020-11-08 08:24:24 +01:00
Alexander Weiss	fef408a8bd	Return context error in mem backend	2020-11-08 00:05:53 +01:00
greatroar	a2d4209322	Don't recurse in local backend's List if not required ... Due to the return if !isFile, the IsDir branch in List was never taken and subdirectories were traversed recursively. Also replaced isFile by an IsRegular check, which has been equivalent since Go 1.12 (golang/go@a2a3dd00c9).	2020-11-07 08:54:13 +01:00
Alexander Weiss	fd33030556	Use in-memory index to rebuild index in prune	2020-11-06 20:23:30 +01:00
Alexander Weiss	38cc4393f6	Add Masterindex.Save(); Add Index.Packs()	2020-11-06 20:23:30 +01:00
Alexander Weiss	aaf1c44362	Fix #3062	2020-11-05 17:05:42 +01:00
Ivan Andreev	ab2790d9de	Fix http2 stream reset between restic and rest backends #3014	2020-11-05 15:57:40 +03:00
Nick Craig-Wood	86b5d8ffaa	s3: add bucket-lookup parameter to select path or dns style bucket lookup ... This is to enable restic working with Alibaba cloud Fixes #2528	2020-11-05 12:20:10 +01:00
Alexander Neumann	ae5302c7a8	Add comment that keepBlobs is modified	2020-11-05 10:33:38 +01:00
Alexander Neumann	866a52ad4e	Remove unneeded seek ... The file returned from DownloadAndHash() is already seeked to the start of the file.	2020-11-05 10:31:49 +01:00
Alexander Neumann	a4507610a0	Fix typo	2020-11-05 10:31:49 +01:00
Alexander Neumann	7def2d8ea7	Use a non-constant seed	2020-11-05 10:31:49 +01:00
Alexander Neumann	ee0112ab3b	Clarify message about expected error	2020-11-05 10:31:49 +01:00
Michael Eischer	b373f164fe	prune: Parallelize repack command	2020-11-05 10:31:49 +01:00
Alexander Neumann	5144141321	Merge pull request #2718 from aawsome/new-cleanup-command ... Reimplementation of prune	2020-11-05 10:12:19 +01:00
fgma	916b2d303b	vss: fix potential crash (not reachable in restic) (#3045) ... HasSufficientPrivilegesForVSS() now returns an error	2020-11-04 22:14:18 +01:00
Alexander Neumann	a06f5c28c0	Merge pull request #3055 from greatroar/cleanup ... Don't separately allocate sync.{Mutex,Once} if not necessary	2020-11-04 11:11:21 +01:00
greatroar	c2f3eee5af	Don't separately allocate sync.{Mutex,Once} if not necessary ... Separate allocation of synchronization devices suggests they're shared between objects, but they're not.	2020-11-04 09:55:43 +01:00
Alexander Weiss	b2f5381737	Make realistic forget --prune --dryrun	2020-11-03 16:42:21 +01:00
Alexander Weiss	ce7d613749	Add Blob.Handle()	2020-11-03 16:42:21 +01:00
Alexander Weiss	581d90cf91	Make some pack parameters public	2020-11-03 16:42:21 +01:00
Alexander Neumann	9a88fb253b	Merge pull request #3051 from greatroar/sanitize-env ... Sanitize environment before starting backend processes (rclone, ssh)	2020-11-02 21:18:57 +01:00
Alexander Neumann	f14436953a	Merge pull request #3050 from greatroar/widechars ... Fix string truncation in ui/termstatus	2020-11-02 16:58:51 +01:00
greatroar	11fbaaae9a	Sanitize environment before starting backend processes (rclone, ssh) ... The restic security model includes full trust of the local machine, so this should not fix any actual security problems, but it's better to be safe than sorry. Fixes #2192.	2020-11-02 16:41:23 +01:00
greatroar	3ed84ff0c6	Fix string truncation in ui/termstatus ... Fixes #3046.	2020-11-02 12:50:49 +01:00
Alexander Neumann	8e965ed4eb	Merge pull request #3039 from MichaelEischer/tar-proper-dirs ... Properly dump directories to tar	2020-11-02 12:30:26 +01:00
Alexander Neumann	5f0fa2129e	Improve readability ... It's time to use a switch statement.	2020-11-02 11:24:26 +01:00
Alexander Neumann	6509c207f4	Merge pull request #2997 from MichaelEischer/faster-excludes ... Speedup exclude/include checking	2020-11-02 11:07:21 +01:00
Alexander Neumann	445b845267	Merge pull request #2978 from MichaelEischer/warn-tree-error ... Warn if backup failed to read tree blob	2020-11-02 10:14:12 +01:00
Alexander Neumann	3ff37215df	Merge pull request #2935 from MichaelEischer/upgrade-minio ... Upgrade minio SDK to version 7	2020-11-02 09:09:10 +01:00
Sergio Rubio	e708628cfd	Remove unused function ... Not currently used, and it'd need to be added to the MasterIndex interface first.	2020-10-28 13:24:49 +01:00
MichaelEischer	bb4b3481a6	Merge pull request #3031 from greatroar/debug-no-seek ... Allow debug logging to pipes and terminals	2020-10-25 17:59:06 +01:00
MichaelEischer	ad3a52e6f0	Merge pull request #3026 from greatroar/refactor-ui ... internal/ui refactoring	2020-10-25 17:51:39 +01:00
Michael Eischer	8d7d6ad2d5	dump: include username in tar	2020-10-24 22:42:01 +02:00
Michael Eischer	fe09e6f865	dump: test proper permissions and directory name	2020-10-24 22:42:01 +02:00
Michael Eischer	1e3c9a2c11	dump: Fix file permission to tar mapping ... The file permissions included a go specific directory bit which accidentially forced the usage of the GNU header format. This leads to problems with 7zip on Windows or when extended attributes are used.	2020-10-24 22:42:01 +02:00
Michael Eischer	e21dcb0eea	dump: Additional ACL tests	2020-10-24 22:41:01 +02:00
MichaelEischer	31b8d7a639	Merge pull request #2274 from fgma/master ... Support for Volume Shadow Copy Service (VSS) on windows	2020-10-24 15:43:34 +02:00
fgma	5695f9ebd2	vss: Implement VSS support for Windows ... The VSS support works for 32 and 64-bit windows, this includes a check that the restic version matches the OS architecture as required by VSS. The backup operation will fail the user has not sufficient permissions to use VSS. Snapshotting volumes also covers mountpoints but skips UNC paths.	2020-10-24 11:35:57 +02:00
greatroar	63be3704d9	Allow debug logging to pipes and terminals	2020-10-19 22:30:30 +02:00
greatroar	35419de232	Simplify ui.StdioWrapper.Write ... Instead of looping to find line breaks, make it look for the last one.	2020-10-17 20:30:46 +02:00
greatroar	863a590a81	Refactor termstatus.Term.{Print,Error} methods	2020-10-17 20:30:46 +02:00
greatroar	7c0b6a82db	Remove unused public method ui.linesWriter.Flush	2020-10-17 20:30:24 +02:00
MichaelEischer	96a912b65a	Merge pull request #3025 from aawsome/fix-index-ids ... Fix setting of ID in DecodeIndex	2020-10-17 09:54:45 +02:00
Alexander Weiss	b44ecde8b0	Fix setting of ID in DecodeIndex	2020-10-17 09:12:58 +02:00
MichaelEischer	39fe1e96fe	Merge pull request #3018 from greatroar/background-check ... Fix IsProcessBackground on Linux with stdin redirection	2020-10-16 22:50:19 +02:00
MichaelEischer	4ba237bb93	Merge pull request #3019 from greatroar/refactor-decodeindex ... Refactor index decoding	2020-10-15 23:22:33 +02:00
greatroar	b27375f5ce	defer close(ch) outside repository.RunWorkers	2020-10-14 15:50:16 +02:00
greatroar	720e0ee0c7	if cond { return true }; return false => return cond	2020-10-13 20:56:43 +02:00
greatroar	27db3ec262	Refactor index decoding ... Decoding old-format indices no longer requires loading and decrypting twice.	2020-10-13 20:47:50 +02:00
greatroar	f80b07b2c8	Fix IsProcessBackground on Linux with stdin redirection ... The previous implementation assumed that stdin was a terminal. It now checks the terminal's fd.	2020-10-13 13:12:02 +02:00
MichaelEischer	41a45ae908	Merge pull request #3016 from greatroar/uid-gid-lookup ... Simplify/optimize cached UID/GID lookups	2020-10-12 23:24:55 +02:00
Alexander Neumann	56883817d8	Merge pull request #2990 from MichaelEischer/fix-goreport-warnings ... Fix some goreport warnings	2020-10-12 20:44:56 +02:00
greatroar	f0cd16e5ea	Cache uint32-typed ids in lookup{Username,Group} ... NodeFillUser-8 1.92µs ± 5% 1.75µs ± 3% -8.89% (p=0.000 n=10+10) NodeFromFileInfo-8 1.89µs ± 7% 1.76µs ± 4% -6.69% (p=0.001 n=10+10) name old alloc/op new alloc/op delta NodeFillUser-8 504B ± 0% 496B ± 0% -1.59% (p=0.000 n=10+10) NodeFromFileInfo-8 504B ± 0% 496B ± 0% -1.59% (p=0.000 n=10+10) name old allocs/op new allocs/op delta NodeFillUser-8 5.00 ± 0% 3.00 ± 0% -40.00% (p=0.000 n=10+10) NodeFromFileInfo-8 5.00 ± 0% 3.00 ± 0% -40.00% (p=0.000 n=10+10)	2020-10-12 15:08:15 +02:00
greatroar	a03fe4562e	Remove unused error return in lookup{Username,Group}	2020-10-12 14:48:23 +02:00
Alexander Neumann	c193cea119	Pass Context to NewRestorer()	2020-10-10 15:24:26 +02:00
Alexander Neumann	f0d49ca600	Merge pull request #2933 from MichaelEischer/less-context-todo ... Replace most usages of context.TODO()	2020-10-10 15:03:44 +02:00
kitone	6099f81692	Fix #1212 restore code produces inconsistent timestamps/permissions. ... Keep track of restored child status so parent and root directory not selected by filter will also restore metadata when traversing tree.	2020-10-10 13:46:44 +02:00
kitone	9d7f616190	Improve restorer debug log information	2020-10-10 13:46:43 +02:00
kitone	295ddb9e57	Add test case for inconsistent timestamps and permissions restoration ... Reproduce from https://github.com/restic/restic/issues/1212	2020-10-10 13:46:43 +02:00
Michael Eischer	e638b46a13	Embed context into ReaderAt ... The io.Reader interface does not support contexts, such that it is necessary to embed the context into the backendReaderAt struct. This has the problem that a reader might suddenly stop working when it's contained context is canceled. However, this is now problem here as the reader instances never escape the calling function.	2020-10-09 22:39:07 +02:00
Michael Eischer	d6cfe857b7	pass proper context into MasterIndex.RebuildIndex	2020-10-09 22:39:07 +02:00
Michael Eischer	37a5e2d681	rest: use global context on repository creation	2020-10-09 22:39:06 +02:00
Michael Eischer	4b0fcaed45	unlock: use proper context for locks cleanup ... The list operation used by RemoveStaleLocks or RemoveAllLocks will already be canceled by the passed in context. Therefore we can also just cancel the remove operation as the unlock command won't process all lock files anyways.	2020-10-09 22:37:56 +02:00
Michael Eischer	645a6efaf2	restorer: remove redundant type specification	2020-10-09 22:37:56 +02:00
Michael Eischer	a449450021	init: pass proper context to master key generation ... This is no change in behavior as a canceled context did later on cause the config file creation to fail. Therefore this change just lets the repository initialization fail a bit earlier.	2020-10-09 22:37:56 +02:00
Michael Eischer	603bb0e309	restore: Use proper context while loading snapshot	2020-10-09 22:37:56 +02:00
Michael Eischer	c458e114d4	pass context to Find / FindSnapshot ... This allows proper interruption of restic while it searches for snapshots or key files.	2020-10-09 22:37:56 +02:00
Michael Eischer	45e9a55c62	Wire context into backend layout detection	2020-10-09 22:37:24 +02:00
Michael Eischer	307a6ba3a3	Upgrade minio sdk to v7 ... This changes are primarily straightforward modifications to pass the parameters in the now expected way.	2020-10-09 22:37:24 +02:00
Michael Eischer	50da20d93d	Warn if backup failed to read tree blob	2020-10-09 22:36:27 +02:00
Alexander Neumann	5fd3dbccb7	Merge pull request #2963 from MichaelEischer/fix-status-deadlock ... backup: Fix possible deadlock of scanner goroutine	2020-10-09 21:35:50 +02:00
Alexander Neumann	35655a481b	Merge pull request #2931 from MichaelEischer/reduce-debug-overhead ... Reduce debug log overhead	2020-10-09 21:35:11 +02:00
Alexander Neumann	30cb553c8d	Merge pull request #2932 from MichaelEischer/proper-rclone-create ... Call rclone.Create to create a new repository for the rclone backend	2020-10-09 21:29:15 +02:00
MichaelEischer	356ac404cd	Merge pull request #2996 from greatroar/setconsolecursorposition ... Use windows.SetConsoleCursorPosition in ui/termstatus	2020-10-09 18:19:02 +02:00
Michael Eischer	88c8e903d2	filter: Fix glob matching on absolute path marker on windows ... A pattern part containing "/" is used to mark a path or a pattern as absolute. However, on Windows the path separator is "\" such that glob patterns like "?" could match the marker. The code now explicitly skips the marker when the pattern does not represent an absolute path.	2020-10-09 16:11:05 +02:00
greatroar	740758a5fa	Optimize filter pattern matching ... By replacing "**" with "", checking for this special path component can be reduced to a length-zero check. name old time/op new time/op delta FilterLines-8 44.7ms ± 5% 44.9ms ± 5% ~ (p=0.631 n=10+10) FilterPatterns/Relative-8 13.6ms ± 4% 13.4ms ± 5% ~ (p=0.165 n=10+10) FilterPatterns/Absolute-8 10.9ms ± 5% 10.7ms ± 4% ~ (p=0.052 n=10+10) FilterPatterns/Wildcard-8 53.7ms ± 5% 50.4ms ± 5% -6.00% (p=0.000 n=10+10) FilterPatterns/ManyNoMatch-8 128ms ± 2% 95ms ± 1% -25.54% (p=0.000 n=10+10) name old alloc/op new alloc/op delta FilterPatterns/Relative-8 3.57MB ± 0% 3.57MB ± 0% ~ (p=1.000 n=9+8) FilterPatterns/Absolute-8 3.57MB ± 0% 3.57MB ± 0% ~ (p=0.903 n=9+8) FilterPatterns/Wildcard-8 19.7MB ± 0% 19.7MB ± 0% -0.00% (p=0.022 n=10+9) FilterPatterns/ManyNoMatch-8 3.57MB ± 0% 3.57MB ± 0% ~ (all equal) name old allocs/op new allocs/op delta FilterPatterns/Relative-8 22.2k ± 0% 22.2k ± 0% ~ (all equal) FilterPatterns/Absolute-8 22.2k ± 0% 22.2k ± 0% ~ (all equal) FilterPatterns/Wildcard-8 88.7k ± 0% 88.7k ± 0% ~ (all equal) FilterPatterns/ManyNoMatch-8 22.2k ± 0% 22.2k ± 0% ~ (all equal)	2020-10-09 15:51:14 +02:00
MichaelEischer	862ee4b2c9	Merge pull request #2970 from labkode/fopenskip ... Skip fopen for file change check to avoid network penalty	2020-10-09 11:43:54 +02:00
Hugo Gonzalez Labrador	958dc6aafc	Skip fopen for file change check to avoid network penalty	2020-10-09 10:53:28 +02:00
MichaelEischer	88cc444779	Merge pull request #2934 from MichaelEischer/upgrade-backoff ... Upgrade github.com/cenkalti/backoff module	2020-10-08 20:33:30 +02:00
Michael Eischer	b79f18209f	Upgrade github.com/cenkalti/backoff module ... We now use v4 of the module. `backoff.WithMaxRetries` no longer repeats an operation endlessly when a retry count of 0 is specified. This required a few fixes for the tests.	2020-10-07 22:04:59 +02:00
Michael Eischer	8388e67c4b	filter: cleanup path separator conversion	2020-10-07 21:14:07 +02:00
Michael Eischer	0acc3c5923	filter: special case patterns without globbing characters ... In case a part of a path is a simple string, we can just check for equality without complex parsing in filepath.Match. name old time/op new time/op delta FilterLines-4 34.8ms ±17% 41.2ms ±23% +18.36% (p=0.000 n=10+10) FilterPatterns/Relative-4 21.7ms ± 6% 12.1ms ±23% -44.46% (p=0.000 n=10+10) FilterPatterns/Absolute-4 10.0ms ± 5% 9.1ms ±11% -9.80% (p=0.006 n=10+9) FilterPatterns/Wildcard-4 47.0ms ± 7% 42.2ms ± 5% -10.19% (p=0.000 n=9+10) FilterPatterns/ManyNoMatch-4 190ms ± 1% 131ms ±20% -31.47% (p=0.000 n=8+10) name old alloc/op new alloc/op delta FilterPatterns/Relative-4 3.57MB ± 0% 3.57MB ± 0% ~ (p=0.870 n=9+9) FilterPatterns/Absolute-4 3.57MB ± 0% 3.57MB ± 0% ~ (p=0.145 n=10+10) FilterPatterns/Wildcard-4 14.3MB ± 0% 19.7MB ± 0% +37.91% (p=0.000 n=10+10) FilterPatterns/ManyNoMatch-4 3.57MB ± 0% 3.57MB ± 0% ~ (p=0.421 n=10+9) name old allocs/op new allocs/op delta FilterPatterns/Relative-4 22.2k ± 0% 22.2k ± 0% ~ (all equal) FilterPatterns/Absolute-4 22.2k ± 0% 22.2k ± 0% ~ (all equal) FilterPatterns/Wildcard-4 88.7k ± 0% 88.7k ± 0% ~ (all equal) FilterPatterns/ManyNoMatch-4 22.2k ± 0% 22.2k ± 0% ~ (all equal)	2020-10-07 20:55:43 +02:00
Michael Eischer	54a124de3b	filter: explicitly test separate ListWithChild function	2020-10-07 20:47:52 +02:00
Michael Eischer	bcc3bddcf4	filter: only check whether a child path could match when necessary ... When checking excludes there is no need to test whether a child path could also match the pattern, as it is by definition excluded. Previously childMayMatch was calculated but then discarded. For simple absolute paths this can account for half the time spent for checking pattern matches. name old time/op new time/op delta FilterPatterns/Relative-4 23.3ms ± 9% 21.7ms ± 6% -6.68% (p=0.004 n=10+10) FilterPatterns/Absolute-4 13.9ms ± 7% 10.0ms ± 5% -27.61% (p=0.000 n=10+10) FilterPatterns/Wildcard-4 51.4ms ± 7% 47.0ms ± 7% -8.51% (p=0.001 n=9+9) FilterPatterns/ManyNoMatch-4 551ms ± 9% 190ms ± 1% -65.41% (p=0.000 n=10+8) name old alloc/op new alloc/op delta FilterPatterns/Relative-4 3.57MB ± 0% 3.57MB ± 0% ~ (p=0.665 n=10+9) FilterPatterns/Absolute-4 3.57MB ± 0% 3.57MB ± 0% ~ (p=0.480 n=9+10) FilterPatterns/Wildcard-4 14.3MB ± 0% 14.3MB ± 0% ~ (p=0.431 n=9+10) FilterPatterns/ManyNoMatch-4 3.57MB ± 0% 3.57MB ± 0% ~ (all equal) name old allocs/op new allocs/op delta FilterPatterns/Relative-4 22.2k ± 0% 22.2k ± 0% ~ (all equal) FilterPatterns/Absolute-4 22.2k ± 0% 22.2k ± 0% ~ (all equal) FilterPatterns/Wildcard-4 88.7k ± 0% 88.7k ± 0% ~ (all equal) FilterPatterns/ManyNoMatch-4 22.2k ± 0% 22.2k ± 0% ~ (all equal)	2020-10-07 20:47:52 +02:00
Michael Eischer	17c53efb0d	filter: Optimize double wildcard expansion ... This only allocates a single slice to expand the double wildcard and only copies the pattern prefix once. name old time/op new time/op delta FilterPatterns/Relative-4 22.7ms ± 5% 23.3ms ± 9% ~ (p=0.353 n=10+10) FilterPatterns/Absolute-4 14.2ms ±13% 13.9ms ± 7% ~ (p=0.853 n=10+10) FilterPatterns/Wildcard-4 266ms ±16% 51ms ± 7% -80.67% (p=0.000 n=10+9) FilterPatterns/ManyNoMatch-4 554ms ± 6% 551ms ± 9% ~ (p=0.436 n=10+10) name old alloc/op new alloc/op delta FilterPatterns/Relative-4 3.57MB ± 0% 3.57MB ± 0% ~ (p=0.349 n=10+10) FilterPatterns/Absolute-4 3.57MB ± 0% 3.57MB ± 0% ~ (p=0.073 n=10+9) FilterPatterns/Wildcard-4 141MB ± 0% 14MB ± 0% -89.89% (p=0.000 n=10+9) FilterPatterns/ManyNoMatch-4 3.57MB ± 0% 3.57MB ± 0% ~ (all equal) name old allocs/op new allocs/op delta FilterPatterns/Relative-4 22.2k ± 0% 22.2k ± 0% ~ (all equal) FilterPatterns/Absolute-4 22.2k ± 0% 22.2k ± 0% ~ (all equal) FilterPatterns/Wildcard-4 1.63M ± 0% 0.09M ± 0% -94.56% (p=0.000 n=10+10) FilterPatterns/ManyNoMatch-4 22.2k ± 0% 22.2k ± 0% ~ (all equal)	2020-10-07 20:47:48 +02:00
Michael Eischer	7959796269	filter: Special case for absolute paths ... name old time/op new time/op delta FilterPatterns/Relative-4 23.6ms ±20% 22.7ms ± 5% ~ (p=0.684 n=10+10) FilterPatterns/Absolute-4 32.3ms ± 8% 14.2ms ±13% -56.01% (p=0.000 n=10+10) FilterPatterns/Wildcard-4 334ms ±17% 266ms ±16% -20.56% (p=0.000 n=10+10) FilterPatterns/ManyNoMatch-4 709ms ± 7% 554ms ± 6% -21.89% (p=0.000 n=10+10) name old alloc/op new alloc/op delta FilterPatterns/Relative-4 3.57MB ± 0% 3.57MB ± 0% +0.00% (p=0.046 n=9+10) FilterPatterns/Absolute-4 3.57MB ± 0% 3.57MB ± 0% ~ (p=0.464 n=10+10) FilterPatterns/Wildcard-4 141MB ± 0% 141MB ± 0% ~ (p=0.163 n=9+10) FilterPatterns/ManyNoMatch-4 3.57MB ± 0% 3.57MB ± 0% ~ (all equal) name old allocs/op new allocs/op delta FilterPatterns/Relative-4 22.2k ± 0% 22.2k ± 0% ~ (all equal) FilterPatterns/Absolute-4 22.2k ± 0% 22.2k ± 0% ~ (all equal) FilterPatterns/Wildcard-4 1.63M ± 0% 1.63M ± 0% ~ (p=0.072 n=10+10) FilterPatterns/ManyNoMatch-4 22.2k ± 0% 22.2k ± 0% ~ (all equal)	2020-10-07 20:47:29 +02:00
Michael Eischer	375c2a56de	filter: Parse filter patterns only once ... name old time/op new time/op delta FilterPatterns/Relative-4 30.3ms ±10% 23.6ms ±20% -22.12% (p=0.000 n=10+10) FilterPatterns/Absolute-4 49.0ms ± 3% 32.3ms ± 8% -33.94% (p=0.000 n=8+10) FilterPatterns/Wildcard-4 345ms ± 9% 334ms ±17% ~ (p=0.315 n=10+10) FilterPatterns/ManyNoMatch-4 3.93s ± 2% 0.71s ± 7% -81.98% (p=0.000 n=9+10) name old alloc/op new alloc/op delta FilterPatterns/Relative-4 4.63MB ± 0% 3.57MB ± 0% -22.98% (p=0.000 n=9+9) FilterPatterns/Absolute-4 8.54MB ± 0% 3.57MB ± 0% -58.20% (p=0.000 n=10+10) FilterPatterns/Wildcard-4 146MB ± 0% 141MB ± 0% -2.93% (p=0.000 n=9+9) FilterPatterns/ManyNoMatch-4 907MB ± 0% 4MB ± 0% -99.61% (p=0.000 n=9+9) name old allocs/op new allocs/op delta FilterPatterns/Relative-4 66.6k ± 0% 22.2k ± 0% -66.67% (p=0.000 n=10+10) FilterPatterns/Absolute-4 88.7k ± 0% 22.2k ± 0% -75.00% (p=0.000 n=10+10) FilterPatterns/Wildcard-4 1.70M ± 0% 1.63M ± 0% -3.92% (p=0.000 n=10+10) FilterPatterns/ManyNoMatch-4 4.46M ± 0% 0.02M ± 0% -99.50% (p=0.000 n=10+10)	2020-10-07 20:47:27 +02:00
Michael Eischer	b8eacd1364	filter: Reduce redundant path and pattern splitting ... A single call to filter.List will split the path only once and also split each search pattern only once and use it for both match and childMatch. name old time/op new time/op delta FilterPatterns/Relative-4 62.1ms ±15% 30.3ms ±10% -51.22% (p=0.000 n=9+10) FilterPatterns/Absolute-4 111ms ±10% 49ms ± 3% -56.08% (p=0.000 n=10+8) FilterPatterns/Wildcard-4 393ms ±15% 345ms ± 9% -12.30% (p=0.000 n=10+10) FilterPatterns/ManyNoMatch-4 10.0s ± 3% 3.9s ± 2% -60.53% (p=0.000 n=10+9) name old alloc/op new alloc/op delta FilterPatterns/Relative-4 16.4MB ± 0% 4.6MB ± 0% -71.76% (p=0.000 n=10+9) FilterPatterns/Absolute-4 31.4MB ± 0% 8.5MB ± 0% -72.77% (p=0.000 n=9+10) FilterPatterns/Wildcard-4 168MB ± 0% 146MB ± 0% -13.19% (p=0.000 n=10+9) FilterPatterns/ManyNoMatch-4 3.23GB ± 0% 0.91GB ± 0% -71.96% (p=0.000 n=10+9) name old allocs/op new allocs/op delta FilterPatterns/Relative-4 178k ± 0% 67k ± 0% -62.50% (p=0.000 n=10+10) FilterPatterns/Absolute-4 266k ± 0% 89k ± 0% -66.67% (p=0.000 n=10+10) FilterPatterns/Wildcard-4 1.87M ± 0% 1.70M ± 0% -9.47% (p=0.000 n=10+10) FilterPatterns/ManyNoMatch-4 17.7M ± 0% 4.5M ± 0% -74.87% (p=0.000 n=9+10)	2020-10-07 18:13:19 +02:00
Michael Eischer	e73c281142	filter: Benchmark absolute paths, wildcards and long filter lists	2020-10-07 17:54:36 +02:00
Michael Eischer	fdd3b14db3	filter: test some corner cases	2020-10-07 17:09:44 +02:00
Michael Eischer	f4282aa6fd	local: mark repository files as read-only ... This is intended to prevent accidental modifications of data files. Marking the files as read-only was accidentally removed in #1258.	2020-10-07 12:29:37 +02:00
Michael Eischer	40ee17167e	local: Ignore permission errors on chmod call in Save/Remove operation ... The file is already created with the proper permissions, thus the chmod call is not critical. However, some file systems have don't allow modifications of the file permissions. Similarly the chmod call in the Remove operation should not prevent it from working.	2020-10-07 12:29:37 +02:00
greatroar	f3e933f0c1	Use windows.SetConsoleCursorPosition in ui/termstatus	2020-10-07 11:18:54 +02:00
MichaelEischer	09d39e260d	Merge pull request #2980 from greatroar/error-wrapping ... Remove repetitive error wrapping from internal/cache	2020-10-06 15:19:00 +02:00
Michael Eischer	1579d2a8ec	Remove some unused assignments	2020-10-06 14:55:13 +02:00
Michael Eischer	eba5dd831f	Fix typos reported by misspell	2020-10-06 14:55:13 +02:00
Michael Eischer	9ffb698c8d	Modernize context import in internal/fuse ... Migrated using `go fix`.	2020-10-06 14:55:13 +02:00
Michael Eischer	efbb850d92	Remove a few redundant type specifiers ... This is the result of running `gofmt -s -w **/*.go`	2020-10-06 14:55:13 +02:00
greatroar	673dda77c0	Less repetitive error messages in internal/cache ... Many instances of errors.Wrap in this package would produce messages like "Open: open <filename>: no such file or directory"; those now omit the first "Open:" (or "Stat:", or "MkdirAll"). The function readVersion now appends its own name to the error message, rather than the function that failed, to make it easier to spot. Other function names (e.g., Load) are already added further up in the call chain.	2020-10-05 20:28:54 +02:00
MichaelEischer	1a490acd67	Merge pull request #2982 from greatroar/archiver-error-handling ... Check error in archiver before calling Select	2020-10-05 19:19:47 +02:00
MichaelEischer	c84643c6a9	Merge pull request #2981 from greatroar/mkdircachedir ... Simplify cache directory creation	2020-10-05 18:15:23 +02:00
greatroar	1e1a1f3078	Upgrade cespare/xxhash to 2.1.1	2020-10-05 17:25:39 +02:00
greatroar	c4e2203e45	Check error in archiver before calling Select ... The archiver first called the Select function for a path before checking whether the Lstat on that path actually worked. The RejectFuncs in exclude.go worked around this by checking whether they received a nil os.FileInfo. Checking first is more obvious and requires less code.	2020-10-05 11:11:04 +02:00
greatroar	7d0fa1a686	Simplify cache directory creation	2020-10-05 10:46:45 +02:00
Ingo Gottwald	8b8e230771	Swap deprecated GCS lib with replacement	2020-10-03 18:55:56 +02:00
Ingo Gottwald	00cedd22aa	Replace deprecated method in gs backend	2020-10-01 10:02:42 +02:00
Michael Eischer	d44df9d00d	backup: Always remove the status lines once a backup ends	2020-09-30 23:13:10 +02:00
Michael Eischer	8d0ba55ecd	backup: Fix possible deadlock of scanner goroutine ... When the backup is interrupted for some reason while the scanner is still active this could lead to a deadlock. Interruptions are triggered by canceling the context object used by both the backup progress UI and the scanner. It is possible that a context is canceled between the respective check in the scanner and it calling the `ReportTotal` method of the UI. The latter method sends a message to the UI goroutine. However, a canceled context will also stop that goroutine, which can cause the channel send operation to block indefinitely. This is resolved by adding a `closed` channel which is closed once the UI goroutine is stopped and serves as an escape hatch for reported UI updates. This change covers not just the ReportTotal method but all potentially affected methods of the progress UI implementation.	2020-09-30 23:13:10 +02:00
MichaelEischer	48d0ab5276	Merge pull request #2959 from restic/keep-text ... backup: Correct keep policy text	2020-09-30 17:57:00 +02:00
MichaelEischer	fd02407863	Merge pull request #2849 from classmarkets/gcs-access-token ... gs: support authentication with access token	2020-09-30 17:42:56 +02:00
Leo R. Lundgren	028c9a5343	backup: Correct keep policy text ... Makes the following corrections to the "Applying Policy:" output: - keep the last 1 snapshots snapshots => keep 1 latest snapshots - keep the last 1 snapshots, 3 hourly, 5 yearly snapshots => keep 1 latest, 3 hourly, 5 yearly snapshots	2020-09-28 14:26:53 +02:00
MichaelEischer	14b312f00d	Merge pull request #2658 from creativeprojects/issue-2241 ... Don't echo authentication passwords (rest backend)	2020-09-22 22:17:53 +02:00
Fred	206cadfab4	Hide password from repository URLs	2020-09-22 22:00:51 +02:00
MichaelEischer	4875f7b659	Merge pull request #2614 from greatroar/simplify-fs ... Simplify internal/fs	2020-09-21 22:05:10 +02:00
greatroar	0d65b78168	Simplify os.ModeType|os.ModeCharDevice => os.ModeType ... Since Go 1.12, ModeCharDevice is included in ModeType: golang/go@a2a3dd00c9	2020-09-21 14:21:32 +02:00
greatroar	95ebba85ff	Remove stray Printf from internal/fs	2020-09-21 14:21:32 +02:00
greatroar	59b343a9bf	Remove OS-specific versions of fs.MkdirAll ... Go has supported Windows paths correctly since 1.11, see https://github.com/golang/go/issues/10900 and the commit referenced there.	2020-09-21 14:21:32 +02:00
greatroar	1557c58eef	Fix and simplify fs.Reader ... fakeDir.{Readdir,Readdirnames} weren't handling the case n == 0 correctly. fakeFileInfo.sys is always nil, so omit the field.	2020-09-21 14:21:32 +02:00
greatroar	c504aa505c	Remove unused fs.Rename	2020-09-21 14:21:32 +02:00
greatroar	1b20f6beec	Remove io.Writer from fs.File ... It was only used in a single test, which now uses plain *os.File instead.	2020-09-21 14:21:32 +02:00
Michael Eischer	8c36317b71	rclone: use configured number of connections during create	2020-09-19 19:11:43 +02:00
Michael Eischer	60a5c35de9	rclone: close connection to rclone if open fails	2020-09-19 19:11:43 +02:00
Michael Eischer	07f4e7d10b	Only log HTTP requests when a debug log is configured ... The logged HTTP requests are only visible when a debug log is configured.	2020-09-19 17:07:53 +02:00
Michael Eischer	f003410402	init: Add --copy-chunker-params option ... This allows creating multiple repositories with identical chunker parameters which is required for working deduplication when copying snapshots between different repositories.	2020-09-19 16:53:05 +02:00
Michael Eischer	9ad8250a78	Reduce overhead of debug calls if no log is enabled ... In case no debug log is configured, then calls to debug.Log only incur the costs to check a single boolean flag making the call really cheap.	2020-09-19 16:45:01 +02:00
greatroar	cb3f531050	Make Cache.Path private ... It's only used inside the package.	2020-09-18 11:17:29 +02:00
greatroar	23fcbb275a	Remove restic.Cache interface ... It was used in one code path, which then asserted its concrete type as *cache.Cache. Privatised some of the interface methods.	2020-09-18 10:48:13 +02:00
greatroar	6e3215a80d	Remove unused error type from internal/cache	2020-09-18 10:38:14 +02:00
greatroar	9abef3bf1a	Move internal/fs.TestChdir to internal/test.Chdir	2020-09-17 10:43:33 +02:00
Alexander Neumann	0c48e515f0	Merge pull request #2630 from MichaelEischer/fix-staticcheck ... Fix lots of small issues reported by staticcheck	2020-09-13 16:19:22 +02:00
Alexander Neumann	97950ab81a	options: Fix test for Go >= 1.15	2020-09-12 17:36:44 +02:00
MichaelEischer	88664ba222	Merge pull request #2893 from MichaelEischer/restore-preallocate ... restorer: Preallocate files	2020-09-08 22:43:05 +02:00
Michael Eischer	121233e1b3	Add preallocate tests	2020-09-07 21:41:47 +02:00
Michael Eischer	8cc9514879	restorer: pre-allocate files before loading chunks	2020-09-07 21:41:47 +02:00
Michael Eischer	2e7d475029	Process packs in order of first appearance	2020-09-06 21:25:35 +02:00
Michael Eischer	c46edcd9d6	error strings should not end with punctuation	2020-09-05 10:07:17 +02:00
Michael Eischer	1ede018ea6	error variable names should start with 'Err'	2020-09-05 10:07:17 +02:00
Michael Eischer	b77e933d80	Convert underscore variable names to camelCase	2020-09-05 10:07:16 +02:00
Michael Eischer	d0329cf3eb	Adjust comments to match name of exported methods	2020-09-05 10:07:16 +02:00
Michael Eischer	dc31529fc3	Unindent else block after if block ending with a return statement	2020-09-05 10:07:16 +02:00
Michael Eischer	4784540f04	repository: Simplify worker group code	2020-09-05 10:07:16 +02:00
Michael Eischer	84ea2389ae	archiver_test: Only skip symlinks on windows	2020-09-05 10:07:16 +02:00
Michael Eischer	b4a7ce86cf	uint cannot be less than zero	2020-09-05 10:07:16 +02:00
Michael Eischer	7ee0964880	crypto: Remove unused error	2020-09-05 10:07:16 +02:00
Michael Eischer	f6f11400c2	restic: Remove unused variable value	2020-09-05 10:07:16 +02:00
Michael Eischer	e2dc5034d3	restic/file: Use FileType for all File constants ... The data type of a variable is only carried on when using iota.	2020-09-05 10:07:16 +02:00
Michael Eischer	9a1b3cb5d9	restorer: Remove unused treeID struct member from test	2020-09-05 10:07:16 +02:00
Michael Eischer	ee05501ce7	archiver: Remove unused chmod function	2020-09-05 10:07:16 +02:00
Michael Eischer	014600bee6	archiver: Remove unused variable from test	2020-09-05 10:07:16 +02:00
Michael Eischer	d9a80e07b9	repository: Simplify index age calculation	2020-09-05 10:07:16 +02:00
Michael Eischer	460e2ffbf6	Collapse a few boolean operations	2020-09-05 10:07:14 +02:00
Michael Eischer	2f8335554c	Remove a few unused variables	2020-09-05 10:06:23 +02:00
Michael Eischer	37113282ca	Merge a few variable declaration and initializations	2020-09-05 10:05:34 +02:00
Michael Eischer	337725c354	Use sort.Strings	2020-09-05 10:05:34 +02:00
Michael Eischer	dcf9ded977	dump: Use slashes as path separators in tar on windows ... This conversion is necessary as paths returned by filepath.Rel use the operating-system native path separator	2020-08-30 18:26:50 +02:00
Simon Beck	fe6445e0f4	dump: Added unit tests for tar output	2020-08-30 16:00:48 +02:00
Simon Beck	80a11960dd	dump: Always dump relative paths into tarballs ... Tarballs should only contain relative paths.	2020-08-29 21:14:34 +02:00
Simon Beck	d6f739ec22	dump: Extract tar code to internal/dump	2020-08-29 21:14:34 +02:00
Michael Eischer	b25978a53c	backup: Fix reporting of directory count in summary ... Previously the directory stats were reported immediately after calling `SaveDir`. However, as the latter method saves the tree asynchronously the stats were still initialized to their nil value. The stats are now reported via a callback similar to the one used for the fileSaver.	2020-08-27 22:43:51 +02:00
Alexander Neumann	b1b3f1ecb6	Merge pull request #2674 from MichaelEischer/prune-strict-checks ... prune: Stricter error checks	2020-08-23 10:30:41 +02:00
Hristo Trendev	51c22f4223	local backend: ignore not supported error on sync() ... Closes #2395	2020-08-22 01:27:07 +02:00
Michael Eischer	f4b9544ab2	prune: Add test that repack aborts on wrong blob	2020-08-16 11:34:01 +02:00
Michael Eischer	367449dede	prune: Reduce memory allocations while repacking ... The slicing operator `slice[low:high]` default to 0 for the lower bound and len(slice) for the upper bound when either or both are not specified. Fix the code to use `cap(slice)` to check for the slice capacity.	2020-08-16 11:34:01 +02:00
Michael Eischer	7042bafea5	prune: Abort repacking when a pack contains a wrong blob ... If a blob in a pack file can be decrypted successfully but contains data that results in a different hash than stated in the header pack, then abort repacking. As both the pack header and the blob are cryptographically verified this either means than a malicious entity tampered with the backup or indicates hardware problems on the client. prune should fail with an error in both cases.	2020-08-16 11:34:01 +02:00
aawsome	0fed6a8dfc	Use "pack file" instead of "data file" (#2885) ... - changed variable names, especially changed DataFile into PackFile - changed in some comments - always use "pack file" in docu	2020-08-16 11:16:38 +02:00
MichaelEischer	eca0f0ad24	Merge pull request #2863 from aawsome/index-no-duplicates ... Don't save exact duplicates in merged index	2020-08-08 18:24:14 +02:00
Alexander Weiss	b112533812	Don't save exact duplicates when merging indexes	2020-08-05 06:32:02 +02:00
Alexander Weiss	5e63294355	Add benchmark MasterIndexAlloc	2020-08-05 06:32:02 +02:00
Michael Eischer	05116e4787	prune: Cleanup progress bar handling while repacking	2020-08-03 19:32:46 +02:00
Alexander Neumann	b7b479b668	Merge pull request #2599 from MichaelEischer/tweak-mem-usage ... Reduce memory usage when searching for used blobs	2020-08-01 13:49:02 +02:00
Alexander Neumann	4cf9656f12	Merge pull request #2861 from MichaelEischer/fix-rclone-crash ... rclone: Don't panic after unexpected subprocess exit	2020-08-01 13:08:50 +02:00
Alexander Neumann	2580eef2aa	Merge pull request #2318 from classmarkets/2175-named-keys ... Allow specifying user and host when adding keys	2020-08-01 13:06:31 +02:00
Michael Eischer	a178e5628e	Remove duplicate TreeLoader interface	2020-08-01 12:29:16 +02:00
Michael Eischer	af66a62c04	FindUsedBlobs: Test that seen blobs are skipped ... This also copies the TreeLoader interface from internal/walker to allow stubbing the repository in the call to `FindUsedBlobs`.	2020-08-01 12:29:16 +02:00
Michael Eischer	9ea1a78bd4	FindUsedBlobs: Check for seen blobs before loading trees ... The only effective change in behavior is that that toplevel nodes can also be skipped.	2020-08-01 12:29:16 +02:00
Michael Eischer	184103647a	FindUsedBlobs: merge seen into blobs BlobSet ... The seen BlobSet always contained a subset of the entries in blobs. Thus use blobs instead and avoid the memory overhead of the second set. Suggested-by: Alexander Weiss <alex@weissfam.de>	2020-08-01 12:29:16 +02:00
Michael Eischer	c81b122374	rclone: Don't panic after unexpected subprocess exit ... As the connection to the rclone child process is now closed after an unexpected subprocess exit, later requests will cause the http2 transport to try to reestablish a new connection. As previously this never should have happened, the connection called panic in that case. This panic is now replaced with a simple error message, as it no longer indicates an internal problem.	2020-08-01 12:17:40 +02:00
Alexander Weiss	f8316948d1	Optimize FUSE - make command restic mount faster and consume less memory ... - Add Open() functionality to dir - only access index for blobs when file is read - Implement NodeOpener and put one-time file stuff there - Add comment about locking as suggested by bazil.org/fuse => Thanks at Michael Eischer for suggesting the last two improvements	2020-07-28 23:01:18 +02:00
Michael Eischer	ea97ff1ba4	rclone: Skip crash test when rclone is not found	2020-07-26 12:06:18 +02:00
Michael Eischer	01b9581453	rclone: Better field names for stdio conn	2020-07-26 00:29:25 +02:00
Michael Eischer	3cd927d180	rclone: Give rclone time to finish before closing stdin pipe ... Calling `Close()` on the rclone backend sometimes failed during test execution with 'signal: Broken pipe'. The stdio connection closed both the stdin and stdout file descriptors at the same moment, therefore giving rclone no chance to properly send any final http2 data frames. Now the stdin connection to rclone is closed first and will only be forcefully closed after a timeout. In case rclone exits before the timeout then the stdio connection will be closed normally.	2020-07-26 00:29:25 +02:00
greatroar	bf7b1f12ea	rclone: Add test for pipe handling when rclone exits	2020-07-26 00:29:25 +02:00
Michael Eischer	8554332894	rclone: Close rclone side of stdio_conn pipes ... restic did not notice when the rclone subprocess exited unexpectedly. restic manually created pipes for stdin and stdout and used these for the connection to the rclone subprocess. The process creating a pipe gets file descriptors for the sender and receiver side of a pipe and passes them on to the subprocess. The expected behavior would be that reads or writes in the parent process fail / return once the child process dies as a pipe would now just have a reader or writer but not both. However, this never happened as restic kept the reader and writer file descriptors of the pipes. `cmd.StdinPipe` and `cmd.StdoutPipe` close the subprocess side of pipes once the child process was started and close the parent process side of pipes once wait has finished. We can't use these functions as we need access to the raw `os.File` so just replicate that behavior.	2020-07-26 00:29:25 +02:00
greatroar	3e93b36ca4	Make rclone.New private	2020-07-26 00:28:45 +02:00
Michael Eischer	c847aace35	Rename Index interface to MasterIndex ... The interface is now only implemented by repository.MasterIndex.	2020-07-25 21:19:46 +02:00
Alexander Weiss	9d1fb94c6c	make Lookup() return all blobs ... + simplify syntax	2020-07-25 21:18:34 +02:00
MichaelEischer	020cab8e08	Merge pull request #2787 from greatroar/no-blobsize-cache ... Remove blob size cache from restic mount	2020-07-25 20:46:35 +02:00
greatroar	07da61baee	Remove blob size cache from restic mount	2020-07-25 19:39:14 +02:00
MichaelEischer	3d530dfc91	Merge pull request #2827 from aawsome/archiver-test-contents ... Make self-healing work when backing up with parent snapshot	2020-07-25 13:13:18 +02:00
MichaelEischer	bbc960f957	Merge pull request #2635 from greatroar/optimize-sortbycached ... Optimize sorting blobs by cache status	2020-07-25 12:35:42 +02:00
greatroar	309598c237	Simplify sortCachedPacksFirst test in internal/repository ... The test now uses the fact that the sort is stable. It's not guaranteed to be, but the test is cleaner and more exhaustive. sortCachedPacksFirst no longer needs a return value.	2020-07-25 12:12:59 +02:00
greatroar	03d23e6faa	Speed up blob sorting in internal/repository ... name old time/op new time/op delta SortCachedPacksFirst-8 208µs ± 3% 186µs ± 3% -10.74% (p=0.000 n=10+8) name old alloc/op new alloc/op delta SortCachedPacksFirst-8 213kB ± 0% 139kB ± 0% -34.62% (p=0.000 n=10+10) name old allocs/op new allocs/op delta SortCachedPacksFirst-8 1.03k ± 0% 1.03k ± 0% -0.19% (p=0.000 n=10+10)	2020-07-25 12:12:59 +02:00
greatroar	b10acd2af7	Test and benchmark blob sorting in internal/repository	2020-07-25 12:12:58 +02:00
Alexander Weiss	9175795fdb	Check contents in archiver ... When backing up with a parent snapshot and the file is not changed, also check if contents are still available in index.	2020-07-25 08:18:28 +02:00
rawtaz	5d8d70542f	Merge pull request #2852 from MichaelEischer/drop-go-1.11 ... Drop support for Go version 1.11	2020-07-25 01:17:36 +02:00
Michael Eischer	7c23381a2b	Drop support for Go version 1.11	2020-07-24 18:52:39 +02:00
MichaelEischer	34181b13a2	Merge pull request #2328 from MichaelEischer/no-repeated-checks ... Fix duplicate tree checks within `restic check`	2020-07-22 22:08:02 +02:00
Alexander Weiss	a666a6d576	Add tests and merge indexes in index benchmarks	2020-07-22 21:54:02 +02:00
Alexander Weiss	e388d962a5	Merge final indexes together for faster index access	2020-07-22 21:54:02 +02:00
Peter Schultz	758b44b9c0	gs: support authentication with access token ... In the Google Cloud Storage backend, support specifying access tokens directly, as an alternative to a credentials file. This is useful when restic is used non-interactively by some other program that is already authenticated and eliminates the need to store long lived credentials. The access token is specified in the GOOGLE_ACCESS_TOKEN environment variable and takes precedence over GOOGLE_APPLICATION_CREDENTIALS.	2020-07-22 16:23:03 +02:00
Alexander Weiss	3b7a3711e6	Add more realistic index benchmarks ... + reduce test size of BenchmarkMasterIndexLookupParallel	2020-07-21 07:18:20 +02:00
Michael Eischer	9b0e718852	checker: Test that blob types are not confused	2020-07-20 23:43:47 +02:00
MichaelEischer	82c908871d	Merge pull request #2812 from greatroar/chaining ... Chaining hash table for repository.Index	2020-07-20 23:29:36 +02:00
Michael Eischer	ddf0b8cd0b	checker: Properly distinguish between data and tree blobs ... If a data blob and a tree blob with the same ID (= same content) exist, then the checker did not report a data or tree blob as unused when the blob of the other type was still in use.	2020-07-20 22:58:39 +02:00
Michael Eischer	2d0c138c9b	checker: Test that check only decodes trees once ... The `DuplicateTree` flag is necessary to ensure that failures cannot be swallowed. The old checker implementation ignores errors from LoadTree if the corresponding tree was already checked.	2020-07-20 22:51:53 +02:00
Michael Eischer	ef325ffc02	checker: Cleanup error handling code ... This change only moves code around but does not result in any change in behavior.	2020-07-20 22:51:53 +02:00
Michael Eischer	7a165f32a9	checker: Traverse trees in depth-first order ... Backups traverse the file tree in depth-first order and saves trees on the way back up. This results in tree packs filled in a way comparable to the reverse Polish notation. In order to check tree blobs in that order, the treeFilter would have to delay the forwarding of tree nodes until all children of it are processed which would complicate the implementation. Therefore do the next similar thing and traverse the tree in depth-first order, but process trees already on the way down. The tree blob ids are added in reverse order to the backlog, which is once again reverted when removing the ids from the back of the backlog.	2020-07-20 22:51:53 +02:00
Michael Eischer	36c69e3ca7	checker: Unify blobs, processed trees and referenced blobs map ... The blobRefs map and the processedTrees IDSet are merged to reduce the memory usage. The blobRefs map now uses separate flags to track blob usage as data or tree blob. This prevents skipping of trees whose content is identical to an already processed data blob. A third flag tracks whether a blob exists or not, which removes the need for the blobs IDSet.	2020-07-20 22:51:47 +02:00
Michael Eischer	35d8413639	checker: Remove dead index map	2020-07-20 22:37:31 +02:00
Michael Eischer	c66a0e408c	checker: Reduce cost of debug log ... Avoid duplicate allocation of the Subtree list.	2020-07-20 22:37:31 +02:00
Michael Eischer	70f4c014ef	checker: Decode identical tree nodes only once ... Even though the checkTreeWorker skips already processed chunks, filterTrees did queue the same tree blob on every occurence. This becomes a serious performance bottleneck for larger number of snapshots that cover mostly the same directories. Therefore decode a tree blob exactly once.	2020-07-20 22:37:31 +02:00
Michael Eischer	f0d8710611	Add benchmark for checker scaling with snapshot count	2020-07-20 22:37:31 +02:00
Alexander Neumann	8074879c5f	Remove 'go generate'	2020-07-19 17:28:42 +02:00
greatroar	7bda28f31f	Chaining hash table for repository.Index ... These are faster to construct but slower to access. The allocation rate is halved, the peak memory usage almost halved compared to standard map. Benchmark results on linux/amd64, -benchtime=3s -count=20: name old time/op new time/op delta PackerManager-8 178ms ± 0% 178ms ± 0% ~ (p=0.231 n=20+20) DecodeIndex-8 4.54s ± 0% 4.30s ± 0% -5.20% (p=0.000 n=18+17) DecodeIndexParallel-8 4.54s ± 0% 4.30s ± 0% -5.22% (p=0.000 n=19+18) IndexHasUnknown-8 44.4ns ± 5% 50.5ns ±11% +13.82% (p=0.000 n=19+17) IndexHasKnown-8 48.3ns ± 0% 51.5ns ±12% +6.68% (p=0.001 n=16+20) IndexAlloc-8 758ms ± 1% 616ms ± 1% -18.69% (p=0.000 n=19+19) IndexAllocParallel-8 234ms ± 3% 204ms ± 2% -12.60% (p=0.000 n=20+18) MasterIndexLookupSingleIndex-8 122ns ± 0% 145ns ± 9% +18.44% (p=0.000 n=14+20) MasterIndexLookupMultipleIndex-8 369ns ± 2% 429ns ± 8% +16.27% (p=0.000 n=20+20) MasterIndexLookupSingleIndexUnknown-8 68.4ns ± 5% 74.9ns ±13% +9.47% (p=0.000 n=20+20) MasterIndexLookupMultipleIndexUnknown-8 315ns ± 3% 369ns ±11% +17.14% (p=0.000 n=20+20) MasterIndexLookupParallel/known,indices=5-8 743ns ± 1% 816ns ± 2% +9.87% (p=0.000 n=17+17) MasterIndexLookupParallel/unknown,indices=5-8 238ns ± 1% 260ns ± 2% +9.14% (p=0.000 n=19+20) MasterIndexLookupParallel/known,indices=10-8 1.01µs ± 3% 1.11µs ± 2% +9.79% (p=0.000 n=19+20) MasterIndexLookupParallel/unknown,indices=10-8 222ns ± 0% 269ns ± 2% +20.83% (p=0.000 n=16+20) MasterIndexLookupParallel/known,indices=20-8 1.06µs ± 2% 1.19µs ± 2% +12.95% (p=0.000 n=19+18) MasterIndexLookupParallel/unknown,indices=20-8 413ns ± 1% 530ns ± 1% +28.19% (p=0.000 n=18+20) SaveAndEncrypt-8 30.2ms ± 1% 30.4ms ± 0% +0.71% (p=0.000 n=19+19) LoadTree-8 540µs ± 1% 576µs ± 1% +6.73% (p=0.000 n=20+20) LoadBlob-8 5.64ms ± 0% 5.64ms ± 0% ~ (p=0.883 n=18+17) LoadAndDecrypt-8 5.93ms ± 0% 5.95ms ± 1% ~ (p=0.247 n=20+19) LoadIndex-8 25.1ms ± 0% 24.5ms ± 1% -2.54% (p=0.000 n=18+17) name old speed new speed delta PackerManager-8 296MB/s ± 0% 296MB/s ± 0% ~ (p=0.229 n=20+20) SaveAndEncrypt-8 139MB/s ± 1% 138MB/s ± 0% -0.71% (p=0.000 n=19+19) LoadBlob-8 177MB/s ± 0% 177MB/s ± 0% ~ (p=0.890 n=18+17) LoadAndDecrypt-8 169MB/s ± 0% 168MB/s ± 1% ~ (p=0.227 n=20+19) name old alloc/op new alloc/op delta PackerManager-8 91.8kB ± 0% 91.8kB ± 0% ~ (p=0.772 n=12+19) IndexAlloc-8 786MB ± 0% 400MB ± 0% -49.04% (p=0.000 n=20+18) IndexAllocParallel-8 786MB ± 0% 401MB ± 0% -49.04% (p=0.000 n=19+15) SaveAndEncrypt-8 21.0MB ± 0% 21.0MB ± 0% +0.00% (p=0.000 n=19+19) name old allocs/op new allocs/op delta PackerManager-8 1.41k ± 0% 1.41k ± 0% ~ (all equal) IndexAlloc-8 977k ± 0% 907k ± 0% -7.18% (p=0.000 n=20+20) IndexAllocParallel-8 977k ± 0% 907k ± 0% -7.17% (p=0.000 n=19+15) SaveAndEncrypt-8 73.0 ± 0% 73.0 ± 0% ~ (all equal)	2020-07-19 13:58:22 +02:00
greatroar	255ba83c4b	Parallel index benchmarks + benchmark optimizations ... createRandomIndex was using the global RNG, which locks on every call It was also using twice as many random numbers as necessary and doing a float division in every iteration of the inner loop. BenchmarkDecodeIndex was using too short an input, especially for a parallel version. (It may now be using one that is a bit large.) Results on linux/amd64, -benchtime=3s -count=20: name old time/op new time/op delta PackerManager-8 178ms ± 0% 178ms ± 0% ~ (p=0.165 n=20+20) DecodeIndex-8 13.6µs ± 2% 4539886.8µs ± 0% +33293901.38% (p=0.000 n=20+18) IndexHasUnknown-8 44.4ns ± 7% 44.4ns ± 5% ~ (p=0.873 n=20+19) IndexHasKnown-8 49.2ns ± 3% 48.3ns ± 0% -1.86% (p=0.000 n=20+16) IndexAlloc-8 802ms ± 1% 758ms ± 1% -5.51% (p=0.000 n=20+19) MasterIndexLookupSingleIndex-8 124ns ± 1% 122ns ± 0% -1.41% (p=0.000 n=20+14) MasterIndexLookupMultipleIndex-8 373ns ± 2% 369ns ± 2% -1.13% (p=0.001 n=20+20) MasterIndexLookupSingleIndexUnknown-8 67.8ns ± 3% 68.4ns ± 5% ~ (p=0.753 n=20+20) MasterIndexLookupMultipleIndexUnknown-8 316ns ± 3% 315ns ± 3% ~ (p=0.846 n=20+20) SaveAndEncrypt-8 30.5ms ± 1% 30.2ms ± 1% -1.09% (p=0.000 n=19+19) LoadTree-8 527µs ± 1% 540µs ± 1% +2.37% (p=0.000 n=19+20) LoadBlob-8 5.65ms ± 0% 5.64ms ± 0% -0.21% (p=0.000 n=19+18) LoadAndDecrypt-8 7.07ms ± 2% 5.93ms ± 0% -16.15% (p=0.000 n=19+20) LoadIndex-8 32.1ms ± 2% 25.1ms ± 0% -21.64% (p=0.000 n=20+18) name old speed new speed delta PackerManager-8 296MB/s ± 0% 296MB/s ± 0% ~ (p=0.159 n=20+20) SaveAndEncrypt-8 138MB/s ± 1% 139MB/s ± 1% +1.10% (p=0.000 n=19+19) LoadBlob-8 177MB/s ± 0% 177MB/s ± 0% +0.21% (p=0.000 n=19+18) LoadAndDecrypt-8 141MB/s ± 2% 169MB/s ± 0% +19.24% (p=0.000 n=19+20) name old alloc/op new alloc/op delta PackerManager-8 91.8kB ± 0% 91.8kB ± 0% ~ (p=0.826 n=19+12) IndexAlloc-8 786MB ± 0% 786MB ± 0% +0.01% (p=0.000 n=20+20) SaveAndEncrypt-8 21.0MB ± 0% 21.0MB ± 0% -0.00% (p=0.012 n=20+19) name old allocs/op new allocs/op delta PackerManager-8 1.41k ± 0% 1.41k ± 0% ~ (all equal) IndexAlloc-8 977k ± 0% 977k ± 0% +0.01% (p=0.022 n=20+20) SaveAndEncrypt-8 73.0 ± 0% 73.0 ± 0% ~ (all equal)	2020-07-19 13:58:05 +02:00
Lars Lehtonen	9ac90cf5cd	internal/fuse: fix dropped test error	2020-07-12 21:42:31 -07:00
greatroar	58719e1f47	Replace mount's per-file cache by a global LRU cache	2020-07-12 18:27:16 +02:00
greatroar	d42c169458	Fix quadratic file reading in restic mount	2020-07-12 18:27:16 +02:00
greatroar	02bec13ef2	Fix repository_test.BenchmarkSaveAndEncrypt ... The benchmark was actually testing the speed of index lookups. name old time/op new time/op delta SaveAndEncrypt-8 101ns ± 2% 31505824ns ± 1% +31311591.31% (p=0.000 n=10+10) name old speed new speed delta SaveAndEncrypt-8 41.7TB/s ± 2% 0.0TB/s ± 1% -100.00% (p=0.000 n=10+10) name old alloc/op new alloc/op delta SaveAndEncrypt-8 1.00B ± 0% 20989508.40B ± 0% +2098950740.00% (p=0.000 n=10+10) name old allocs/op new allocs/op delta SaveAndEncrypt-8 0.00 123.00 ± 0% +Inf% (p=0.000 n=10+9) (The actual speed is ca. 131MiB/s.)	2020-07-05 17:41:42 +02:00
MichaelEischer	212607dc8a	Merge pull request #2760 from greatroar/backend-benchmark ... Fix backend benchmarks + a micro-optimization	2020-06-17 23:17:05 +02:00
greatroar	190d8e2f51	Flatten backend.LimitedReadCloser structure ... This inlines the io.LimitedReader into the LimitedReadCloser body to achieve fewer allocations. Results on linux/amd64: name old time/op new time/op delta Backend/BenchmarkLoadPartialFile-8 412µs ± 4% 413µs ± 4% ~ (p=0.634 n=17+17) Backend/BenchmarkLoadPartialFileOffset-8 455µs ±13% 441µs ±10% ~ (p=0.072 n=20+18) name old speed new speed delta Backend/BenchmarkLoadPartialFile-8 10.2GB/s ± 3% 10.2GB/s ± 4% ~ (p=0.817 n=16+17) Backend/BenchmarkLoadPartialFileOffset-8 9.25GB/s ±12% 9.54GB/s ± 9% ~ (p=0.072 n=20+18) name old alloc/op new alloc/op delta Backend/BenchmarkLoadPartialFile-8 888B ± 0% 872B ± 0% -1.80% (p=0.000 n=15+15) Backend/BenchmarkLoadPartialFileOffset-8 888B ± 0% 872B ± 0% -1.80% (p=0.000 n=15+15) name old allocs/op new allocs/op delta Backend/BenchmarkLoadPartialFile-8 18.0 ± 0% 17.0 ± 0% -5.56% (p=0.000 n=15+15) Backend/BenchmarkLoadPartialFileOffset-8 18.0 ± 0% 17.0 ± 0% -5.56% (p=0.000 n=15+15)	2020-06-17 13:11:45 +02:00
greatroar	f4cd2a7120	Make backend benchmarks fairer by removing checks ... Checking whether the right data is returned takes up half the time in some benchmarks. Results for local backend benchmarks on linux/amd64: name old time/op new time/op delta Backend/BenchmarkLoadFile-8 4.89ms ± 0% 2.72ms ± 1% -44.26% (p=0.008 n=5+5) Backend/BenchmarkLoadPartialFile-8 936µs ± 6% 439µs ±15% -53.07% (p=0.008 n=5+5) Backend/BenchmarkLoadPartialFileOffset-8 940µs ± 1% 456µs ±10% -51.50% (p=0.008 n=5+5) Backend/BenchmarkSave-8 23.9ms ±14% 24.8ms ±41% ~ (p=0.690 n=5+5) name old speed new speed delta Backend/BenchmarkLoadFile-8 3.43GB/s ± 0% 6.16GB/s ± 1% +79.40% (p=0.008 n=5+5) Backend/BenchmarkLoadPartialFile-8 4.48GB/s ± 6% 9.63GB/s ±14% +114.78% (p=0.008 n=5+5) Backend/BenchmarkLoadPartialFileOffset-8 4.46GB/s ± 1% 9.22GB/s ±10% +106.74% (p=0.008 n=5+5) Backend/BenchmarkSave-8 706MB/s ±13% 698MB/s ±31% ~ (p=0.690 n=5+5)	2020-06-17 13:11:45 +02:00
Alexander Weiss	1361341c58	don't save duplicate packIDs when using internal/repository/Index.Store	2020-06-14 07:56:24 +02:00
Alexander Weiss	ce4a2f4ca6	save packIDs and duplicates separately ... A side remark to the definition of Index.blob: Another possibility would have been to use: blob map[restic.BlobHandle]*indexEntry This would have led to the following sizes: key: 32 + 1 = 33 bytes value: 8 bytes indexEntry: 8 + 4 + 4 = 16 bytes each packID: 32 bytes To save N index entries, we would therefore have needed: N * OF * (33 + 8) bytes + N * 16 + N * 32 bytes / BP = N * 82 bytes More precicely, using a pointer instead of a direct entry is the better memory choice if: OF * 8 bytes + entrysize < OF * entrysize <=> entrysize > 8 bytes * OF/(OF-1) Under the assumption of OF=1.5, this means using pointers would have been the better choice if sizeof(indexEntry) > 24 bytes.	2020-06-14 07:56:21 +02:00
Alexander Weiss	cf979e2b81	make offset and length uint32	2020-06-14 07:50:19 +02:00
Michael Eischer	d92e2c5769	simplify index code	2020-06-14 07:50:19 +02:00
Alexander Weiss	7419844885	add changelog, benchmark, memory calculation	2020-06-14 07:50:15 +02:00
Alexander Weiss	d3c59d18e5	Fix inconsistency of saving/loading config file ... Fix saving/loading config file: Always set ID to a zero ID.	2020-06-13 16:30:23 +02:00
MichaelEischer	dd7b4f54f5	Merge pull request #2709 from greatroar/minio-sha256 ... Use Minio's optimized SHA-256	2020-06-12 23:32:58 +02:00
MichaelEischer	6896c6449b	Merge pull request #2779 from greatroar/archiver-comment ... Fix up comment on archiver.BlobSaver.Save	2020-06-12 23:04:21 +02:00
MichaelEischer	735a8074d5	Merge pull request #2773 from aawsome/index-uploads+knownblobs ... Fix non-intuitive repo behavior	2020-06-12 22:41:04 +02:00
Alexander Weiss	70347e95d5	disable index uploads for prune command ... + modifications of changelog	2020-06-12 09:24:38 +02:00
greatroar	0fa3091c78	Fix up comment on archiver.BlobSaver.Save	2020-06-11 13:40:30 +02:00
Alexander Weiss	91906911b0	Fix non-intuitive repository behavior ... - The SaveBlob method now checks for duplicates. - Moves handling of pending blobs to MasterIndex. -> also cleans up pending index entries when they are saved in the index -> when using SaveBlob no need to care about index any longer - Always check for full index and save it when storing packs. -> removes the need of an index uploader -> also removes the verbose "uploaded intermediate index" messages - The Flush method now also saves the index - Fix race condition when checking and saving full/non-finalized indexes	2020-06-11 13:05:23 +02:00
MichaelEischer	6856d1e422	Merge pull request #2749 from aawsome/fix-fullindex ... Change condition for full index	2020-06-10 20:40:19 +02:00
Alexander Weiss	8c1261ff02	changed condition for full index	2020-06-07 22:00:49 +02:00
MichaelEischer	84475aa3a8	Merge pull request #2730 from greatroar/mount-build-tags ... Simplify build tags for restic mount	2020-05-23 20:33:43 +02:00
Martin Michlmayr	5cc1760fdf	Fix typos	2020-05-16 14:05:26 +08:00
greatroar	649cbec6c5	Simplify build tags for restic mount ... This command can only be built on Darwin, FreeBSD and Linux (and if we upgrade bazil.org/fuse, only FreeBSD and Linux: https://github.com/bazil/fuse/issues/224). Listing the few supported operating systems explicitly here makes porting restic to new platforms easier.	2020-05-12 11:30:41 +02:00
greatroar	9f7cd69f13	Move Index.FindBlob to tests	2020-04-29 10:57:01 +02:00
greatroar	f97a680887	Fix repository benchmarks ... BenchmarkLoad{AndDecrypt,Blob} were spending between 38% and 50% of their time measuring SHA-256 performance in their checks.	2020-04-28 07:57:29 +02:00
greatroar	42a3db05b0	Use Minio's optimized SHA-256 ... internal/repository benchmarks on an Intel i7-3770k: name old speed new speed delta PackerManager-8 209MB/s ± 1% 291MB/s ± 1% +38.94% (p=0.008 n=5+5) SaveAndEncrypt-8 112MB/s ± 1% 135MB/s ± 1% +20.25% (p=0.008 n=5+5)	2020-04-28 07:57:18 +02:00
greatroar	e7d7b85d59	Merge Repository.{LoadBlob,loadBlob} ... Pushing the allocation logic down into the former loadBlob body means that fewer allocations have to be performed: name old time/op new time/op delta LoadTree-8 478µs ± 1% 481µs ± 2% ~ (p=0.315 n=9+10) LoadBlob-8 11.6ms ± 1% 11.6ms ± 2% ~ (p=0.393 n=10+10) LoadAndDecrypt-8 13.3ms ± 3% 13.3ms ± 3% ~ (p=0.905 n=10+9) LoadIndex-8 33.6ms ± 2% 33.2ms ± 1% -1.15% (p=0.028 n=10+9) name old alloc/op new alloc/op delta LoadTree-8 41.2kB ± 0% 41.1kB ± 0% -0.23% (p=0.000 n=10+10) LoadBlob-8 2.28kB ± 0% 2.18kB ± 0% -4.21% (p=0.000 n=10+10) LoadAndDecrypt-8 2.10MB ± 0% 2.10MB ± 0% ~ (all equal) LoadIndex-8 5.22MB ± 0% 5.22MB ± 0% ~ (p=0.631 n=10+10) name old allocs/op new allocs/op delta LoadTree-8 652 ± 0% 651 ± 0% -0.15% (p=0.000 n=10+10) LoadBlob-8 24.0 ± 0% 23.0 ± 0% -4.17% (p=0.000 n=10+10) LoadAndDecrypt-8 30.0 ± 0% 30.0 ± 0% ~ (all equal) LoadIndex-8 30.2k ± 0% 30.2k ± 0% ~ (p=0.610 n=10+10) name old speed new speed delta LoadBlob-8 86.4MB/s ± 1% 85.9MB/s ± 2% ~ (p=0.393 n=10+10) LoadAndDecrypt-8 75.4MB/s ± 3% 75.4MB/s ± 3% ~ (p=0.858 n=10+9)	2020-04-23 10:04:20 +02:00
greatroar	be5a0ff59f	Centralize buffer allocation and size checking in Repository.LoadBlob ... Benchmark results for internal/repository: name old time/op new time/op delta LoadTree-8 479µs ± 2% 478µs ± 1% ~ (p=0.780 n=10+9) LoadBlob-8 11.6ms ± 2% 11.6ms ± 1% ~ (p=0.631 n=10+10) LoadAndDecrypt-8 13.2ms ± 2% 13.3ms ± 3% ~ (p=0.631 n=10+10) name old alloc/op new alloc/op delta LoadTree-8 41.2kB ± 0% 41.2kB ± 0% ~ (all equal) LoadBlob-8 2.28kB ± 0% 2.28kB ± 0% ~ (all equal) LoadAndDecrypt-8 2.10MB ± 0% 2.10MB ± 0% ~ (all equal) name old allocs/op new allocs/op delta LoadTree-8 652 ± 0% 652 ± 0% ~ (all equal) LoadBlob-8 24.0 ± 0% 24.0 ± 0% ~ (all equal) LoadAndDecrypt-8 30.0 ± 0% 30.0 ± 0% ~ (all equal) name old speed new speed delta LoadBlob-8 86.2MB/s ± 2% 86.4MB/s ± 1% ~ (p=0.594 n=10+10) LoadAndDecrypt-8 75.7MB/s ± 2% 75.4MB/s ± 3% ~ (p=0.617 n=10+10)	2020-04-23 10:04:20 +02:00
MichaelEischer	4f00564574	Merge pull request #2621 from greatroar/fixes ... Some small fixes	2020-04-18 18:07:08 +02:00
MichaelEischer	f77477129f	Merge pull request #2584 from greatroar/mount-cache-uid-gid ... Cache uid and gid for top directories in internal/fuse	2020-04-18 17:45:14 +02:00
greatroar	2e31120f89	Remove unused argument to restic.fakeFile	2020-04-18 17:40:13 +02:00
greatroar	8fb2c0d3c1	Typo in crypto test name	2020-04-18 17:39:06 +02:00
greatroar	072cf7b02d	Fix debug messages in internal/fuse	2020-04-18 17:39:06 +02:00
greatroar	df66daa5c9	Fix context usage in backend tests ... Found by go vet. This is also the only complaint is has.	2020-04-18 17:39:06 +02:00
MichaelEischer	16710454f4	Merge pull request #2628 from MichaelEischer/one-element-pack-lists ... cache: Don't sort one element pack lists	2020-04-18 17:09:06 +02:00
MichaelEischer	7910ff4c0e	Merge pull request #2648 from nairb774/iowritestring ... termstatus: Use io.WriteString to output messages.	2020-04-18 13:48:42 +02:00
MichaelEischer	c4da9d1e90	Merge pull request #2638 from greatroar/no-close-in-packer ... Don't Close in Packer.Finalize	2020-04-18 13:07:21 +02:00
MichaelEischer	a1352906e2	Merge pull request #2622 from greatroar/optimize-packer-manager ... Fix PackerManager benchmark and optimize hashing.Writer	2020-04-18 12:46:34 +02:00
MichaelEischer	b7c0d4d8bf	Merge pull request #2644 from greatroar/signal-notify-buffered ... Make all signal.Notify channels buffered	2020-04-18 11:31:02 +02:00
Michael Eischer	bdf7ba20cb	archiver: Fix race condition triggered by TestArchiverAbortEarlyOnError ... The Save methods of the BlobSaver, FileSaver and TreeSaver return early on when the archiver is stopped due to an error. For that they select on both the tomb.Dying() and context.Done() channels, which can lead to a race condition when the tomb is killed due to an error: The tomb first closes its Dying channel before canceling all child contexts. Archiver.SaveDir only aborts its execution once the context was canceled. When the tomb killing is paused between closing its Dying channel and canceling the child contexts, this lets the FileSaver/TreeSaver.Save methods return immediately, however, ScanDir still reads further files causing the test case to fail. As a killed tomb always cancels all child contexts and as the Savers always use a context bound to the tomb, it is sufficient to just use context.Done() as escape hatch in the Save functions. This fixes the mismatch between SaveDir and Save. Adjust the tests to use contexts bound to the tomb for all interactions with the Savers.	2020-04-13 18:23:17 +02:00
Brian Atkinson	b8da7b1f4d	termstatus: Use io.WriteString to output messages. ... The previous implementation was repeating the implementation that is found inside of io.WriteString. Simplify by making use of the stdlib's implementation.	2020-03-26 14:55:00 -07:00
Peter Schultz	90fc639a67	Allow specifying user and host when adding keys ... The username and hostname for new keys can be specified with the new --user and --host flags, respectively. The flags are used only by the `key add` command and are otherwise ignored. This allows adding keys with for a desired user and host without having to run restic as that particular user on that particular host, making automated key management easier. Co-authored-by: James TD Smith <ahktenzero@mohorovi.cc>	2020-03-23 13:11:10 +01:00
rawtaz	9efbe98879	Merge pull request #2623 from alrs/internal-restic-err-before-close ... internal/restic: close os.File after checking for error	2020-03-18 22:23:20 +01:00
rawtaz	7d9300efca	Merge pull request #2637 from greatroar/unused ... Remove Go 1.5 compatibility code	2020-03-18 22:22:15 +01:00
greatroar	47d4d5bf1b	Make all signal.Notify channels buffered	2020-03-12 20:59:39 +01:00
greatroar	74a64c47e4	Move testing logic to test file in internal/pack	2020-03-09 14:32:28 +01:00
greatroar	a23e9c86ba	Remove closing logic from Packer.Finalize ... The method only ever receives *hashing.Writers, which don't implement io.Closer. These come from packerManager.findPacker and have their actual writers closed in Repository.savePacker. Moving the closing logic to hashing.Writer results in "file already closed" errors.	2020-03-09 14:31:45 +01:00
greatroar	4de12bf593	Remove restic.RandReader ... math/rand.Rand has implemented Reader since Go 1.6. The repacking tests are not deterministic, but they weren't before, either.	2020-03-09 10:00:28 +01:00
greatroar	8cf3bb8737	Revert "Put host last in SSH command line" ... This reverts commit e1969d1e33.	2020-03-08 16:45:33 +01:00
Michael Eischer	b46cc6d57e	repository: Don't sort one element pack lists ... When loading a blob, restic first looks up pack files containing the blob. To avoid unnecessary work an already cached pack file is preferred. However, if there is only a single pack file to choose from (which is the normal case) sorting the one-element list won't change anything. Therefore avoid the unnecessary cache check in that case.	2020-03-07 10:26:06 +01:00
Lars Lehtonen	4a2156d3f0	internal/restic: close os.File after checking for error	2020-03-05 16:22:46 -08:00
greatroar	41fee11f66	Micro-optimization for hashing.Writer/PackerManager ... name old time/op new time/op delta PackerManager-8 247ms ± 1% 246ms ± 1% -0.43% (p=0.001 n=18+18) name old speed new speed delta PackerManager-8 213MB/s ± 1% 214MB/s ± 1% +0.43% (p=0.001 n=18+18) name old alloc/op new alloc/op delta PackerManager-8 92.2kB ± 0% 91.5kB ± 0% -0.82% (p=0.000 n=19+20) name old allocs/op new allocs/op delta PackerManager-8 1.43k ± 0% 1.41k ± 0% -1.67% (p=0.000 n=20+20)	2020-03-05 22:30:04 +01:00
greatroar	b592614061	Improve PackerManager benchmark ... The previous benchmark spent much of its time allocating RNGs and generating too many random numbers. It now spends 90% of its time hashing and half of the rest writing to files. name old time/op new time/op delta PackerManager-8 319ms ± 1% 247ms ± 1% -22.48% (p=0.000 n=20+18) name old speed new speed delta PackerManager-8 143MB/s ± 1% 213MB/s ± 1% +48.63% (p=0.000 n=10+18) name old alloc/op new alloc/op delta PackerManager-8 635kB ± 0% 92kB ± 0% -85.48% (p=0.000 n=10+19) name old allocs/op new allocs/op delta PackerManager-8 1.64k ± 0% 1.43k ± 0% -12.76% (p=0.000 n=10+20)	2020-03-05 22:30:03 +01:00
greatroar	b7c3039eb2	Remove Go 1.5 compatibility code from PackerManager benchmark ... This alone is enough to speed up the benchmark by ~10%.	2020-03-05 22:29:06 +01:00
Alexander Neumann	52abec967f	Merge pull request #2605 from middelink/fix-2604 ... Fix running tests on a SELinux enabled system	2020-03-01 20:25:28 +01:00
Pauline Middelink	2828a9c2b0	Fix running tests on a SELinux enabled system ... Archivers TestMetadataChanged incorrectly clears the Extended Attributes from the expected metadata of the temporary file. This is incorrect as on SELinux enabled filesystem, as the kernel will automaticly add a SElinux label. However, since ExtendedAttributes{} != ExtendedAttributes{nil} we still need to clear them if there are no attributes found.	2020-03-01 20:23:22 +01:00
Alexander Neumann	95da6c1c1d	Merge pull request #2589 from greatroar/no-stable-sort ... Replace sort.Stable by sort.Strings	2020-03-01 19:40:28 +01:00
Alexander Neumann	0c03a80fc4	Merge pull request #2592 from greatroar/sftp-ipv6 ... Support IPv6 in SFTP backend	2020-03-01 19:38:44 +01:00
Alexander Neumann	c50f91b7f9	Merge pull request #2602 from greatroar/no-bufpool ... Remove sync.Pool from internal/repository	2020-03-01 10:50:57 +01:00
Alexander Neumann	e851d29565	Merge pull request #2608 from greatroar/simplify-termstatus ... Simplify termstatus	2020-03-01 10:48:29 +01:00
Alexander Neumann	b67b7ebfe6	Merge pull request #2583 from greatroar/unused ... Remove some unused or duplicated code	2020-03-01 10:46:17 +01:00
greatroar	751eba0e68	Remove unnecessary pipe checks in termstatus ... canUpdateStatus has already determined that the file descriptor is not a pipe.	2020-02-29 18:03:49 +01:00
greatroar	7447c44484	Use golang.org/x/sys/windows in termstatus ... Some functionality is missing, but at least the types are all defined. Replaced short, word, dword by their Go names to match the x/sys convention.	2020-02-29 18:03:49 +01:00
greatroar	c8a672fa29	Remove code copy-pasted from x/crypto/ssh/terminal	2020-02-29 18:03:49 +01:00
greatroar	863ba76494	Drop mattn/go-isatty in favor of crypto/ssh/terminal	2020-02-29 18:03:47 +01:00
greatroar	8526cc6647	Remove sync.Pool from internal/repository ... The pool was used improperly, causing more allocations to be performed than without it. name old time/op new time/op delta SaveAndEncrypt-8 36.8ms ± 2% 36.9ms ± 2% ~ (p=0.218 n=10+10) name old speed new speed delta SaveAndEncrypt-8 114MB/s ± 2% 114MB/s ± 2% ~ (p=0.218 n=10+10) name old alloc/op new alloc/op delta SaveAndEncrypt-8 21.1MB ± 0% 21.0MB ± 0% -0.44% (p=0.000 n=10+10) name old allocs/op new allocs/op delta SaveAndEncrypt-8 79.0 ± 0% 77.0 ± 0% -2.53% (p=0.000 n=10+10)	2020-02-29 17:54:46 +01:00
greatroar	5cd0bce452	Honor RESTIC_CACHE_DIR on Mac and Windows	2020-02-28 15:44:32 +01:00
rawtaz	58bd165253	Merge pull request #2581 from aawsome/multiple-hostnames ... Allow multiple hostnames tags	2020-02-27 08:35:23 +01:00
rawtaz	65d3fb6b33	Merge pull request #2603 from greatroar/restorer-waitgroup ... Fix unsafe sync.WaitGroup usage in restorer.fileRestorer	2020-02-27 00:30:59 +01:00
greatroar	de5516a90e	Fix sync.WaitGroup usage in restorer.fileRestorer	2020-02-27 00:07:49 +01:00
greatroar	4f6fd9fb98	Remove remnant of Go 1.9 compatibility code from tests	2020-02-26 22:23:38 +01:00
Alexander Weiss	9a9101d144	Support specifying multiple host flags for various commands ... The `dump`, `find`, `forget`, `ls`, `mount`, `restore`, `snapshots`, `stats` and `tag` commands will now take into account multiple `--host` and `-H` flags.	2020-02-26 22:17:59 +01:00
Alexander Neumann	99fd80a585	Remove all workarounds for Go < 1.11	2020-02-26 20:35:13 +01:00
Igor Fedorenko	f17ffa0283	restorer: Allow writing target file blobs out of order ... Much simpler implementation that guarantees each required pack is downloaded only once (and hence does not need to manage pack cache). Also improves large file restore performance. Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>	2020-02-26 16:14:45 +01:00
greatroar	5e2afd91e7	Assert that archiver.Tree implements fmt.Stringer	2020-02-26 11:05:38 +01:00
greatroar	79b882e901	Merge duplicated readdir functionality ... internal/archiver.readdir and internal/fs.ReadDir were unused. internal/fs.ReadDirNames and internal/archiver.readdirnames were doing nearly the same thing, except one sorted its output and opened with fs.O_NOFOLLOW. Both were only used in internal/archiver.	2020-02-26 11:05:38 +01:00
greatroar	1b502fa9ef	Cache uid and gid for top directories in internal/fuse	2020-02-24 10:46:09 +01:00
greatroar	e1969d1e33	Put host last in SSH command line ... This is how the SSH manpage says the command line should look, and the "--" prevents mistakes in hostnames from being interpreted as options.	2020-02-19 15:53:20 +01:00
greatroar	6ac6bca7a1	Support IPv6 in SFTP backend ... The previous code was doing its own hostname:port splitting, which caused IPv6 addresses to be misinterpreted.	2020-02-19 15:42:12 +01:00
greatroar	3a6feb0596	Replace sort.Stable by sort.Strings ... Calling the slow, O(n lg² n) sort.Stable is equivalent to sort.Strings for a slice of unique strings.	2020-02-18 19:41:06 +01:00
greatroar	2f8aa2ce30	Remove unused fs.FS from archiver.FileSaver	2020-02-18 10:39:14 +01:00
Michael Eischer	e1f722d266	archiver: Fix flaky TestArchiverAbortEarlyOnError ... Each of the random test files was split into the same five blobs. The test fails once the fifth blob is passed on to `SaveBlob`. That is for certain interleavings of goroutine execution it would be possible for the test to trigger the testErr just after storing the first file. The fixed test uses a different file content for each of the nine files and fails after writing the fourth blob. The file content is also small enough to ensure that for each file only a single blob is saved. This guarantees that the test cannot fail before reading the first four files. FileReadConcurrency = 2 allows up to two files queued for processing. Therefore the test can at most open the sixth file before it has to save the fourth file / blob which triggers the testErr.	2020-02-14 23:16:13 +01:00
Michael Eischer	42a3292bcf	Better name for jsonstatus package ... internal/ui/jsonstatus and termstatus sound similar but are not related in any way. Instead `internal/ui/backup` and `internal/ui/jsonstatus/status` are the counterparts. Rename the latter to `internal/ui/json/backup` to make this clear.	2020-02-13 21:14:20 +01:00
Michael Eischer	ef70a2fcb3	Fix mangled JSON output by backup command ... jsonstatus wrote the JSON output without synchronization to the stdio_wrapper which caused mangling between different status lines. Use the Print and Error methods of termstatus instead which use a central goroutine to synchronize output.	2020-02-13 21:14:20 +01:00
rawtaz	680a14afa1	Merge pull request #2530 from restic/fix-sftp-mkdirall ... sftp: Use MkdirAll provided by the client	2020-02-13 01:12:59 +01:00
rawtaz	d8da9c4401	Merge pull request #2577 from alrs/fix-internal-errs ... internal: Fix code and test dropped errors	2020-02-12 23:41:54 +01:00
rawtaz	299f5971f2	Merge pull request #2560 from brualan/master ... Two small improvements to code quality	2020-02-12 22:40:20 +01:00
Lars Lehtonen	72734d59b5	internal/archiver: fix dropped error	2020-02-12 13:37:37 -08:00
Lars Lehtonen	3ed54e762e	internal/backend/sftp: fix dropped test error	2020-02-12 13:36:21 -08:00
Lars Lehtonen	fea835b4e2	internal/repository: fix dropped test error	2020-02-12 13:33:54 -08:00
Lars Lehtonen	16b321b140	internal/restic: fix dropped test error	2020-02-12 13:32:45 -08:00
Michael Eischer	760863e7f9	archiver: Fix TestRacyFileSwap on windows	2020-02-11 21:09:47 +01:00
Michael Eischer	a135699397	Close file if file type has changed after initial stat	2020-02-11 21:09:47 +01:00
Lars Lehtonen	57815d9cd6	restic/internal/ui: fix gofmt nit that popped up in Go 1.13	2020-02-10 11:06:47 -08:00
Daniel Hoffend	e7cdf2acbb	fix backup --json total_bytes_processed output ... Closes #2429	2020-02-09 02:02:45 +01:00
Alexander Bruyako	38ddfbc4d3	simpler error return	2020-01-27 18:41:46 +03:00
Alexander Bruyako	da48b925ff	remove unnecessary error return ... I was running "golangci-lint" and found this two warnings internal/checker/checker.go:135:18: (*Checker).LoadIndex$3 - result 0 (error) is always nil (unparam) final := func() error { ^ internal/repository/repository.go:457:18: (*Repository).LoadIndex$3 - result 0 (error) is always nil (unparam) final := func() error { ^ It turns out that these functions are used only in "RunWorkers(...)", which is used only two times in whole project right after this "final" functions. And because these "final" functions always return "nil", I've descided, that it would be better to remove requriments for "final" func to return error to avoid magick "return nil" at their end.	2020-01-27 18:28:21 +03:00
rawtaz	d70a4a9350	Merge pull request #2373 from vrusinov/issue-2372 ... Ignore username difference in TestMetadataChanged	2020-01-05 21:07:34 +01:00
Alexander Neumann	2cd9c7ef16	sftp: Use MkdirAll provided by the client ... Closes #2518	2020-01-01 17:26:38 +01:00
Ward Vandewege	69a0d0ee90	fix typos	2019-12-21 21:00:28 -05:00
streambinder	97e5ce4344	internal: backend: sftp: support user@domain parsing as user	2019-12-19 13:15:37 +01:00
rawtaz	da4473aba6	Merge pull request #2391 from kastenhq/refresh-lock-time ... Update Lock.Time in lock.Refresh()	2019-11-26 21:44:29 +01:00
rawtaz	6e85a58045	Merge pull request #2465 from whs-dot-hk/fix-test-metadata-changed ... Fix test metadata changed	2019-11-26 21:42:42 +01:00
rawtaz	e14c4b1737	Merge pull request #2484 from restic/add-s3-region ... s3: Allow specifying region	2019-11-22 15:51:17 +01:00
Alexander Neumann	409909a7f5	Add option description for Region	2019-11-22 15:09:09 +01:00
mdauphin	df500a372d	Add AWS_REGION env var to specify s3 region	2019-11-22 15:04:48 +01:00
Alexander Neumann	a6e8af7e0f	Update minio-go	2019-11-22 14:50:46 +01:00
rawtaz	41fe9318b1	Merge pull request #2425 from thiell/restic_cache_dir_env ... Add support for $RESTIC_CACHE_DIR	2019-11-20 21:18:57 +01:00
Leo R. Lundgren	4d2aa18273	Accept both singular and plural for --group-by (#2330)	2019-11-20 19:25:48 +01:00
rawtaz	26d1f9f4ba	Merge pull request #2307 from gary-kim/fix-2306-password-retries ... Allow multiple retries for interactive password input	2019-11-20 18:30:20 +01:00
whs	7cacba0394	Assume WithAtime default to false	2019-11-06 16:38:46 +08:00
Stephane Thiell	0e897ef7b8	Add support for $RESTIC_CACHE_DIR ... Add support for restic-specific $RESTIC_CACHE_DIR environment variable to override the cache directory like --cache-dir would have.	2019-09-26 15:59:56 -07:00
Eric Hamilton	1596d06f8e	Update Lock.Time in lock.Refresh()	2019-09-04 11:38:35 -07:00
Vladimir Rusinov	db20c0b8d0	Ignore username difference in TestMetadataChanged. ... In some (rare) cases "fake" UID 51234 may exist in a system running a test. When this is the case, `cmp.Equal(want, node3)` will fail based on difference between empty string and an actual username present in a system. Fixes github issue #2372	2019-08-13 22:25:00 +01:00
Alexander Neumann	01c51b3449	Merge pull request #2321 from restic/fix-index-error-handling ... Check errors returned by LoadIndex()	2019-07-27 09:37:18 +02:00
Alexandr Bruyako	e19622e4b1	start using fileMode const	2019-07-01 00:26:00 +03:00
Alexandr Bruyako	38ea7ed4f6	remove unused code	2019-07-01 00:24:45 +03:00
Alexandr Bruyako	76d1866444	avoiding unnecessary type conversions	2019-06-30 23:58:00 +03:00
Alexandr Bruyako	8b22fe29cf	improved slice copying	2019-06-30 23:56:36 +03:00
Alexandr Bruyako	02014be76c	simplified prefix removal, removed unnecessary if-else statements	2019-06-30 23:34:47 +03:00
Alexandr Bruyako	16eeed2ad5	simplified string sorting by using a more suitable function	2019-06-30 23:20:32 +03:00
Alexander Neumann	88716794e3	Check errors returned by LoadIndex() ... Bug was reported in the forum here: https://forum.restic.net/t/check-rebuild-index-prune/1848/13	2019-06-30 21:34:53 +02:00
Gary Kim	fea2464d4d	Allow multiple retries for interactive password input ... Restic used to quit if the repository password was typed incorrectly once. Restic will now ask the user again for the repository password if typed incorrectly. The user will now get three tries to input the correct password before restic quits.	2019-06-13 20:11:02 +08:00
Garry McNulty	8066195e6e	fs: Handle absolute pathname for --stdin-filename ... Return valid directory info from Lstat() for parent directories of the specified filename. Previously only "/" and "." were valid directories. Also set directory mode as this is checked by archiver. Closes #2063	2019-05-08 15:21:27 +02:00
Garry McNulty	f7f14cf8c9	fs: Add file info base name check in reader tests (#2063)	2019-05-08 15:21:27 +02:00
Garry McNulty	5096f3b491	fs: Update directory check in reader tests (#2063)	2019-05-08 15:21:27 +02:00
Alexander Neumann	920d458a4a	archiver: Use untyped constants for testing FileInfo	2019-05-05 14:57:38 +02:00
Alexander Neumann	b016dc2ff0	archiver/Windows: Skip test new-content-same-filestamp	2019-05-05 14:02:11 +02:00
Alexander Neumann	355db0bc29	windows: Use LastWriteTime for ctime and mtime ... Windows does not have a concept of a `change time` in the sense as Unix has it: the field `CreationTime` of the `Win32FileAttributeData` struct is not updated when attributes or content is changed. So from now on we're using the `LastWriteTime` as the `change time` on Windows.	2019-05-05 14:02:11 +02:00
Alexander Neumann	6e2fe73189	archiver: Move tests back into the same file ... Move all Archiver tests back into `archiver_test.go` and add some tiny helpers to mock what `lstat` returns (for Windows and Unix separately).	2019-05-05 14:02:11 +02:00
Alexander Neumann	303a5dab6a	archiver: Clarify value in test struct ... Since I could not remember what the value for `Check` means this commit renames it to `SameFile`: when set to true, the test should make sure that `FileChanged` should return false (=file is unmodified).	2019-05-05 12:57:00 +02:00
Gábor Lipták	bce6438d22	Don't run TestMetadataChanged test on Windows ... Signed-off-by: Gábor Lipták <gliptak@gmail.com>	2019-04-27 21:23:47 -04:00
Alexander Neumann	919dd2ac84	Merge pull request #2252 from restic/fix-2249 ... Read fresh metadata for unmodified files	2019-04-25 09:15:50 +02:00
Alexander Neumann	870bc5108e	Merge pull request #2253 from restic/fix-2174 ... Make sure timestamps are valid	2019-04-25 09:14:43 +02:00
Alexander Neumann	a6481b3707	Merge pull request #2212 from cbane/check-ctime ... Examine file ctime when checking if files have changed.	2019-04-25 08:58:27 +02:00
Courtney Bane	0ebfc55ee3	Use existing setTimestamp function for ctime test and improve error checking.	2019-04-24 20:53:08 -05:00
Courtney Bane	35b7607802	Don't check ctime when ignoring inode.	2019-04-24 20:53:08 -05:00
Alexander Neumann	939f3e972c	node: Make sure year of all timestamps is valid ... Sometimes restic gets bogus timestamps which cannot be converted to JSON, because the stdlib JSON encoder returns an error if the year is not within [0, 9999]. We now make sure that we at least record _some_ timestamp and cap the year either to 0000 or 9999. Before, restic would refuse to save the file at all, so this improves the status quo. This fixes #2174 and #1173	2019-04-24 16:39:36 +02:00
Alexander Neumann	389067fb8b	Only use list of blobs for old node ... Closes #2249	2019-04-24 15:07:26 +02:00
Alexander Neumann	4b0ca9ddab	Add test for #2249	2019-04-24 15:07:23 +02:00
Courtney Bane	b8c2544dcb	Examine file ctime when checking if files have changed.	2019-04-23 21:54:35 -05:00
Alexander Neumann	18ec27a0da	Merge pull request #2243 from treymo/ls-length-2104 ... backend find: Check prefix length < snapshot IDs when searching	2019-04-23 09:41:19 +02:00
Jan Niklas Richter	733519d895	Refactor duplicate code for grouping snapshots ... This commit is a followup to the addition of the --group-by flag for the snapshots command. Adding the grouping code there introduced duplicated code (the forget command also does grouping). This commit refactors boths sides to only use shared code.	2019-04-22 14:57:53 +02:00
Trey Moore	5ac9c1157a	Prevent "slice bounds out of range" error if prefix is longer than snapshot IDs. ... This includes tests as well as other tests for the backend_find.Find function. Issue #2104.	2019-04-16 20:26:20 -07:00
Alexander Neumann	6b700d02f5	Merge pull request #2217 from restic/improve-memory-usage ... WIP: improve memory usage	2019-04-13 15:07:07 +02:00
Alexander Neumann	7304738872	check: Reduce default parallelism from 40 to 5	2019-04-13 13:38:39 +02:00
Alexander Neumann	66efa425bf	Reuse buffer in worker functions	2019-04-13 13:38:39 +02:00
Alexander Neumann	d51e9d1b98	Add []byte to repo.LoadAndDecrypt and utils.LoadAll ... This commit changes the signatures for repository.LoadAndDecrypt and utils.LoadAll to allow passing in a []byte as the buffer to use. This buffer is enlarged as needed, and returned back to the caller for further use. In later commits, this allows reducing allocations by reusing a buffer for multiple calls, e.g. in a worker function.	2019-04-13 13:38:39 +02:00
Alexander Neumann	e046428c94	Replace FilesInParallel with an errgroup.Group	2019-04-13 13:38:39 +02:00
Alexander Neumann	75906edef5	mock: Remove unused repository type	2019-04-13 13:38:39 +02:00
Alexander Neumann	203d775190	restic: Make JSON unmarshal for ID more efficient ... This commit reduces several allocations in UnmarshalJSON() by decoding the hex string directly in a single step.	2019-04-13 13:38:39 +02:00
Benoît Knecht	3c112d9cae	s3: Add config option to set storage class ... The `s3.storage-class` option can be passed to restic (using `-o`) to specify the storage class to be used for S3 objects created by restic. The storage class is passed as-is to S3, so it needs to be understood by the API. On AWS, it can be one of `STANDARD`, `STANDARD_IA`, `ONEZONE_IA`, `INTELLIGENT_TIERING` and `REDUCED_REDUNDANCY`. If unspecified, the default storage class is used (`STANDARD` on AWS). You can mix storage classes in the same bucket, and the setting isn't stored in the restic repository, so be sure to specify it with each command that writes to S3. Closes #706	2019-03-26 16:37:07 +01:00
Alexander Neumann	c1058005c3	Fix build on FreeBSD for Go < 1.12	2019-03-16 20:52:37 +01:00
Alexander Neumann	ca73808649	Merge pull request #2197 from dmgk/master ... Fix build on FreeBSD with Go 1.12	2019-03-16 20:23:19 +01:00
Igor Fedorenko	bf9a507148	Fix rare 'file already closed' during restore ... Fixes #2183 Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>	2019-03-16 17:01:20 +01:00
Alexander Neumann	65b476ead9	Fix gofmt	2019-03-16 13:29:05 +01:00
Alexander Neumann	aaa1cc2c26	Merge pull request 2193 from restic/allow-empty-rclone-args ... rclone: Rework backend option parsing	2019-03-16 12:17:38 +01:00
Alexander Neumann	95434cff16	Merge pull request 2156 from kayrus/swift-app-cred ... Swift: introduce application credential auth support	2019-03-16 12:17:17 +01:00
Heiko Bornholdt	db8f5864fc	Add --ignore-inode option to backup cmd ... revised version of https://github.com/restic/restic/pull/2047	2019-03-10 21:24:29 +01:00
Dmitri Goutnik	1d8b21cdad	Fix build on FreeBSD with Go 1.12	2019-03-06 09:41:49 -05:00
Alexander Neumann	3865b59716	rclone: Rework backend option parsing ... This change allows passing no arguments to rclone, using `-o rclone.args=""`. It is helpful when running rclone remotely via SSH using a key with a forced command (via `command=` in `authorized_keys`).	2019-03-02 10:36:42 +01:00
Alexander Neumann	e277a92a2f	self-update: Don't cancel download after 30 seconds	2019-02-23 11:15:18 +01:00
kayrus	6ebcfe7c18	Swift: introduce application credential auth support	2019-02-14 14:19:05 +01:00
Alexander Neumann	fed25714a4	Merge pull request 2139 from restic/fail-zero-bytes-stdin ... Return error when reading zero byte from stdin	2019-02-10 12:34:05 +01:00
Alexander Neumann	8906d85ab8	Merge pull request 2138 from restic/fix-2136 ... Use processed bytes for summary	2019-02-10 12:32:50 +01:00
Matthew Holt	6cf13483b5	Add snapshot ID to summary output	2019-02-10 12:09:09 +01:00
Matthew Holt	f645306a18	Initial implementation of JSON progress reporter	2019-02-10 12:09:09 +01:00
Alexander Neumann	1f246c5309	Merge pull request 2090 from plumbeo/within-hours ... Increase the granularity of the "keep within" snapshot retention policy	2019-01-06 15:04:55 +01:00
Alexander Neumann	c4fbf2c779	Return error when reading zero byte from stdin ... This commit changes the internal file system implementation for reading data from stdin, it now returns an error when no bytes could be read. I think it's worth failing in this case, the user instructed restic to read some data from stdin, and no data was read at all. Maybe it was in a pipe and some earlier stage failed. See #2135 for a short discussion.	2019-01-06 14:52:20 +01:00
Alexander Neumann	7c084014fa	Use processed bytes for summary ... When restic reads the backup from stdin, the number of bytes processed was always displayed as zero. The reason is that the UI for the archive uses the total bytes as returned by the scanner, which is zero for stdin. So instead we keep track of the real number of bytes processed and print that at the end. Closes #2136	2019-01-06 13:15:23 +01:00
Alexander Neumann	98526b8dbe	Merge pull request 2100 from restic/fix-1989 ... gs: Respect bandwidth limiting	2018-11-28 21:39:42 +01:00
Alexander Neumann	0083680d33	Merge pull request 2099 from restic/fix-1229 ... cache: Don't return an error for truncated files	2018-11-28 21:39:35 +01:00
plumbeo	71891b340c	Support time ranges expressed in hours in snapshot retention policies ... Make restic forget --keep-within accept time ranges measured in hours and choose accordingly which snapshots to keep and which to forget. Add relative tests.	2018-11-26 14:27:42 +01:00
Alexander Neumann	c9745cd47e	gs: Respect bandwidth limiting ... In 0dfdc11ed9, accidentally we dropped using the provided http.RoundTripper, this commits adds it back. Closes #1989	2018-11-25 18:52:32 +01:00
Chris Howie	1688713400	Add key hinting (#2097)	2018-11-25 09:13:18 -05:00
Alexander Neumann	00597284de	cache: Don't return an error for truncated files ... Closes #1229 Closes #1328	2018-11-25 14:18:02 +01:00
plumbeo	7486bfea5b	Extend custom Duration granularity to hours and add tests	2018-11-14 17:27:02 +01:00
Alexander Neumann	5b95bb7059	Fix error message on Windows	2018-11-11 14:06:32 +01:00
Alexander Neumann	bfa18ee8ec	DownloadAndHash: Check error returned by Load()	2018-10-28 21:28:56 +01:00
Alexander Neumann	0b600d6cef	index.New: Test various error conditions	2018-10-28 14:02:31 +01:00
Alexander Neumann	3ae2a79bdf	restic: Use local rand source instead of global one	2018-10-28 14:01:30 +01:00
Alexander Neumann	f7c0893f76	index: Add tests for error conditions	2018-10-28 13:16:36 +01:00
Alexander Neumann	c3de301fc8	index: Use interfaces instead of *repository.Repository	2018-10-28 12:58:34 +01:00
Alexander Neumann	944b446ac0	index: Cancel context for tests	2018-10-28 11:17:04 +01:00
Alexander Neumann	b096fc7abf	index: Correctly process errors listing all files ... This also removes the now unused `list` and `worker` packages.	2018-10-28 11:16:29 +01:00
Alexander Neumann	7ac683c360	rclone: Inject debug logger for HTTP	2018-10-21 19:58:40 +02:00
Gábor Lipták	e5d7879622	Correct ineffassign ... Signed-off-by: Gábor Lipták <gliptak@gmail.com>	2018-10-19 16:58:14 -04:00
Alexander Neumann	c0fca3f50a	restore: Close files tested with --verify ... Before, the target files were opened once per blob and never closed, this commit fixes that.	2018-10-14 21:00:14 +02:00
Alexander Neumann	4c2072d875	Merge pull request 1719 from ifedorenko/mt-restorer ... new optimized multithreaded restore implementation	2018-10-14 20:29:16 +02:00
Igor Fedorenko	bda8d7722e	restorer: Optimize empty file restore ... don't create fileInfo structs for empty files. this saves memory. this also avoids extra serial scan of all fileInfo, which should make restore faster and more consistent. Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>	2018-10-14 17:39:42 +02:00
Igor Fedorenko	c2bcb764cd	restorer: Add a note on hardlink metadata ... Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>	2018-10-14 17:39:42 +02:00
Igor Fedorenko	9e24154ec9	restore: significantly reduce memory footprint ... reworked restore error callback to use file location path instead of much heavier Node. this reduced restore memory usage by as much as 50% in some of my tests. Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>	2018-10-14 17:39:42 +02:00
Igor Fedorenko	9f3ca97ee8	restore: Chang fileInfo to use snapshot location instead of target path ... * uses less memory as common prefix is only stored once * stepping stone for simpler error callback api, which will allow further memory footprint reduction Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>	2018-10-14 17:39:42 +02:00
Igor Fedorenko	32d5ceba87	restore: Fix packcache capacity math with failed downloads ... Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>	2018-10-14 17:39:42 +02:00
Igor Fedorenko	e010f3b884	restore: Fix small memory leak in filesWriter, add tests ... Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>	2018-10-14 17:39:42 +02:00
Alexander Neumann	941202c119	restore: Rename things, match the rest of the project	2018-10-14 17:39:37 +02:00
Alexander Neumann	c021ad2334	restore: Move documentation to godoc	2018-10-14 17:39:35 +02:00
Igor Fedorenko	da57302fca	restore: Removed legacy restore implementation ... Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>	2018-10-14 17:37:30 +02:00
Igor Fedorenko	1869930d95	restore: New optimized multithreaded implementation ... Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>	2018-10-14 17:37:30 +02:00
Igor Fedorenko	1213d8fef4	Cleanup: more realistic restorer test data setup ... Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>	2018-10-14 17:37:30 +02:00
Alexander Neumann	a432b42c81	self-update: Check current version before download	2018-10-14 17:29:46 +02:00
Alexander Neumann	0cb241b7d3	Remove build tag release	2018-10-11 19:40:38 +02:00
Toby Burress	8ceda538ef	b2: simplify object iteration ... Blazer is moving to a simpler object list interface, so I'm changing this here as well.	2018-10-05 11:39:02 -07:00
Alexander Neumann	5971650f77	cache: Fix recheck logic, remove channel from inProgress	2018-10-04 17:09:43 +02:00
Alexander Neumann	19725954ee	cache: Recheck before downloading	2018-10-04 14:31:18 +02:00
Alexander Neumann	fb31d66951	backup: Improve error message for invalid pattern	2018-10-03 11:12:51 +02:00
Igor Fedorenko	541d232f1c	termstatus: detect and respect dumb terminals on Unix ... Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>	2018-09-21 07:24:41 -04:00
Jakob Unterwurzacher	e42d2d1da8	restore: suppress lchown errors when not running as root ... Like "cp -a" and "rsync -a" do, only report lchown errors if we run as root. Like cp from GNU coreutils does, we check Geteuid() to determine if we are running as root ( http://git.savannah.gnu.org/cgit/coreutils.git/tree/src/copy.c#n3012 ). On Windows, lchown errors are always reported. Fixes https://github.com/restic/restic/issues/1766	2018-09-09 15:39:10 +02:00
Alexander Neumann	1140950d7b	scanner: Use context only for cancellation ... When the scanner is slower than the actual backup, the tomb cancels the context passed to Scan(), which then returns ctx.Err(). In the end, the main function prints an error message that is not helpful ("Context cancelled") and exits with an error code although no error occurred. The code now ignores the error in the context and just uses it for cancellation. The scanner is not supposed to return an error anyway. Closes #1978	2018-09-08 18:53:12 +02:00
George Armhold	bfc1bc6ee6	clean up some errors from 'go vet ./...'	2018-09-05 08:04:55 -04:00