Commit graph

170 commits

Author SHA1 Message Date
Alexander Neumann
303a5dab6a archiver: Clarify value in test struct
Since I could not remember what the value for `Check` means this commit
renames it to `SameFile`: when set to true, the test should make sure
that `FileChanged` should return false (=file is unmodified).
2019-05-05 12:57:00 +02:00
Gábor Lipták
bce6438d22
Don't run TestMetadataChanged test on Windows
Signed-off-by: Gábor Lipták <gliptak@gmail.com>
2019-04-27 21:23:47 -04:00
Alexander Neumann
919dd2ac84 Merge pull request #2252 from restic/fix-2249
Read fresh metadata for unmodified files
2019-04-25 09:15:50 +02:00
Courtney Bane
0ebfc55ee3 Use existing setTimestamp function for ctime test and improve error checking. 2019-04-24 20:53:08 -05:00
Courtney Bane
35b7607802 Don't check ctime when ignoring inode. 2019-04-24 20:53:08 -05:00
Alexander Neumann
389067fb8b Only use list of blobs for old node
Closes #2249
2019-04-24 15:07:26 +02:00
Alexander Neumann
4b0ca9ddab Add test for #2249 2019-04-24 15:07:23 +02:00
Courtney Bane
b8c2544dcb Examine file ctime when checking if files have changed. 2019-04-23 21:54:35 -05:00
Alexander Neumann
65b476ead9 Fix gofmt 2019-03-16 13:29:05 +01:00
Heiko Bornholdt
db8f5864fc Add --ignore-inode option to backup cmd
revised version of https://github.com/restic/restic/pull/2047
2019-03-10 21:24:29 +01:00
Alexander Neumann
c4fbf2c779 Return error when reading zero byte from stdin
This commit changes the internal file system implementation for reading
data from stdin, it now returns an error when no bytes could be read. I
think it's worth failing in this case, the user instructed restic to
read some data from stdin, and no data was read at all. Maybe it was in
a pipe and some earlier stage failed.

See #2135 for a short discussion.
2019-01-06 14:52:20 +01:00
Alexander Neumann
1140950d7b scanner: Use context only for cancellation
When the scanner is slower than the actual backup, the tomb cancels the
context passed to Scan(), which then returns ctx.Err(). In the end, the
main function prints an error message that is not helpful ("Context
cancelled") and exits with an error code although no error occurred.

The code now ignores the error in the context and just uses it for
cancellation. The scanner is not supposed to return an error anyway.

Closes #1978
2018-09-08 18:53:12 +02:00
George Armhold
bfc1bc6ee6 clean up some errors from 'go vet ./...' 2018-09-05 08:04:55 -04:00
Alexander Neumann
aaff8803ef Fix formatting for gofmt with Go 1.11 2018-08-31 21:10:26 +02:00
Andreas Skielboe
b07bb3d8c3 Reject files excluded by name before calling lstat to improve scan speed
Adds a SelectByName method to the archive and scanner which only require
the filename as input, and can thus be run before calling lstat on the
file. Can speed up scanning significantly if a lot of filename excludes
are used.
2018-08-12 17:51:12 +02:00
Alexander Neumann
9ffc26883a archiver: Unroll tree 2018-06-20 22:56:41 +02:00
Alexander Neumann
c44e808aa5 backup: Fix deadlock
When the archiver is faster than the scanner, restic deadlocks. This
commit adds a `finished` channel to the struct in `ui/backup.go` so that
scanner results are ignored when the archiver is already finished.

Closes #1834
2018-06-09 12:15:19 +02:00
Alexander Neumann
adb682bc43 archiver: Don't open files with O_NONBLOCK
This is not necessary any more, we're doing an lstat() before opening
an item, so we already known it's a file and not a pipe.
2018-05-20 16:11:51 +02:00
Alexander Neumann
1e9744c9a4 archiver: Refuse to save an empty snapshot 2018-05-20 16:11:51 +02:00
Alexander Neumann
9a02f17cc2 archiver: Add tests for Save() for fs.Reader 2018-05-20 16:11:36 +02:00
Alexander Neumann
347a645450 Fix double error message 2018-05-15 11:03:33 +02:00
Alexander Neumann
60ea2435be Improve error message for readdir/readdirnames
As mentioned in the forum[1], restic does not include the dir name when
readdir/readdirnames fails.

[1] https://forum.restic.net/t/readdirnames-readdirent-no-such-file-or-directory/653
2018-05-13 10:34:50 +02:00
Alexander Neumann
e43c9202a6 archiver: Make sure backend error is passed up 2018-05-12 23:55:59 +02:00
Alexander Neumann
c5e75d1c98 archiver: Add test for early abort on unhandled error 2018-05-12 23:55:59 +02:00
Alexander Neumann
526956af35 archiver: Read files/dirs in order 2018-05-12 23:55:54 +02:00
Alexander Neumann
256104111d archiver: Clarify names 2018-05-12 23:55:54 +02:00
Alexander Neumann
21c83b1725 archiver: Add high-level documentation 2018-05-12 23:55:54 +02:00
Alexander Neumann
581c62ee72 archiver: Improve error handling
This commit changes how the worker goroutines for saving e.g. blobs
interact. Before, it was possible to get stuck sending an instruction to
archive a file or dir when no worker goroutines were available any more.
This commit introduces a `done` channel for each of the worker pools,
which is set to the channel returned by `tomb.Dying()`, so it is closed
when the first worker returned an error.
2018-05-12 23:55:54 +02:00
Alexander Neumann
ca4af43c03 archiver: Return low-level errors
This commit changes the archiver so that low-level errors saving data to
the repo are returned to the caller (instead of being handled by the
error callback function). This correctly bubbles up errors like a full
temp file system and makes restic abort early and makes all other worker
goroutines exit.
2018-05-10 21:30:09 +02:00
Alexander Neumann
1f2463f42e archiver: Return correct error 2018-05-10 20:48:00 +02:00
Alexander Neumann
fd77646f8b Fix panic for nil-pointer dereference
Closes #1755
2018-05-03 20:49:30 +02:00
Alexander Neumann
2218ecd049 archiver: Use lstat before open/fstat
The previous code tried to be as efficient as possible and only do a
single open() on an item to save, and then fstat() on the fd to find out
what the item is (file, dir, other). For normal files, it would then
start reading the data without opening the file again, so it could not
be exchanged for e.g. a symlink.

This behavior starts the watchdog on my machine when /dev is saved
with restic, and after a few seconds, the machine reboots.

This commit reverts the behavior to the strategy the old archiver code
used: run lstat(), then decide what to do. For normal files, open the
file and then run fstat() on the fd to verify it's still a normal file,
then start reading the data.

The downside is that for normal files we now do two stat() calls
(lstat+fstat) instead of only one. On the upside, this does not start
the watchdog. :)
2018-05-01 23:05:50 +02:00
Alexander Neumann
c83c03ed63 archiver: Fix blocking on pipes 2018-04-30 15:34:58 +02:00
Alexander Neumann
4e34325035 archiver: Process dirs concurrently 2018-04-30 15:13:28 +02:00
Alexander Neumann
78bd591c7c archiver: Improve buffer pool 2018-04-30 15:13:28 +02:00
Alexander Neumann
39ac12f6ea archiver: Correct comment 2018-04-30 14:19:07 +02:00
Alexander Neumann
400730afca archiver: Improve memory usage, tune buffer pool 2018-04-30 14:19:07 +02:00
Alexander Neumann
0e78ac92d8 Use new archiver code for backup 2018-04-28 22:08:11 +02:00
Alexander Neumann
38926d8576 Use new archiver code in tests 2018-04-25 14:42:45 +02:00
Alexander Neumann
f279731168 Add new archiver code 2018-04-25 14:42:45 +02:00
Alexander Neumann
fd12a3af20 Remove old archiver code 2018-04-23 21:40:33 +02:00
Alexander Neumann
4e0b2a8e3a snapshot: correct error handling for filepath.Abs 2018-04-22 11:37:05 +02:00
Alexander Neumann
a472868e06 fs: Add TestChdir() 2018-04-22 11:37:05 +02:00
Eri Bastos
901e1b129c Fixed issue #1608 - Use --time argument properly
Backups via stdin will now handle --time argument and pass it down as
expected
2018-04-03 14:40:42 -03:00
Alexander Neumann
5253ef218c Remove TestParallelSaveWithDuplication 2018-03-11 19:49:48 +01:00
Alexander Neumann
0923976909 Remove TestArchiverDuplication 2018-03-11 19:44:25 +01:00
Alexander Neumann
99f7fd74e3 backend: Improve Save()
As mentioned in issue [#1560](https://github.com/restic/restic/pull/1560#issuecomment-364689346)
this changes the signature for `backend.Save()`. It now takes a
parameter of interface type `RewindReader`, so that the backend
implementations or our `RetryBackend` middleware can reset the reader to
the beginning and then retry an upload operation.

The `RewindReader` interface also provides a `Length()` method, which is
used in the backend to get the size of the data to be saved. This
removes several ugly hacks we had to do to pull the size back out of the
`io.Reader` passed to `Save()` before. In the `s3` and `rest` backend
this is actively used.
2018-03-03 15:49:44 +01:00
Igor Fedorenko
d58ae43317 Reworked Backend.Load API to retry errors during ongoing download
Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>
2018-02-16 21:12:14 -05:00
Alexander Neumann
f99c95c766 archiver: Fix intermediate index upload
A user discovered[1] that when the backup finishes during the upload of
an intermediate index, the upload is cancelled and the index never fully
saved, but the snapshot is saved and the backup finalizes without an
error. This lead to a situation where a snapshot references data that is
contained in the repo, but not referenced in any index, leading to
strange error messages.

This commit uses a dedicated context to signal the intermediate index
uploading routine to terminate after the last index has been uploaded.
This way, an upload running when the backup finishes is completed before
the routine terminates and the snapshot is saved.

[1] https://forum.restic.net/t/error-loading-tree-check-prune-and-forget-gives-error-b2-backend/406
2018-01-26 22:01:07 +01:00
Alexander Neumann
663c57ab4d debug: Remove manual Str() call Log() 2018-01-25 20:49:41 +01:00
Alexander Neumann
9c55e8d69c Merge pull request #1549 from MJDSys/more_index_lookup_avoids
More optimizations to avoid calling Index.Lookup()
2018-01-24 20:53:30 +01:00
Matthew Dawson
3a16148447
archiver/archiver: Use Index.Has() instead of Index.Lookup() in isKnownBlob
Index.Has() is a faster then Index.Lookup() for checking if a blob exists
in the index.  As the returned data is never used, this avoids a ton
of allocations.
2018-01-23 22:26:10 -05:00
Matthew Dawson
df2c03a6a4
repository/master_index: Optimize Index.Lookup()
When looking up a blob in the master index, with several
indexes present in the master index, a significant amount of time
is spent generating errors for each failed lookup.  However, these
errors are often used to check if a blob is present, but the contents
are not inspected making the overhead of the error not useful.

Instead, change Index.Lookup (and Index.LookupSize) to instead return
a boolean denoting if the blob was found instead of an error.  Also change
all the calls to these functions to handle the new function signature.

benchmark                                            old ns/op     new ns/op     delta
BenchmarkMasterIndexLookupSingleIndex-6              820           897           +9.39%
BenchmarkMasterIndexLookupMultipleIndex-6            12821         2001          -84.39%
BenchmarkMasterIndexLookupSingleIndexUnknown-6       5378          492           -90.85%
BenchmarkMasterIndexLookupMultipleIndexUnknown-6     17026         1649          -90.31%

benchmark                                            old allocs     new allocs     delta
BenchmarkMasterIndexLookupSingleIndex-6              9              9              +0.00%
BenchmarkMasterIndexLookupMultipleIndex-6            59             19             -67.80%
BenchmarkMasterIndexLookupSingleIndexUnknown-6       22             6              -72.73%
BenchmarkMasterIndexLookupMultipleIndexUnknown-6     72             16             -77.78%

benchmark                                            old bytes     new bytes     delta
BenchmarkMasterIndexLookupSingleIndex-6              160           160           +0.00%
BenchmarkMasterIndexLookupMultipleIndex-6            3200          240           -92.50%
BenchmarkMasterIndexLookupSingleIndexUnknown-6       1232          48            -96.10%
BenchmarkMasterIndexLookupMultipleIndexUnknown-6     4272          128           -97.00%
2018-01-23 22:25:56 -05:00
Alexander Neumann
b0c6e53241 Fix calls to repo/backend.List() everywhere 2018-01-21 21:15:09 +01:00
George Armhold
d886cb5c27 replace ad-hoc context.TODO() with gopts.ctx, so that cancellation
can properly trickle down from cmd_*.

gh-1434
2017-12-03 07:22:14 -05:00
Alexander Neumann
eddb8549ef backup: By default, do not save the access time
This can be re-enabled with `--with-atime`.
2017-11-28 21:31:35 +01:00
Alexander Neumann
931e6ed2ac Use Seal/Open everywhere 2017-11-01 10:30:40 +01:00
Herbert
3473c3f7b6 Remove all dot-imports 2017-10-02 15:06:39 +02:00
Alexander Neumann
ce180de9b8 Merge pull request #1243 from restic/improve-error-reporting
Improve error reporting
2017-09-16 14:54:30 +02:00
Alexander Neumann
d4e994de7b Improve error reporting
This will print the error (including a stack trace) if available before
exiting.
2017-09-16 10:55:13 +02:00
Alexander Neumann
a60e751217 Use .Equal() instead of == for time.Time
Closes #1238
2017-09-15 20:57:35 +02:00
Tobias Klein
f26c0cb70f testcase updated 2017-09-09 15:33:12 +02:00
Tobias Klein
087c3fe1dc tests updated 2017-09-09 13:26:35 +02:00
Tobias Klein
43ff971dfd new sub-option for backup: time
New option to specify the timestamp for a backup
2017-09-09 13:26:35 +02:00
Alexander Neumann
83eb075e3a Resolve name collisions
At the moment when two items to be saved have the same directory name,
restic only saves the first one to the repo. Let's say we have a
structure like this:

    dir1
    └── subdir
        └── file
    dir2
    └── subdir
        └── file

When restic is run on `dir1/subdir` and `dir2/subdir`, it will only save
the first `subdir`:

    $ restic backup dir1/subdir dir2/subdir
    [...]

    $ restic ls -l latest
    drwxr-xr-x  1000   100      0 2017-08-27 20:56:39 /subdir
    -rw-r--r--  1000   100     17 2017-08-27 20:56:39 /subdir/file

That's obviously a bad thing, caused by an early decision to strip the
full path to the files/dirs to save and only leave the last directory.

This commit partly resolves this by handling colliding names and
resolving the conflicts. Restic will now append a counter to the file
(`-123`) until the conflict is resolved. So in the example above, we'll
end up with the following structure:

    $ restic ls -l latest
    drwxr-xr-x  1000   100      0 2017-08-27 20:56:39 /subdir
    -rw-r--r--  1000   100     17 2017-08-27 20:56:39 /subdir/file
    drwxr-xr-x  1000   100      0 2017-08-27 20:56:46 /subdir-1
    -rw-r--r--  1000   100     17 2017-08-27 20:56:46 /subdir-1/file

This partly addresses #549 and closes #1179.

At first I thought that the obvious correction would be to archive the
full path. But it turns out that collisions may still occur: Suppose you
have a file named `foo` in the current directory, and the parent directory
also contains a file `foo`. Archiving these with restic also causes a
collision, since restic strips the `../` from the first file:

    $ restic backup ../foo foo

This also happens with `tar`, which does not handle the collision and
will happily archive two files called `foo`.

So, the best way forward is to handle name collisions and archive the
whole path. The latter will be tackled in a separate PR.
2017-09-05 21:47:02 +02:00
Alexander Neumann
2444522243 Add test for colliding names 2017-09-05 21:10:02 +02:00
Emil Hessman
c2ff7150aa internal: check error before deferring file Close()
If there is an error, file will be `nil`. We should check the returned error before deferring file `Close()`.
2017-08-13 19:28:13 +02:00
Alexander Neumann
23c903074c Move restic package to internal/restic 2017-07-24 17:43:32 +02:00
Alexander Neumann
6caeff2408 Run goimports 2017-07-23 14:21:03 +02:00
Alexander Neumann
83d1a46526 Moves files 2017-07-23 14:19:13 +02:00