Commit graph

5930 commits

Author SHA1 Message Date
Michael Eischer
53656f019a filter: address review comments 2022-03-20 13:33:08 +01:00
Michael Eischer
cd190bee14 filter: short circuit if no negative patterns 2022-03-20 13:33:08 +01:00
Vincent Bernat
2ee07ded2b filter: ability to use negative patterns
This is quite similar to gitignore. If a pattern is suffixed by an
exclamation mark and match a file that was previously matched by a
regular pattern, the match is cancelled. Notably, this can be used
with `--exclude-file` to cancel the exclusion of some files.

Like for gitignore, once a directory is excluded, it is not possible
to include files inside the directory. For example, a user wanting to
only keep `*.c` in some directory should not use:

    ~/work
    !~/work/*.c

But:

    ~/work/*
    !~/work/*.c

I didn't write documentation or changelog entry. I would like to get
feedback if this is the right approach for excluding/including files
at will for backups. I use something like this as an exclude file to
backup my home:

    $HOME/**/*
    !$HOME/Documents
    !$HOME/code
    !$HOME/.emacs.d
    !$HOME/games
    # [...]
    node_modules
    *~
    *.o
    *.lo
    *.pyc
    # [...]
    $HOME/code/linux/*
    !$HOME/code/linux/.git
    # [...]

There are some limitations for this change:

 - Patterns are not mixed accross methods: patterns from file are
   handled first and if a file is excluded with this method, it's not
   possible to reinclude it with `--exclude !something`.

 - Patterns starting with `!` are now interpreted as a negative
   pattern. I don't think anyone was relying on that.

 - The whole list of patterns is walked for each match. We may
   optimize later by exiting early if we know no pattern is starting
   with `!`.

Fix #233
2022-03-20 13:33:08 +01:00
Michael Eischer
12606b575f filter: Cleanup variable naming 2022-03-20 13:33:08 +01:00
Michael Eischer
5f145f0c7e filter: introduce pattern struct 2022-03-20 13:33:08 +01:00
Vincent Bernat
13c40d4199 filter: additional tests for filter.List() 2022-03-20 13:33:08 +01:00
Alexander Neumann
13aae82635
Merge pull request #3673 from restic/update-go
Update go
2022-03-20 12:02:32 +01:00
Alexander Neumann
b85d035956 Fix calens install step 2022-03-20 11:36:45 +01:00
Alexander Neumann
47aa4613f7 Force Go to use Module Mode 2022-03-20 11:30:01 +01:00
Alexander Neumann
a9a5acb8ce Update golangci-lint 2022-03-20 11:26:56 +01:00
Alexander Neumann
6dee59b789 Install gox before checking out code
Otherwise newer Go versions complain that the hash for the installed
version of gox is not in the go.mod, which we don't want anyways because
the tests should use the latest version of gox.
2022-03-20 11:26:56 +01:00
Alexander Neumann
2e19d19216 Use latest Go version for cross-compile and lint 2022-03-20 11:26:56 +01:00
Alexander Neumann
18a1de0de1 Use "go get" or "go install" selectively
Go 1.18 dropped support for installing binaries via "go get", Go <= 1.16
does not support it. So we need to use the right verb depending on the
Go version.
2022-03-20 11:26:56 +01:00
Alexander Neumann
9b57fcc6b0 Fix build.go, minimum Go version is 1.14 2022-03-20 10:54:33 +01:00
Alexander Neumann
17878036d8 Update tests to Go 1.18 2022-03-20 10:54:24 +01:00
Jason Lenz
2b1932a258 Report symlink sizes from FUSE mount for snapshot dir
Fixes #3667.
2022-03-17 22:21:47 -05:00
greatroar
fdc738fb70 Report symlink sizes from FUSE mount
Fixes #3667.
2022-03-13 16:48:35 +01:00
MichaelEischer
daea461f15
Merge pull request #3663 from jimt/find-msgs
Remove period from find messages
2022-03-07 22:23:49 +01:00
Jim Tittsler
a3d99217a4 Remove period from find messages
Simplifies cut-and-paste of IDs (and makes stylistically
consistent with other messages) #3659
2022-03-07 11:16:04 +09:00
MichaelEischer
e0ab689ccd
Merge pull request #3664 from DanielG/fix-doc-warning
doc: Fix block quote warning
2022-03-06 21:56:26 +01:00
Michael Eischer
7af69fd7b9 list: Never lock the repository when listing lock files
There's no point in locking the repository just to list the currently
existing lock files. This won't work for an exclusively locked
repository and is also confusing to users.
2022-03-06 21:44:51 +01:00
Daniel Gröber
49b67c8aaa doc: Fix block quote warning 2022-03-06 18:15:55 +01:00
rawtaz
44d543ede3
Merge pull request #3653 from MichaelEischer/fix-ls-option-help
ls: Fix description for --host, --tag and --path options
2022-02-19 23:06:24 +01:00
Michael Eischer
5ef4ee7760 ls: Fix description for --host, --tag and --path options 2022-02-19 22:36:02 +01:00
Michael Eischer
254c8743fc Limit number of large tree blobs loaded in parallel by StreamTrees
Load tree blobs with more than 50MB only from a single goroutine. Very
large tree blobs with for example 400 MB size can otherwise require
roughly 1GB * streamTreeParallelism memory.
2022-02-19 12:26:09 +01:00
MichaelEischer
ad4f4dbc7a
Merge pull request #3645 from greatroar/stdin-parent
Don't set a parent for --stdin backups
2022-02-19 11:36:51 +01:00
greatroar
63f6a9b085 Don't set a parent for --stdin backups
Loading any parent tree for these only wastes time and memory.
Fixes #3641, where it was shown that the most recent tree will get
picked.

--parent is now implicitly ignored when --stdin is given.
2022-02-19 10:41:33 +01:00
MichaelEischer
4a2d5a146d
Merge pull request #3507 from ahmgithubahm/document-AWS_PROFILE-support
Document AWS_PROFILE support
2022-02-18 23:40:25 +01:00
Michael Eischer
1efc26899d Update docs for AWS_PROFILE and AWS_SHARED_CREDENTIALS_FILE 2022-02-18 23:31:10 +01:00
MichaelEischer
8df246d0f3
Merge pull request #3628 from gum3ng/issue_3127
[#issue 3127] Add xattr support for Solaris
2022-02-17 22:07:39 +01:00
Michael Eischer
27524979e8 restorer: Remove dead code 2022-02-13 11:43:09 +01:00
gum3ng
dd30083c2b [#issue 3127] Add xattr support for Solaris 2022-02-13 14:24:37 +05:30
MichaelEischer
fb4c5af5c4
Merge pull request #3642 from gco/master
Fix test failures on Solaris
2022-02-12 22:07:14 +01:00
MichaelEischer
18ec49ddfa
Merge pull request #3644 from duritong/centos-epel-repo
add a note about installation via epel for RHEL / CentOS Stream 8 & 9
2022-02-12 21:41:46 +01:00
Michael Eischer
5ec312ca06 sftp: Implement atomic uploads
Create a temporary file with a sufficiently random name to essentially
avoid any chance of conflicts. Once the upload has finished remove the
temporary suffix. Interrupted upload thus will be ignored by restic.
2022-02-12 20:30:49 +01:00
Michael Eischer
aebd24e414 Add changelog 2022-02-12 20:21:58 +01:00
Michael Eischer
d72181c8c1 Ensure that the lock cleanup handler is run after the global one
cleanup handlers run in the order in which they are added. As Go calls
init() functions in lexical order, the cleanup handler from global.go
was registered before that from lock.go, which is the correct order.

Make this order explicit to ensure that this won't break accidentally.
2022-02-12 20:21:58 +01:00
Michael Eischer
c6fd13425b remember the refreshed lock file even if removal failed
This ensures that restic won't create lots of new lock files without
deleting them later on.

In some cases a Delete operation on a backend can return a "File does
not exist" error even though the Delete operation succeeded. This can
for example be caused by request retries. This caused restic to forget
about the new lock file and continue trying to remove the old (already
deleted) lock file.
2022-02-12 20:21:58 +01:00
Michael Eischer
bba8ba7a5b repository: cancel streampack context after error 2022-02-12 20:18:25 +01:00
Michael Eischer
47554a3428 repository: Fix error handling in repack
When storing a blob fails, this is a fatal error which must not be
retried.
2022-02-12 20:18:25 +01:00
Michael Eischer
4b3dc415ef checker: cleanup header extraction 2022-02-12 20:18:25 +01:00
Michael Eischer
930a00ad54 checker: reuse bufio reader 2022-02-12 20:18:25 +01:00
Michael Eischer
34ebafb8b6 repository: don't crash if blob size is too short 2022-02-12 20:18:25 +01:00
Michael Eischer
becebf5d88 repository: remove unused DownloadAndHash 2022-02-12 20:18:25 +01:00
Michael Eischer
f1e58e7c7f checker: rewrite ReadData to stream packs 2022-02-12 20:18:25 +01:00
Michael Eischer
f40abd92fa restorer: convert to use StreamPack 2022-02-12 20:18:25 +01:00
Michael Eischer
f00f690658 repository: stream packs during repacking 2022-02-12 20:18:25 +01:00
Michael Eischer
c4a2bfcb39 repository: Add StreamPacks function
The function supports efficiently loading a specified list of blobs from
a single pack in a streaming fashion. That is there's no need for
temporary files independent of the pack size.
2022-02-12 20:18:25 +01:00
Michael Eischer
153e2ba859 repository: Implement lisiting blobs per pack file 2022-02-12 20:18:24 +01:00
MichaelEischer
cc90f2ba6b
Merge pull request #2816 from greatroar/noatime
Set O_NOATIME flag on Linux
2022-02-07 21:38:31 +01:00