Commit graph

1048 commits

Author SHA1 Message Date
Ryan Hitchman
77bf148460 backup: add --dry-run/-n flag to show what would happen.
This can be used to check how large a backup is or validate exclusions.
It does not actually write any data to the underlying backend. This is
implemented as a simple overlay backend that accepts writes without
forwarding them, passes through reads, and generally does the minimal
necessary to pretend that progress is actually happening.

Fixes #1542

Example usage:

$ restic -vv --dry-run . | grep add
new       /changelog/unreleased/issue-1542, saved in 0.000s (350 B added)
modified  /cmd/restic/cmd_backup.go, saved in 0.000s (16.543 KiB added)
modified  /cmd/restic/global.go, saved in 0.000s (0 B added)
new       /internal/backend/dry/dry_backend_test.go, saved in 0.000s (3.866 KiB added)
new       /internal/backend/dry/dry_backend.go, saved in 0.000s (3.744 KiB added)
modified  /internal/backend/test/tests.go, saved in 0.000s (0 B added)
modified  /internal/repository/repository.go, saved in 0.000s (20.707 KiB added)
modified  /internal/ui/backup.go, saved in 0.000s (9.110 KiB added)
modified  /internal/ui/jsonstatus/status.go, saved in 0.001s (11.055 KiB added)
modified  /restic, saved in 0.131s (25.542 MiB added)
Would add to the repo: 25.892 MiB
2021-08-04 21:19:29 +02:00
MichaelEischer
533ac4fd95
Merge pull request #3467 from greatroar/wrappedconn-pointer
Use rclone.wrappedConn by pointer
2021-08-04 21:18:53 +02:00
David le Blanc
326fefcd80 Allow --tag and --keep-tag to match untagged snapshots 2021-08-02 23:06:20 +02:00
greatroar
fa3eed1998 Use rclone.wrappedConn by pointer
This shaves a kilobyte off the Linux binary by not generating a
non-pointer interface implementation.
2021-08-01 09:11:50 +02:00
MichaelEischer
5571c3f7fd
Merge pull request #3453 from MichaelEischer/http2-zero-length-workaround
rest: Workaround Http2 zero-length reply bug
2021-07-31 20:30:06 +02:00
Alexander Neumann
d8ea10db8c rest: Rework handling HTTP2 zero-length replies bug
Add comment that the check is based on the stdlib HTTP2 client. Refactor
the checks into a function. Return an error if the value in the
Content-Length header cannot be parsed.
2021-07-31 17:12:24 +02:00
Magnus Thor Torfason
2081bd12fb forget: Ensure future snapshots do not affect --keep-within-*
Ensure that only snapshots made in the past are taken into account when running restic forget with the within switches (--keep-within, --keep-within- hourly, and friends)
2021-07-24 16:14:43 +00:00
Magnus Thor Torfason
74ebc650ab forget: Add --keep-within-hourly (and friends)
Allow keeping hourly/daily/weekly/monthly/yearly snapshots for a given time period.

This adds the following flags/parameters to restic forget:
  --keep-within-hourly duration
  --keep-within-daily duration
  --keep-within-weekly duration
  --keep-within-monthly duration
  --keep-within-yearly duration

Includes following changes:
  - Add tests for --keep-within-hourly (and friends)
  - Add documentation for --keep-within-hourly (and friends)
  - Add changelog for --keep-within-hourly (and friends)
2021-07-24 16:14:43 +00:00
Michael Eischer
097ed659b2 rest: test that zero-length replies over HTTP2 work correctly
The first test function ensures that the workaround works as expected.
And the second test function is intended to fail as soon as the issue
has been fixed in golang to allow us to eventually remove the
workaround.
2021-07-10 17:22:42 +02:00
Michael Eischer
185a55026b rest: workaround for HTTP2 zero-length replies bug
The golang http client does not return an error when a HTTP2 reply
includes a non-zero content length but does not return any data at all.
This scenario can occur e.g. when using rclone when a file stored in a
backend seems to be accessible but then fails to download.
2021-07-10 16:59:01 +02:00
Michael Eischer
bd316d3893 restore: Test partial pack downloads in filerestorer 2021-06-29 21:11:30 +02:00
Michael Eischer
e8bbb05328 restore: Correctly handle partial pack download errors
Failed pack/blob downloads should be retried. For blobs that fail
decryption assume that the pack file is really damaged and try to
restore the remaining blobs.
2021-06-29 20:54:16 +02:00
MichaelEischer
c1eb7ac1a1
Merge pull request #3420 from greatroar/local-errs
Modernize error handling in local backend
2021-06-20 14:20:40 +02:00
greatroar
e5f0f67ba0 Modernize error handling in local backend
* Stop prepending the operation name: it's already part of os.PathError,
  leading to repetitive errors like "Chmod: chmod /foo/bar: operation not
  permitted".

* Use errors.Is to check for specific errors.
2021-06-18 11:13:27 +02:00
MichaelEischer
a476752962
Merge pull request #3421 from greatroar/s3-fileinfo
Return s3.fileInfos by pointer
2021-06-12 18:55:18 +02:00
MichaelEischer
e8d20ea32c
Merge pull request #3409 from greatroar/lchown-mknod
Make restic.{lchown,mknod} regular functions
2021-06-12 18:22:38 +02:00
greatroar
0d4f16b6ba Return s3.fileInfos by pointer
Since the fileInfos are returned in a []interface, they're already
allocated on the heap. Making them pointers explicitly means the
compiler doesn't need to generate fileInfo and *fileInfo versions of the
methods on this type. The binary becomes about 7KiB smaller on
Linux/amd64.
2021-06-07 19:48:43 +02:00
greatroar
0666c4d244 Make restic.{lchown,mknod} regular functions
This way, they can be inlined and dead code can be removed on Windows.
Also fixed some comments.
2021-05-27 22:51:40 +02:00
rawtaz
fdbd65485e
Merge pull request #3402 from MichaelEischer/misc-fixes
Various small code cleanups
2021-05-24 11:30:31 +02:00
Alexander Neumann
4cabad8c34
Merge pull request #3325 from MichaelEischer/fix-mintty-output
Fix windows terminal output for mintty
2021-05-18 09:29:24 +02:00
Michael Eischer
cf92c58460 Properly wrap errors in readerat helper 2021-05-17 21:08:23 +02:00
Michael Eischer
75c990504d azure/gs: Fix default value in connections help text 2021-05-17 20:56:51 +02:00
Michael Eischer
55bea6e7a6 filter: Fix crash for '**' pattern 2021-05-14 23:50:31 +02:00
MichaelEischer
6d8ceefd67
Merge pull request #3373 from greatroar/simplify-limiter
Simplify internal/limiter
2021-05-14 20:27:31 +02:00
MichaelEischer
64b00d28b1
Merge pull request #3345 from greatroar/sftp-enospc
Check for ENOSPC and remove broken files in SFTP
2021-05-13 20:09:38 +02:00
greatroar
ae170e2b38 Simplify internal/limiter 2021-04-24 11:54:43 +02:00
Michael Eischer
7cb8ea69ba Add test to mintty pipe detection 2021-04-11 20:02:09 +02:00
Michael Eischer
80564a9bc9 Properly detect mintty output redirection
mintty on windows always uses pipes to connect stdout between processes
and for the terminal output. The previous implementation always assumed
that stdout connected to a pipe means that stdout is displayed on a
mintty terminal. However, this detection breaks when using pipes to
connect processes and for powershell which uses pipes when redirecting
to a file.

Now the pipe filename is queried and matched against the pattern used by
msys / cygwin when connected to the terminal. In all other cases assume
that a pipe is just a regular pipe.
2021-04-11 20:02:09 +02:00
Michael Eischer
5e6af77b7a Unify interactive terminal detection code
Previously the progress bar / status update interval used
stdoutIsTerminal to determine whether it is possible to update the
progress bar or not. However, its implementation differed from the
detection within the backup command which included additional checks to
detect the presence of mintty on Windows. mintty behaves like a terminal
but uses pipes for communication.

This adds stdoutCanUpdateStatus() which calls the same terminal detection
code used by backup. This ensures that all commands consistently switch
between interactive and non-interactive terminal mode.

stdoutIsTerminal() now also returns true whenever stdoutCanUpdateStatus()
does so. This is required to properly handle the special case of mintty.
2021-04-11 20:02:09 +02:00
MichaelEischer
cc254dfefe
Merge pull request #3362 from greatroar/darwin-preallocate
Use FcntlFstore to preallocate on Mac
2021-04-10 22:47:56 +02:00
greatroar
23531be272 Use FcntlFstore to preallocate on Mac 2021-04-10 16:54:07 +02:00
rawtaz
74c0607c92
Merge pull request #3319 from MichaelEischer/skip-prealloc-test
restorer: Skip preallocate test if not supported by the filesystem
2021-04-07 18:59:06 +02:00
greatroar
dc88ca79b6 Handle lack of space and remove broken files in SFTP backend 2021-03-27 15:02:14 +01:00
Alexander Neumann
88a23521dd
Merge pull request #3327 from MichaelEischer/fix-s3-sanity-check
s3: Fix sanity check
2021-03-11 13:13:46 +01:00
greatroar
187a77fb27 Upgrade pkg/sftp to 1.13 and simplify SFTP.IsNotExist 2021-03-10 21:05:24 +01:00
Michael Eischer
2a9f0f19b6 s3: Fix sanity check
The sanity check shouldn't replace the error message if there is already
one.
2021-03-08 20:23:57 +01:00
Michael Eischer
f70aca6f6f restorer: Skip preallocate test if not supported by the filesystem 2021-03-04 20:33:46 +01:00
Michael Eischer
a293fd9aef local: Ignore files in intermediate folders
For example the data folder of a repository might contain hidden files
which caused list operations to fail.
2021-02-27 13:47:55 +01:00
Alexander Neumann
9985368d46
Merge pull request #3255 from MichaelEischer/restorer-check-error
restorer: Check dropped error
2021-02-03 16:10:46 +01:00
Alexander Neumann
2dd592a06c
Merge pull request #3254 from lorenz/allow-http2
Allow HTTP/2
2021-02-02 21:08:20 +01:00
Michael Eischer
47faf69230 restorer: Check dropped error 2021-01-31 18:06:28 +01:00
Alexander Weiss
5e22ae10f1 Add error handling for fileRestorer 2021-01-31 14:22:57 +01:00
Alexander Weiss
573221aa40 Add failing test for fileRestorer 2021-01-31 13:40:42 +01:00
Lorenz Brun
5427119205 Allow HTTP/2 2021-01-31 02:44:30 +01:00
Michael Eischer
e0867c9682 backend: try to cleanup test leftovers 2021-01-30 21:23:20 +01:00
Michael Eischer
f740b2fb23 mem: check upload length before storing upload 2021-01-30 21:23:20 +01:00
Alexander Neumann
0e5f2fff71
Merge pull request #3243 from restic/fix-scanner-overlap
backup: Fix total size for overlapping targets
2021-01-30 21:17:21 +01:00
Alexander Neumann
04ca69cc78 Address issues reported by golint 2021-01-30 20:45:57 +01:00
Alexander Neumann
f867e65bcd Fix issues reported by staticcheck 2021-01-30 20:43:53 +01:00
Alexander Neumann
0858fbf6aa Add more error handling 2021-01-30 20:19:47 +01:00