alexvanin/restic
1
0
Fork 0
forked from TrueCloudLab/restic
Code Pull requests Activity
8687 commits 8 branches 49 tags 79 MiB
Commit graph

541 commits

Author SHA1 Message Date
Michael Eischer
a60ee9b764 retry: limit retries based on elapsed time not count 
Depending on how long an operation takes to fail, the total retry
duration can currently vary between 1.5 and 15 minutes. In particular
for temporarily interrupted network connections, the former timeout is
too short. Thus always use a limit of 15 minutes.
 2024-05-24 20:24:02 +02:00
Michael Eischer
a3633cad9e retry: explicitly log failed requests 
This simplifies finding the request in the log output that cause an
operation to fail.
 2024-05-24 20:24:02 +02:00
Michael Eischer
c56ecec9aa azure: deduplicate cli and default credentials case 2024-05-18 22:15:54 +02:00
Maik Riechert
355f520936 Azure: add option to force use of CLI credential 2024-05-18 22:15:54 +02:00
Michael Eischer
0c1ba6d95d backend: remove unused Location method 2024-05-18 21:38:31 +02:00
Michael Eischer
1d6d3656b0 repository: move backend.LoadAll to repository.LoadRaw 
LoadRaw also includes improved context cancellation handling similar to the
implementation in repository.LoadUnpacked.

The removed cache backend test will be added again later on.
 2024-05-18 21:26:00 +02:00
Michael Eischer
47232bf8b0 backend: move LimitReadCloser to util package 
The helper is only intended for usage by backend implementations.
 2024-05-18 21:26:00 +02:00
Michael Eischer
53d15bcd1b retry: add circuit breaker to load method 
If a file exhausts its retry attempts, then it is likely not accessible
the next time. Thus, immediately fail all load calls for that file to
avoid useless retries.
 2024-05-18 19:59:26 +02:00
Michael Eischer
394c8ca3ed rest/rclone/s3/sftp/swift: move short file detection behind feature gate 
These backends tend to use a large variety of server implementations.
Some of those implementations might prove problematic with the new
checks.
 2024-05-18 19:59:26 +02:00
Michael Eischer
aeb7eb245c retry: do not retry permanent errors 
This is currently gated behind a feature flag as some unexpected
interactions might show up in the wild.
 2024-05-18 19:59:26 +02:00
Michael Eischer
bf8cc59889 Use generic backend-error-redesign feature flag instead of http-timeouts 
An individual flag for each change of the backend error handling would
be too finegrained. Thus, add a generic flag.
 2024-05-18 19:54:52 +02:00
Michael Eischer
4740528a0b backend: add tests for IsPermanentError 2024-05-18 19:54:52 +02:00
Michael Eischer
6a85df7297 backend: add IsPermanentError() method to interface 2024-05-18 19:54:52 +02:00
Michael Eischer
cfc420664a mem: stricter handling of out of bounds requests 2024-05-18 19:54:52 +02:00
Michael Eischer
d40f23e716 azure/b2/gs/s3/swift: adapt cloud backend 2024-05-18 19:54:51 +02:00
Michael Eischer
e793c002ec local: stricter handling of short files 2024-05-18 19:54:21 +02:00
Michael Eischer
b4895ebd76 rest: rework error reporting and report too short files 2024-05-18 19:54:21 +02:00
Michael Eischer
eaa3f81d6b sftp: check for truncated files without an extra backend request 2024-05-18 19:54:21 +02:00
Michael Eischer
c6d74458ee sftp: improve handling of too short files 2024-05-18 19:54:21 +02:00
Michael Eischer
b5fdb1d637
Merge pull request #4782 from MichaelEischer/fix-sftp-performance 
Fix sftp upload performance
 2024-05-12 15:28:33 +02:00
Michael Eischer
ebd01a4675 backend: add tests for watchdogRoundTripper 2024-05-09 23:46:17 +02:00
Michael Eischer
8778670232 backend: cancel stuck http requests 
requests that make no upload or download progress within a timeout are
canceled.
 2024-05-09 23:46:17 +02:00
Michael Eischer
0987c731ec backend: configure protocol-level connection health checks 
This should detect a connection that is stuck for more than 2 minutes.
 2024-05-09 23:46:17 +02:00
Michael Eischer
935327d480 sftp: slightly increase write concurrency 
This should increase upload throughput for high latency links a bit.
 2024-04-28 11:50:09 +02:00
Michael Eischer
669a669603 sftp: Fix upload performance issue 
Since pkg/sftp 1.13.0 files were uploaded sequentially using 32kb chunks
instead of sending 64 chunks in parallel.
 2024-04-28 11:48:26 +02:00
Michael Eischer
b15d867414
Merge pull request #4763 from MichaelEischer/refactor-prune 
Refactor repair index / prune into the repository package
 2024-04-22 22:24:53 +02:00
Michael Eischer
35277b7797 backend/mem: cleanup not found error message 2024-04-14 13:45:06 +02:00
Michael Eischer
55d56db31b
Merge pull request #4743 from MichaelEischer/deprecate-s3legacy-layout 
Deprecate s3legacy layout
 2024-04-11 22:09:34 +02:00
Michael Eischer
591b421c4a Deprecate s3legacy layout 2024-04-10 21:27:56 +02:00
Adam Eijdenberg
6e775d3787 Enhancement: option to send HTTP over unix socket 
add tests for unix socket connection

switch HTTP rest-server test to use any free port

allow rest-server test graceful shutdown opportunity
 2024-03-28 17:41:41 +01:00
Alexander Neumann
c0514dd8ba Fix linter errors (except for tests) 2024-02-10 22:58:10 +01:00
Michael Eischer
a763a5c67d s3: minor cleanups for archive storage class handling 2024-01-20 11:25:28 +01:00
Vladislav Belous
8ca58b487c S3: do not set storage class for metadata when using archive storage 2024-01-20 11:04:15 +01:00
Michael Eischer
bd883caae1 CI: enable bodyclose linter 2024-01-19 21:17:18 +01:00
Michael Eischer
b1a8fd1d03 rest: fix and cleanup closing of http response body 
If client.Do returns an error, then there's no body that has to be
closed. For requests for which we are not interested in the response
body, immediately drain and close the body to make sure it isn't
forgotten later on.

This change in particular adds the missing `Close()` call for the
`List()` command.
 2024-01-19 21:17:17 +01:00
Michael Eischer
7b2de84763
Merge pull request #4618 from MichaelEischer/workaround-rclone-list-errors 
rclone: Workaround for incorrect "not found" errors while listing files
 2024-01-09 18:28:31 +01:00
Michael Eischer
2e8de9edfd rclone: Workaround for incorrect "not found" errors while listing files 
rclone returns a "not found" error if an internal error occurs while
listing a folder. Ignoring this error lets restic erroneously think that
there are no files, which can cause `prune` to wipe the whole
repository.
 2024-01-09 18:20:16 +01:00
Michael Eischer
02bc73f5eb s3: minor code cleanups 2024-01-06 21:44:53 +01:00
Michael Eischer
20cf4777cb s3: check for EnvAWS credentials before Static credentials 
EnvAWS considers more environment variables, including AWS_SESSION_TOKEN
and thus should be checked first.
 2024-01-06 21:43:47 +01:00
Erik Kristensen
5ffb536aae feat: support AWS assume role 2024-01-06 21:19:58 +01:00
Andrea Gelmini
241916d55b
Fix typos 2023-12-06 13:11:55 +01:00
Michael Eischer
c7b770eb1f convert MemorizeList to be repository based 
Ideally, code that uses a repository shouldn't directly interact with
the underlying backend. Thus, move MemorizeList one layer up.
 2023-10-25 23:01:35 +02:00
Michael Eischer
1b8a67fe76 move Backend interface to backend package 2023-10-25 23:00:18 +02:00
Michael Eischer
ceb0774af1 backend: make LoadAll independent of restic package 2023-10-25 22:58:39 +02:00
Michael Eischer
b6d79bdf6f restic: decouple restic.Handle 2023-10-25 22:54:07 +02:00
Michael Eischer
7881309d63 backend: move backend implementation helpers to util package 
This removes code that is only used within a backend implementation from
the backend package. The latter now only contains code that also has
external users.
 2023-10-25 22:54:07 +02:00
Michael Eischer
8e6fdf5edf
Merge pull request #4520 from awannabeengineer/load-retry-nonexistent 
retry: Do not retry Load() if file does not exist
 2023-10-25 20:42:05 +00:00
Arash Farr
d15ffd9c92 retry: Do not retry Load() if file does not exist 2023-10-22 13:25:32 -05:00
Michael Eischer
7f05af02b9
Merge pull request #4480 from AgathaSorceress/add-rest-auth-env 
Support reading basic auth credentials for REST server from environment variables
 2023-10-21 17:41:08 +00:00
Michael Eischer
ab6defbace
Merge pull request #4519 from adamantike/backend/sftp/add-args-option 
backend/sftp: Add sftp.args option
 2023-10-21 17:37:46 +00:00
Michael Eischer
6a4d6d5da4 sftp: test that Args and Command options cannot be set at the same time 2023-10-21 19:26:39 +02:00
Michael Manganiello
41f70f1f4f backend/sftp: Add sftp.args option 
Allow setting custom arguments for the `sftp` backend, by using the
`sftp.args` option. This is similar to the approach already implemented
in the `rclone` backend, to support new arguments without requiring
future code changes for each different SSH argument.

Closes #4241
 2023-10-21 19:25:44 +02:00
Michael Eischer
6c7560e537 backend/b2: Switch to official library Backblaze/blazer 
According to https://github.com/kurin/blazer Readme, the repository is
no longer maintained, and the official one at
https://github.com/Backblaze/blazer took over.

Closes #4148
 2023-10-21 19:18:13 +02:00
Agatha V. Lovelace
f1877e721e
feat: support reading REST credentials from env 2023-10-20 13:04:58 +02:00
Michael Eischer
f750aa8dfb backend/limiter: replace juju/ratelimit with x/time/rate 2023-10-17 22:49:25 +02:00
Michael Eischer
4a33370072 backend: Fix unreliable TestListCancel/Timeout test 
The test uses `WithTimeout` to create a context that cancels the List
operation after a given delay. Several backends internally use a derived
child context created using WithCancel.

The cancellation of a context first closes the done channel of the
context (here: the `WithTimeout` context) and _afterwards_ propagates
the cancellation to child contexts (here: the `WithCancel` context).
Therefor if the List implementation uses a child context, then it may
take a moment until that context is also cancelled. Thus give the
context cancellation a moment to propagate.
 2023-07-23 11:47:56 +02:00
Michael Eischer
96eada3d5f backend/sema: add test for freeze functionality 2023-07-22 23:23:17 +02:00
Michael Eischer
f490288738 lock: freeze backend operations while refreshing stale lock 
Freeze new backend operations while trying to refresh a stale lock.
 2023-07-22 19:55:44 +02:00
Michael Eischer
978ebaac49 rest: use http status code constants 2023-07-14 22:39:38 +02:00
Michael Eischer
307aeb6849 rest: Don't return error if listing non-existent directory 
When transferring a repository from S3 to, for example, a local disk
then all empty folders will be missing.

When saving files, the missing intermediate folders are created
automatically. Therefore, missing directories can be ignored by the
`List()` operation.
 2023-07-14 22:39:38 +02:00
Michael Eischer
e990d3d483 azure: Support authentication using managed / workload identity 
See https://github.com/Azure/azure-sdk-for-go/tree/sdk/azidentity/v1.3.0/sdk/azidentity
 2023-07-09 14:47:03 +02:00
Michael Eischer
e36d17a6f8 azure: tweak documentation for endpoint suffix 2023-07-07 23:10:26 +02:00
arjunajesh
068b115abc added azure domain parameter 2023-07-07 23:10:26 +02:00
Michael Eischer
93038ed8f4 s3: restore retries for minio tests 2023-06-17 15:25:08 +02:00
Michael Eischer
b5511e8e4c Fix linter warnings 2023-06-17 15:15:58 +02:00
Michael Eischer
50e0d5e6b5 backend: Hardcode backend scheme in Factory 
Our ParseConfig implementations always expect a specific scheme, thus no
other scheme would work.
 2023-06-17 15:15:58 +02:00
Michael Eischer
705ad51bcc backend: check that StripPassword can be called 2023-06-17 15:15:58 +02:00
Michael Eischer
13a8b5822f backend: Adjust tests to use the Factory to instantiate the backend 
This drastically reduces the amount of duplicated test code.
 2023-06-17 15:15:58 +02:00
Michael Eischer
3d3bb88745 b2: remove duplicate check for config file during repository creation 
No other backend implements that check. The check that a repository is
not yet initialized is handled by the Repository later on.
 2023-06-17 15:15:58 +02:00
Michael Eischer
3a3cf608f5 b2/s3: Move config validation from ApplyEnvironment to Open/Create 
Conceptually the backend configuration should be validated when creating
or opening the backend, but not when filling in information from
environment variables into the configuration.
 2023-06-17 15:15:58 +02:00
Michael Eischer
19ac12d95b location: make genericBackendFactory private 2023-06-17 15:15:58 +02:00
Michael Eischer
3325a7c862 location: extract backend specific part of StripPassword 
The tests for the rest backend now reside there.
 2023-06-17 15:15:58 +02:00
Michael Eischer
555be49a79 location: Make ParseConfig-test backend agnostic 
The backend specific parts of the test are now directly handled by the
respective backend. Duplicate tests were removed.
 2023-06-17 15:15:58 +02:00
Michael Eischer
9aa9e0d1ec local/sftp: move limiter setup into backend 2023-06-17 15:15:58 +02:00
Michael Eischer
7d12c29286 backend: Unify backend construction using factory and registry 
This unified construction removes most backend-specific code from
global.go. The backend registry will also enable integration tests to
use custom backends if necessary.
 2023-06-17 15:15:57 +02:00
Michael Eischer
56836364a4 backend: pass context into every backend constructor 2023-06-17 15:15:57 +02:00
DRON-666
c37d587f81 Use PreallocateFile in local backend 2023-06-09 11:49:06 +03:00
Michael Eischer
609367195a backend: Fix test compilation with Go 1.18 and 1.19 2023-06-07 22:31:15 +02:00
Michael Eischer
18eb1d3ab0 backend: test cleanup 2023-06-07 22:31:15 +02:00
Michael Eischer
32a6b66267 backend: add standardized Config.ApplyEnvironment 
This removes the backend specific special cases while parsing the
configuration in `global.go`.
 2023-06-07 22:31:15 +02:00
Michael Eischer
f903db492c backend: let ParseConfig return a Config pointer 
In order to change the backend initialization in `global.go` to be able
to generically call cfg.ApplyEnvironment() for supported backends, the
`interface{}` returned by `ParseConfig` must contain a pointer to the
configuration.

An alternative would be to use reflection to convert the type from
`interface{}(Config)` to `interface{}(*Config)` (from value to pointer
type). However, this would just complicate the type mess further.
 2023-06-07 22:31:15 +02:00
Michael Eischer
25a0be7f26 backend: fix linter warnings 2023-06-07 22:31:15 +02:00
Michael Eischer
a27b7f1370 backend: use generic instead of any type for test suite 2023-06-07 22:31:14 +02:00
Michael Eischer
fa361dbfbd backend: use generic implementation for ParseConfig tests 2023-06-07 22:30:35 +02:00
Michael Eischer
5260d38980 backend: let ParseConfig return concrete type 2023-06-07 22:30:35 +02:00
Michael Eischer
2f7b4ceae1 backend: Move environment based configuration into backend 2023-06-07 22:30:35 +02:00
sohalt
ed5b2c2c9b gs: support other regions than us 2023-05-26 19:54:42 +02:00
Michael Eischer
658aa4c0f7
Merge pull request #4339 from greatroar/ioctl 
backend, termstatus: fix ioctl calls for 64-bit big-endian platforms
 2023-05-26 19:34:14 +02:00
greatroar
51dc80be5b backend: Use IoctlSetPointerInt for tcsetpgrp 
This function casts its argument to int32 before passing it to the
system call, so that big-endian CPUs read the lower rather than the
upper 32 bits of the pid.

This also gets rid of the last import of "unsafe" in the Unix build.
I changed syscall to x/sys/unix while I was at it, to remove one more
import line. The constants and types there are aliases for their syscall
counterparts.
 2023-05-25 17:31:51 +02:00
Michael Eischer
16ba237d8b azure: Fix missing rate limiting 2023-05-18 21:17:53 +02:00
Michael Eischer
22c9276719 rest: remove unused parameter 2023-05-18 21:17:53 +02:00
Michael Eischer
472bf5184f Replace lots of unused parameters with _ 
The parameters are required by the implemented function signature or interface.
 2023-05-18 21:17:53 +02:00
Michael Eischer
d1a5ec7839 Rename unused testing parameter to _ 
The parameter is an additional marker that the test helper must only be
used for tests.
 2023-05-18 21:17:53 +02:00
Michael Eischer
1514593f22 Remove unused context or testing parameters 2023-05-18 21:17:53 +02:00
Michael Eischer
5e4e268bdc Use _ as parameter name for unused Context 
The context is required by the implemented interface.
 2023-05-18 21:15:45 +02:00
Michael Eischer
2fa8b96843 Don't shadow builtins 2023-05-18 21:15:45 +02:00
Michael Eischer
319087c056 Remove redundant if ...; err != nil; return pattern 2023-05-18 21:15:45 +02:00
Michael Eischer
f9850b79b5 rest/sftp: Remove redundant fatal from error message 
This caused restic to exit with error messages like
`Fatal: parsing repository location failed: Fatal: sftp path [...]`
`Fatal: create repository at rest:http://localhost:12345/ failed: Fatal: config file already exists`
 2023-05-18 17:22:26 +02:00
Michael Eischer
d05f6211d1 lock: Do not limit backend concurrency for lock files 
restic must be able to refresh lock files in time. However, large
uploads over slow connections can cause the lock refresh to be stuck
behind the large uploads and thus time out.
 2023-05-08 19:04:46 +02:00
Michael Eischer
831f593b87 backend/sema: Add tests 2023-04-22 13:20:20 +02:00