Commit graph

104 commits

Author SHA1 Message Date
Alexander Neumann
5dc8d3588d GS: Use generic http transport
During the development of #1524 I discovered that the Google Cloud
Storage backend did not yet use the HTTP transport, so things such as
bandwidth limiting did not work. This commit does the necessary magic to
make the GS library use our HTTP transport.
2018-01-27 20:12:34 +01:00
Alexander Neumann
c34db983d8 Read TLS client cert and key from the same file 2018-01-27 14:02:01 +01:00
Bryce Chidester
e805b968b1 Support for TLS client certificate authentication
This adds --tls-client-cert and --tls-client-key parameters and enables use
of that certificate/key pair when connecting to https servers.
2018-01-27 13:18:22 +01:00
Alexander Neumann
e11a183578 Merge pull request #1588 from restic/fix-sftp-without-tty
sftp: Allow running ssh without a tty
2018-01-26 21:56:41 +01:00
Alexander Neumann
7719cf88d9 b2: Check timeout 2018-01-26 21:07:05 +01:00
Alexander Neumann
00e905ebe6 sftp: Allow running ssh without a tty 2018-01-26 19:21:14 +01:00
Igor Fedorenko
abc4027083 Use errors.Cause in backend TestListCancel
Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>
2018-01-25 08:53:50 -05:00
Alexander Neumann
7e6bfdae79 backend/rest: Implement REST API v2 2018-01-23 23:15:26 +01:00
Alexander Neumann
e835abeceb backend/test: Reliably trigger timeout error 2018-01-23 23:14:05 +01:00
Alexander Neumann
b0c6e53241 Fix calls to repo/backend.List() everywhere 2018-01-21 21:15:09 +01:00
Alexander Neumann
e9ea268847 Change List() implementation for all backends 2018-01-21 21:15:09 +01:00
Alexander Neumann
c4e9d5d11e backend: Add tests for new List() function 2018-01-21 18:35:37 +01:00
Alexander Neumann
52230b8f07 backend: Rework List()
For a discussion see #1567
2018-01-21 18:35:37 +01:00
Alexander Neumann
2130897ce0 rest: Add test for external server 2018-01-20 10:25:47 +01:00
Alexander Neumann
67da240068 rest: Refactor backend tests 2018-01-20 10:25:37 +01:00
Alexander Neumann
1046eabf95 rest: Remove unneeded tempdir 2018-01-20 10:13:04 +01:00
Alexander Neumann
0bdb131521 Remove SuspendSignalHandler 2018-01-17 23:14:47 +01:00
Alexander Neumann
05958caf6e sftp: Prompt for password, don't terminate on SIGINT
This is a follow-up on fb9729fdb9, which
runs the `ssh` in its own process group and selects that process group
as the foreground group. After the sftp connection is established,
restic switches back to the previous foreground process group.

This allows `ssh` to prompt for the password, but it won't receive
the interrupt signal (SIGINT, ^C) later on, because it is not in the
foreground process group any more, allowing a clean tear down.
2018-01-17 23:02:47 +01:00
Igor Fedorenko
8c550ca011 fixed restic-check does not retry backend.Test failures
added missing RetryBackend.Test implementation

Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>
2018-01-06 23:22:35 -05:00
Alexander Neumann
b45fc89512 local/sftp: Create repo dirs on demand in Save() 2018-01-05 17:51:09 +01:00
Alexander Neumann
6c2b2a58ad backend: Retry deletes 2017-12-22 22:41:28 +01:00
Alexander Neumann
7d8765a937 backend: Only return top-level files for most dirs
Fixes #1478
2017-12-14 19:14:16 +01:00
Harshavardhana
27ccea6371 Since upgrade to minio-go 4.0 remove workaround
We added previously a code to fix the issue of chaining
credentials, we do not need this anymore since the
upstream minio-go already has this relevant change.
2017-12-09 02:01:42 -08:00
Alexander Neumann
8b3b7bc5ef s3: Use context 2017-12-08 22:04:55 +01:00
Alexander Neumann
934ae1b559 Update to minio-go 4 2017-12-08 21:52:50 +01:00
George Armhold
1695c8ed55 use global context for check, debug, dump, find, forget, init, key,
list, mount, tag, unlock commands

gh-1434
2017-12-06 07:02:55 -05:00
George Armhold
0dc31c03e1 remove check for context.Canceled
gh-1434
2017-12-06 05:38:29 -05:00
George Armhold
be24237063 make retry code context-aware.
detect cancellation in backend, so that retry code does not keep trying
once user has hit ^c

gh-1434
2017-12-03 07:22:14 -05:00
George Armhold
d886cb5c27 replace ad-hoc context.TODO() with gopts.ctx, so that cancellation
can properly trickle down from cmd_*.

gh-1434
2017-12-03 07:22:14 -05:00
Alexander Neumann
0b44c629f2 retry: Remove file after failed save 2017-11-30 22:05:14 +01:00
Alexander Neumann
134abbd82b rest: Use client for creating the repository
Before, creating a new repo via REST would use the defaut HTTP client,
which is not a problem unless the server uses HTTPS and a TLS
certificate which isn't signed by a CA in the system's CA store. In this
case, all commands work except the 'init' command, which fails with a
message like "invalid certificate".
2017-11-25 20:56:40 +01:00
Alexander Neumann
1ebf0e8de8 Merge pull request #1437 from restic/fix-1292
s3: Document and remove default prefix
2017-11-25 11:34:26 +01:00
Alexander Neumann
47b326b7b5 Merge pull request #1423 from harshavardhana/creds
Fix chaining of credentials for minio-go
2017-11-24 21:57:52 +01:00
Alexander Neumann
262b0cd9d4 s3: Remove default prefix "/restic" 2017-11-21 21:33:09 +01:00
Alexander Neumann
e83ec17e95 s3: Correct comment 2017-11-20 22:21:39 +01:00
Harshavardhana
41c8c946ba Fix chaining of credentials for minio-go
chaining failed because chaining provider
was only looking for subsequent credentials
provider after an error. Writer a new
chaining provider which proceeds to fetch
new credentials also under situations where
providers do not return but instead return
no keys at all.

Fixes https://github.com/restic/restic/issues/1422
2017-11-18 02:51:12 -08:00
George Armhold
0268d0e7d6 swift backend: limit http concurrency in Save(), Stat(), Test(), Remove(),
List().

move comment regarding problematic List() backend api (it's s3's ListObjects
that has a problem, NOT swift's ObjectsWalk).

As per discussion in PR #1399.
2017-11-02 18:29:32 -04:00
George Armhold
8515d093e0 swift backend: fix premature release of semaphore in Load() & document
concurrency issue in List().

refactor wrapReader from b2 -> semaphore so it can be used elsewhere.

As per discussion in PR #1399.
2017-11-02 12:38:17 -04:00
George Armhold
99ac0da4bc s3 backend: limit http concurrency in Save(), Stat(), Test(), Remove()
NB: List() is NOT currently limited, as it would cause deadlock due to
be.client.ListObjects() implementation.

as per discussion in PR #1399
2017-11-01 09:40:54 -04:00
George Armhold
d069ee31b2 GS backend: limit http concurrency in Save(), Stat(), Test(), Remove(), List()
as per discussion in PR #1399
2017-10-31 08:01:43 -04:00
George Armhold
981752ade0 Azure backend: limit http concurrency in Stat(), Test(), Remove()
as per discussion in PR #1399
2017-10-31 07:32:30 -04:00
George Armhold
2f8147af59 log unexpected errs from b2 ListCurrentObject()
gh-1385
2017-10-29 08:53:39 -04:00
Alexander Neumann
f854a41ba9 Merge pull request #1399 from armhold/deadlock2
prevent deadlock in List() for B2 when b2.connections=1
2017-10-29 09:26:46 +01:00
George Armhold
3304b0fcf0 prevent deadlock in List() for B2 when b2.connections=1
This is a fix for the following situation (gh-1188):

List() grabs a semaphore token upon entry, starts a goroutine, and
does not release the token until the routine exits (via a defer).

The goroutine iterates over the results from ListCurrentObjects(),
sending them one at a time to a channel, where they are ultimately
processed by be.Load().

Since be.Load() also needs a token, this will result in deadlock if
b2.connections=1.

This fix changes List() so that the token is only held during the call
to ListCurrentObjects().
2017-10-28 18:46:47 -04:00
George Armhold
d8938e259a sftp ReadDir: add path to return error messages (gh-1323)
fix missing "Close" string in debug log fmt
2017-10-28 14:16:27 -04:00
Alexander Neumann
7a99418dc5 Merge pull request #1393 from armhold/lint-errcheck
detect errors from fs.Walk() in local backend List()
2017-10-28 09:56:11 +02:00
Alexander Neumann
c71ba466ea Merge pull request #1391 from armhold/b2-listmax
pass in defaultListMaxItems to b2Backend constructor
2017-10-28 09:54:57 +02:00
George Armhold
8a37c07295 send errors from fs.Walk() to debug log
clarify non-err returns from Walk where err is already proved to be nil
2017-10-27 08:41:17 -04:00
George Armhold
bd0ada7842 go fmt 2017-10-26 16:37:11 -04:00
George Armhold
eea96f652d go fmt 2017-10-26 16:22:10 -04:00