From 9649a9c62bd7a9db182470fda1d227ca30c5357a Mon Sep 17 00:00:00 2001 From: Michael Eischer Date: Mon, 26 Aug 2024 12:17:43 +0200 Subject: [PATCH 1/3] doc: use regional urls for Amazon S3 and add generic s3 provider section Split description for non-Amazon S3 providers into separate section. The section now also includes the `s3.bucket-lookup` extended option. Switch to using regional URLs for Amazon S3 to replace the need for setting the region. --- doc/030_preparing_a_new_repo.rst | 64 ++++++++++++++++++-------------- 1 file changed, 36 insertions(+), 28 deletions(-) diff --git a/doc/030_preparing_a_new_repo.rst b/doc/030_preparing_a_new_repo.rst index 87975f9fa..462a66d75 100644 --- a/doc/030_preparing_a_new_repo.rst +++ b/doc/030_preparing_a_new_repo.rst @@ -249,28 +249,22 @@ while creating the bucket. $ export AWS_ACCESS_KEY_ID= $ export AWS_SECRET_ACCESS_KEY= +When using temporary credentials make sure to include the session token via +the environment variable ``AWS_SESSION_TOKEN``. + You can then easily initialize a repository that uses your Amazon S3 as -a backend. If the bucket does not exist it will be created in the -default location: +a backend. Make sure to use the endpoint for the correct region. The example +uses ``us-east-1``. If the bucket does not exist it will be created in that region: .. code-block:: console - $ restic -r s3:s3.amazonaws.com/bucket_name init + $ restic -r s3:s3.us-east-1.amazonaws.com/bucket_name init enter password for new repository: enter password again: - created restic repository eefee03bbd at s3:s3.amazonaws.com/bucket_name + created restic repository eefee03bbd at s3:s3.us-east-1.amazonaws.com/bucket_name Please note that knowledge of your password is required to access the repository. Losing your password means that your data is irrecoverably lost. -If needed, you can manually specify the region to use by either setting the -environment variable ``AWS_DEFAULT_REGION`` or calling restic with an option -parameter like ``-o s3.region="us-east-1"``. If the region is not specified, -the default region is used. Afterwards, the S3 server (at least for AWS, -``s3.amazonaws.com``) will redirect restic to the correct endpoint. - -When using temporary credentials make sure to include the session token via -then environment variable ``AWS_SESSION_TOKEN``. - Until version 0.8.0, restic used a default prefix of ``restic``, so the files in the bucket were placed in a directory named ``restic``. If you want to access a repository created with an older version of restic, specify the path @@ -278,25 +272,14 @@ after the bucket name like this: .. code-block:: console - $ restic -r s3:s3.amazonaws.com/bucket_name/restic [...] + $ restic -r s3:s3.us-east-1.amazonaws.com/bucket_name/restic [...] -For an S3-compatible server that is not Amazon (like Minio, see below), -or is only available via HTTP, you can specify the URL to the server -like this: ``s3:http://server:port/bucket_name``. - .. note:: restic expects `path-style URLs `__ - like for example ``s3.us-west-2.amazonaws.com/bucket_name``. + like for example ``s3.us-west-2.amazonaws.com/bucket_name`` for Amazon S3. Virtual-hosted–style URLs like ``bucket_name.s3.us-west-2.amazonaws.com``, where the bucket name is part of the hostname are not supported. These must be converted to path-style URLs instead, for example ``s3.us-west-2.amazonaws.com/bucket_name``. - -.. note:: Certain S3-compatible servers do not properly implement the - ``ListObjectsV2`` API, most notably Ceph versions before v14.2.5. On these - backends, as a temporary workaround, you can provide the - ``-o s3.list-objects-v1=true`` option to use the older - ``ListObjects`` API instead. This option may be removed in future - versions of restic. - + See below for configuration options for S3-compatible storage from other providers. Minio Server ************ @@ -321,13 +304,38 @@ this command. .. code-block:: console - $ ./restic -r s3:http://localhost:9000/restic init + $ restic -r s3:http://localhost:9000/restic init enter password for new repository: enter password again: created restic repository 6ad29560f5 at s3:http://localhost:9000/restic1 Please note that knowledge of your password is required to access the repository. Losing your password means that your data is irrecoverably lost. +S3-compatible Storage +********************* + +For an S3-compatible server that is not Amazon, you can specify the URL to the server +like this: ``s3:https://server:port/bucket_name``. + +If needed, you can manually specify the region to use by either setting the +environment variable ``AWS_DEFAULT_REGION`` or calling restic with an option +parameter like ``-o s3.region="us-east-1"``. If the region is not specified, +the default region ``us-east-1`` is used. + +To select between path-style and virtual-hosted access, the extended option +``-o s3.bucket-lookup=auto`` can be used. It supports the following values: + +- ``auto``: Default behavior. Uses ``dns`` for Amazon and Google endpoints. Uses + ``path`` for all other endpoints +- ``dns``: Use virtual-hosted-style bucket access +- ``path``: Use path-style bucket access + +Certain S3-compatible servers do not properly implement the ``ListObjectsV2`` API, +most notably Ceph versions before v14.2.5. On these backends, as a temporary +workaround, you can provide the ``-o s3.list-objects-v1=true`` option to use the +older ``ListObjects`` API instead. This option may be removed in future versions +of restic. + Wasabi ************ From 7de53a51b8c30e52de590b4e7c08a94684ed9d76 Mon Sep 17 00:00:00 2001 From: Michael Eischer Date: Mon, 26 Aug 2024 12:21:13 +0200 Subject: [PATCH 2/3] doc: shrink wasabi / alibaba cloud example Remove descriptions for both providers and shorten the example to the minimum. --- doc/030_preparing_a_new_repo.rst | 60 ++++++-------------------------- 1 file changed, 10 insertions(+), 50 deletions(-) diff --git a/doc/030_preparing_a_new_repo.rst b/doc/030_preparing_a_new_repo.rst index 462a66d75..a169f34cc 100644 --- a/doc/030_preparing_a_new_repo.rst +++ b/doc/030_preparing_a_new_repo.rst @@ -337,73 +337,33 @@ older ``ListObjects`` API instead. This option may be removed in future versions of restic. Wasabi -************ +****** -`Wasabi `__ is a low cost Amazon S3 conformant object storage provider. -Due to its S3 conformance, Wasabi can be used as a storage provider for a restic repository. +S3 storage from `Wasabi `__ can be used as follows. -- Create a Wasabi bucket using the `Wasabi Console `__. -- Determine the correct Wasabi service URL for your bucket `here `__. - -You must first setup the following environment variables with the -credentials of your Wasabi account. +- Determine the correct Wasabi service URL for your bucket `here `__. +- Set environment variables with the necessary account credentials .. code-block:: console $ export AWS_ACCESS_KEY_ID= $ export AWS_SECRET_ACCESS_KEY= - -Now you can easily initialize restic to use Wasabi as a backend with -this command. - -.. code-block:: console - - $ ./restic -r s3:https:/// init - enter password for new repository: - enter password again: - created restic repository xxxxxxxxxx at s3:https:/// - Please note that knowledge of your password is required to access - the repository. Losing your password means that your data is irrecoverably lost. + $ restic -r s3:https:/// init Alibaba Cloud (Aliyun) Object Storage System (OSS) ************************************************** -`Alibaba OSS `__ is an -encrypted, secure, cost-effective, and easy-to-use object storage -service that enables you to store, back up, and archive large amounts -of data in the cloud. +S3 storage from `Alibaba OSS `__ can be used as follows. -Alibaba OSS is S3 compatible so it can be used as a storage provider -for a restic repository with a couple of extra parameters. - -- Determine the correct `Alibaba OSS region endpoint `__ - this will be something like ``oss-eu-west-1.aliyuncs.com`` -- You'll need the region name too - this will be something like ``oss-eu-west-1`` - -You must first setup the following environment variables with the -credentials of your Alibaba OSS account. +- Determine the correct `Alibaba OSS region endpoint `__ - this will be something like ``oss-eu-west-1.aliyuncs.com`` +- You will need the region name too - this will be something like ``oss-eu-west-1`` +- Set environment variables with the necessary account credentials .. code-block:: console $ export AWS_ACCESS_KEY_ID= $ export AWS_SECRET_ACCESS_KEY= - -Now you can easily initialize restic to use Alibaba OSS as a backend with -this command. - -.. code-block:: console - - $ ./restic -o s3.bucket-lookup=dns -o s3.region= -r s3:https:/// init - enter password for new backend: - enter password again: - created restic backend xxxxxxxxxx at s3:https:/// - Please note that knowledge of your password is required to access - the repository. Losing your password means that your data is irrecoverably lost. - -For example with an actual endpoint: - -.. code-block:: console - - $ restic -o s3.bucket-lookup=dns -o s3.region=oss-eu-west-1 -r s3:https://oss-eu-west-1.aliyuncs.com/bucketname init + $ restic -o s3.bucket-lookup=dns -o s3.region= -r s3:https:/// init OpenStack Swift *************** From 8a8f5f3986944eec170a85f850ca8dd3d13334a9 Mon Sep 17 00:00:00 2001 From: Michael Eischer Date: Mon, 26 Aug 2024 12:24:02 +0200 Subject: [PATCH 3/3] doc: fix typos --- doc/030_preparing_a_new_repo.rst | 2 +- doc/manual_rest.rst | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/doc/030_preparing_a_new_repo.rst b/doc/030_preparing_a_new_repo.rst index a169f34cc..fd5b31127 100644 --- a/doc/030_preparing_a_new_repo.rst +++ b/doc/030_preparing_a_new_repo.rst @@ -307,7 +307,7 @@ this command. $ restic -r s3:http://localhost:9000/restic init enter password for new repository: enter password again: - created restic repository 6ad29560f5 at s3:http://localhost:9000/restic1 + created restic repository 6ad29560f5 at s3:http://localhost:9000/restic Please note that knowledge of your password is required to access the repository. Losing your password means that your data is irrecoverably lost. diff --git a/doc/manual_rest.rst b/doc/manual_rest.rst index a7a0f96e0..e3a02b386 100644 --- a/doc/manual_rest.rst +++ b/doc/manual_rest.rst @@ -8,7 +8,7 @@ Usage help is available: .. code-block:: console - $ ./restic --help + $ restic --help restic is a backup program which allows saving multiple revisions of files and directories in an encrypted repository stored on different backends. @@ -85,7 +85,7 @@ command: .. code-block:: console - $ ./restic backup --help + $ restic backup --help The "backup" command creates a new snapshot and saves the files and directories given as the arguments.