From 9c1d49e3123f2e1dd117f16a17af60a39dce868c Mon Sep 17 00:00:00 2001 From: Luc Gommans Date: Sat, 27 Nov 2021 02:55:18 +0100 Subject: [PATCH 1/4] Document "forget" security considerations and add references Removing data based on a policy when the attacker had the opportunity to add data to your repository comes with some considerations. This is added to the 060_forget.rst documentation. That document is also updated to reflect that restic now considers the current system time while running "forget". References to the security considerations section are added: - In `restic forget --help` - In the threat model (design.rst) - In the (030) setup section where an append-only setup is referenced A reference is also to be added to the `rest-server` readme's append-only paragraph (see my fork). This commit also resolves a typo (amount->number for countable noun), changes a password length recommendation into the metric that actually matters when creating passwords (entropy) since I was editing these doc files anyway, and updates the outdated copyright year in `conf.py`. Some wording in 060_forget (line 21..22) was changed to clarify what "forget" and "prune" do, to try and avoid the apparent misconception that "forget" does not remove any data. --- cmd/restic/cmd_forget.go | 2 ++ doc/030_preparing_a_new_repo.rst | 4 ++- doc/060_forget.rst | 48 ++++++++++++++++++++++++++++---- doc/conf.py | 2 +- doc/design.rst | 22 ++++++++++----- 5 files changed, 64 insertions(+), 14 deletions(-) diff --git a/cmd/restic/cmd_forget.go b/cmd/restic/cmd_forget.go index fd34f4038..01848745c 100644 --- a/cmd/restic/cmd_forget.go +++ b/cmd/restic/cmd_forget.go @@ -18,6 +18,8 @@ The "forget" command removes snapshots according to a policy. Please note that this command really only deletes the snapshot object in the repository, which is a reference to data stored there. In order to remove this (now unreferenced) data after 'forget' was run successfully, see the 'prune' command. +When using this command on special append-only repositories, please see the +documentation for security considerations. EXIT STATUS =========== diff --git a/doc/030_preparing_a_new_repo.rst b/doc/030_preparing_a_new_repo.rst index ebeec8800..47bd7439d 100644 --- a/doc/030_preparing_a_new_repo.rst +++ b/doc/030_preparing_a_new_repo.rst @@ -657,7 +657,9 @@ credentials) is encrypted/decrypted locally, then sent/received via A more advanced version of this setup forbids specific hosts from removing files in a repository. See the `blog post by Simon Ruderich `_ -for details. +for details. When using ``restic forget`` on such a repository, additional +security considerations apply: please review the documentation on removing +backup snapshots. The rclone command may also be hard-coded in the SSH configuration or the user's public key, in this case it may be sufficient to just start the SSH diff --git a/doc/060_forget.rst b/doc/060_forget.rst index 2c537f1f0..afa4f2c80 100644 --- a/doc/060_forget.rst +++ b/doc/060_forget.rst @@ -18,13 +18,13 @@ All backup space is finite, so restic allows removing old snapshots. This can be done either manually (by specifying a snapshot ID to remove) or by using a policy that describes which snapshots to forget. For all remove operations, two commands need to be called in sequence: -``forget`` to remove a snapshot and ``prune`` to actually remove the -data that was referenced by the snapshot from the repository. This can +``forget`` to remove a snapshot and ``prune`` to remove the remaining +data that was only referenced by this snapshot from the repository. This can be automated with the ``--prune`` option of the ``forget`` command, which runs ``prune`` automatically if snapshots have been removed. Pruning snapshots can be a time-consuming process, depending on the -amount of snapshots and data to process. During a prune operation, the +number of snapshots and data to process. During a prune operation, the repository is locked and backups cannot be completed. Please plan your pruning so that there's time to complete it and it doesn't interfere with regular backup runs. @@ -164,6 +164,9 @@ The most important command-line parameter here is ``--dry-run`` which instructs restic to not remove anything but print which snapshots would be removed. +.. note:: If you use append-only backups, some best practices apply. + Please refer to the security considerations below. + When ``forget`` is run with a policy, restic loads the list of all snapshots, then groups these by host name and list of directories. The grouping options can be set with ``--group-by``, to only group snapshots by paths and @@ -206,6 +209,8 @@ The ``forget`` command accepts the following parameters: .. note:: All calendar related ``--keep-*`` options work on the natural time boundaries and not relative to when you run the ``forget`` command. Weeks are Monday 00:00 -> Sunday 23:59, days 00:00 to 23:59, hours :00 to :59, etc. + Snapshots seemingly made in the future (relative to when you run the + ``forget`` command) will be ignored and never removed. .. note:: Specifying ``--keep-tag ''`` will match untagged snapshots only. @@ -312,8 +317,9 @@ four Sundays, but remove the rest: --------------------------------------------------------------- 8 snapshots -The result of the ``forget --keep-daily`` operation does not depend on when it -is run, it will only count the days for which a snapshot exists. This is a +The result of the ``forget --keep-daily`` operation only partially depends on when it +is run: it will only count the days for which a snapshot exists, although +with a `time` lying in the future are ignored and never removed. This is a safety feature: it prevents restic from removing snapshots when no new ones are created. Otherwise, running ``forget --keep-daily 4`` on a Friday (without any snapshot Monday to Thursday) would remove all snapshots! @@ -336,6 +342,38 @@ could specify: (Note that `1w` is not a recognized duration, so you will have to specify `7d` instead) +Security considerations in append-only mode +=========================================== + +To prevent data from being deleted by a compromised backup client (for example +due to a ransomware infection), a so-called append-only mode can be used. This +requires the server to deny delete and overwrite operations, which is not +possible on many standard back-ends. Software such as `rest-server`_ or +`rclone`_ can be used instead or in addition. + +.. _rest-server: https://github.com/restic/rest-server/ +.. _rclone: https://rclone.org/ + +To recover disk space from obsolete snapshots, ``forget`` and ``prune`` must be +run on a repository with full read-write access. If an attacker can do this, +the protection offered by append-only mode is void. However, even if only the +trusted client runs the ``forget`` command, it can be possible for the attacker +to add snapshots in such a pattern that all legitimate snapshots are removed. + +If the ``forget`` policy is to keep three weekly snapshots, the attacker can +add an empty backup for each of the last three weeks with a `time` slightly +newer than the existing snapshots (but still within the target week). The next +time the repository administrator (or cron job) runs the ``forget`` policy, the +legitimate snapshots will be removed. Even without pruning, recovering data +would be messy and some metadata will be lost. + +To avoid this, policies applied to append-only repositories should use the +``--keep-within`` option. If the system time is set correctly when ``forget`` +runs, this will allow you to notice problems with the backup or the compromised +host. This is, of course, limited to the specified duration: if +``restic forget --keep-within 7d`` is run 8 days after the last good snapshot, +then the attacker can still use that opportunity to remove all good snapshots. + Customize pruning ***************** diff --git a/doc/conf.py b/doc/conf.py index 3c0af927b..97c728ac6 100644 --- a/doc/conf.py +++ b/doc/conf.py @@ -35,7 +35,7 @@ master_doc = 'index' # General information about the project. project = 'restic' -copyright = '2018, restic authors' +copyright = '2021, restic authors' author = 'fd0' # The version info for the project you're documenting, acts as replacement for diff --git a/doc/design.rst b/doc/design.rst index c313c1a70..196ddaa66 100644 --- a/doc/design.rst +++ b/doc/design.rst @@ -607,7 +607,7 @@ examples of things an adversary could achieve in various circumstances. An adversary with read access to your backup storage location could: - Attempt a brute force password guessing attack against a copy of the - repository (even more reason to use long, 30+ character passwords). + repository (use strong passwords with sufficient entropy). - Infer which packs probably contain trees via file access patterns. - Infer the size of backups by using creation timestamps of repository objects. @@ -648,18 +648,26 @@ An adversary with write access to your files at the storage location could: An adversary who compromises a host system with append-only access to the backup repository could: +- Capture the password and decrypt backups from the past and in the future. + See the "leaked key" circumstance below. - Render new backups untrustworthy *after* the host has been compromised (due to having complete control over new backups). An attacker cannot delete or manipulate old backups. As such, restoring old snapshots created *before* a host compromise remains possible. - *Note: It is **not** recommended to ever run forget automatically for an - append-only backup to which a potentially compromised host has access - because an attacker using fake snapshots could cause forget to remove - correct snapshots.* +- Potentially manipulate the ``restic forget`` command into deleting all + legitimate snapshots, keeping only bogus snapshots added by the attacker. + Ransomware might try this in order to leave only one option to get your data + back: paying the ransom. For safe use of ``restic forget``, see the + documentation on removing backup snapshots. -An adversary who has a leaked key for a repository which has not been re-encrypted -could: +An adversary who has a leaked key for a repository could: - Decrypt existing and future backup data. If multiple hosts backup into the same repository, an attacker will get access to the backup data of every host. + Since the local encryption key gives access to the master key, a password + change will not prevent this. Changing the master key can currently be done + using ``restic copy`` which moves the data into a new repository with a new + master key, or by making a completely new repository and new backup. + Re-encrypting all data without creating a new repository is tracked in + :issue:`1602`. From 676d5d498c8542f5d9f7563fe2ee1f84224451ec Mon Sep 17 00:00:00 2001 From: "Leo R. Lundgren" Date: Sat, 19 Feb 2022 22:47:35 +0100 Subject: [PATCH 2/4] doc: Update forget security considerations and thread model --- cmd/restic/cmd_forget.go | 7 +- doc/030_preparing_a_new_repo.rst | 5 +- doc/060_forget.rst | 127 +++++++++++++++++-------------- doc/conf.py | 2 +- doc/design.rst | 43 +++++------ 5 files changed, 96 insertions(+), 88 deletions(-) diff --git a/cmd/restic/cmd_forget.go b/cmd/restic/cmd_forget.go index 01848745c..c814472f7 100644 --- a/cmd/restic/cmd_forget.go +++ b/cmd/restic/cmd_forget.go @@ -16,10 +16,9 @@ var cmdForget = &cobra.Command{ Long: ` The "forget" command removes snapshots according to a policy. Please note that this command really only deletes the snapshot object in the repository, which -is a reference to data stored there. In order to remove this (now unreferenced) -data after 'forget' was run successfully, see the 'prune' command. -When using this command on special append-only repositories, please see the -documentation for security considerations. +is a reference to data stored there. In order to remove the unreferenced data +after "forget" was run successfully, see the "prune" command. Please also read +the documentation for "forget" to learn about important security considerations. EXIT STATUS =========== diff --git a/doc/030_preparing_a_new_repo.rst b/doc/030_preparing_a_new_repo.rst index 47bd7439d..d6fb830c6 100644 --- a/doc/030_preparing_a_new_repo.rst +++ b/doc/030_preparing_a_new_repo.rst @@ -657,9 +657,8 @@ credentials) is encrypted/decrypted locally, then sent/received via A more advanced version of this setup forbids specific hosts from removing files in a repository. See the `blog post by Simon Ruderich `_ -for details. When using ``restic forget`` on such a repository, additional -security considerations apply: please review the documentation on removing -backup snapshots. +for details and the documentation for the ``forget`` command to learn about +important security considerations. The rclone command may also be hard-coded in the SSH configuration or the user's public key, in this case it may be sufficient to just start the SSH diff --git a/doc/060_forget.rst b/doc/060_forget.rst index afa4f2c80..5bf87f024 100644 --- a/doc/060_forget.rst +++ b/doc/060_forget.rst @@ -14,14 +14,13 @@ Removing backup snapshots ######################### -All backup space is finite, so restic allows removing old snapshots. -This can be done either manually (by specifying a snapshot ID to remove) -or by using a policy that describes which snapshots to forget. For all -remove operations, two commands need to be called in sequence: -``forget`` to remove a snapshot and ``prune`` to remove the remaining -data that was only referenced by this snapshot from the repository. This can -be automated with the ``--prune`` option of the ``forget`` command, -which runs ``prune`` automatically if snapshots have been removed. +All backup space is finite, so restic allows removing old snapshots. This can +be done either manually (by specifying a snapshot ID to remove) or by using a +policy that describes which snapshots to forget. For all remove operations, two +commands need to be called in sequence: ``forget`` to remove snapshots, and +``prune`` to remove the remaining data that was referenced only by the removed +snapshots. The latter can be automated with the ``--prune`` option of ``forget``, +which runs ``prune`` automatically if any snapshots were actually removed. Pruning snapshots can be a time-consuming process, depending on the number of snapshots and data to process. During a prune operation, the @@ -164,8 +163,8 @@ The most important command-line parameter here is ``--dry-run`` which instructs restic to not remove anything but print which snapshots would be removed. -.. note:: If you use append-only backups, some best practices apply. - Please refer to the security considerations below. +.. note:: If you use an append-only repository, some best practices apply. + Please refer to the security considerations below for more information. When ``forget`` is run with a policy, restic loads the list of all snapshots, then groups these by host name and list of directories. The grouping @@ -175,27 +174,27 @@ snapshots separately. This is a safety feature. The ``forget`` command accepts the following parameters: -- ``--keep-last n`` never delete the ``n`` last (most recent) snapshots -- ``--keep-hourly n`` for the last ``n`` hours in which a snapshot was - made, keep only the last snapshot for each hour. +- ``--keep-last n`` keep the ``n`` last (most recent) snapshots. +- ``--keep-hourly n`` for the last ``n`` hours which have one or more + snapshots, keep only the most recent one for each hour. - ``--keep-daily n`` for the last ``n`` days which have one or more - snapshots, only keep the last one for that day. + snapshots, keep only the most recent one for that day. - ``--keep-weekly n`` for the last ``n`` weeks which have one or more - snapshots, only keep the last one for that week. + snapshots, keep only the most recent one for that week. - ``--keep-monthly n`` for the last ``n`` months which have one or more - snapshots, only keep the last one for that month. + snapshots, keep only the most recent one for that month. - ``--keep-yearly n`` for the last ``n`` years which have one or more - snapshots, only keep the last one for that year. + snapshots, keep only the most recent one for that year. - ``--keep-tag`` keep all snapshots which have all tags specified by this option (can be specified multiple times). -- ``--keep-within duration`` keep all snapshots which have been made within - the duration of the latest snapshot. ``duration`` needs to be a number of - years, months, days, and hours, e.g. ``2y5m7d3h`` will keep all snapshots - made in the two years, five months, seven days, and three hours before the - latest snapshot. +- ``--keep-within duration`` keep all snapshots having a timestamp within + the specified duration of the latest snapshot, where ``duration`` is a + number of years, months, days, and hours. E.g. ``2y5m7d3h`` will keep all + snapshots made in the two years, five months, seven days, and three hours + before the latest (most recent) snapshot. - ``--keep-within-hourly duration`` keep all hourly snapshots made within - specified duration of the latest snapshot. The duration is specified in - the same way as for ``--keep-within`` and the method for determining + the specified duration of the latest snapshot. The ``duration`` is specified + in the same way as for ``--keep-within`` and the method for determining hourly snapshots is the same as for ``--keep-hourly``. - ``--keep-within-daily duration`` keep all daily snapshots made within specified duration of the latest snapshot. @@ -206,11 +205,13 @@ The ``forget`` command accepts the following parameters: - ``--keep-within-yearly duration`` keep all yearly snapshots made within specified duration of the latest snapshot. -.. note:: All calendar related ``--keep-*`` options work on the natural time +.. note:: All calendar related options (``--keep-*``) work on the natural time boundaries and not relative to when you run the ``forget`` command. Weeks - are Monday 00:00 -> Sunday 23:59, days 00:00 to 23:59, hours :00 to :59, etc. - Snapshots seemingly made in the future (relative to when you run the - ``forget`` command) will be ignored and never removed. + are Monday 00:00 to Sunday 23:59, days 00:00 to 23:59, hours :00 to :59, etc. + +.. note:: All duration related options (``--keep-within`` and ``--keep-within-*``) + ignore snapshots with a timestamp in the future (relative to when the + ``forget`` command is run) and these snapshots will hence not be removed. .. note:: Specifying ``--keep-tag ''`` will match untagged snapshots only. @@ -317,12 +318,12 @@ four Sundays, but remove the rest: --------------------------------------------------------------- 8 snapshots -The result of the ``forget --keep-daily`` operation only partially depends on when it -is run: it will only count the days for which a snapshot exists, although -with a `time` lying in the future are ignored and never removed. This is a -safety feature: it prevents restic from removing snapshots when no new ones are -created. Otherwise, running ``forget --keep-daily 4`` on a Friday (without any -snapshot Monday to Thursday) would remove all snapshots! +The result of the ``forget --keep-daily`` operation only partially depends on +when it is run; it will only count the days for which a snapshot exists, +although snapshots with a `time` lying in the future are ignored and never +removed. This is a safety feature: it prevents restic from removing snapshots +when no new ones are created. Otherwise, running ``forget --keep-daily 4`` on +a Friday (without any snapshot Monday to Thursday) would remove all snapshots! Another example: Suppose you make daily backups for 100 years. Then ``forget --keep-daily 7 --keep-weekly 5 --keep-monthly 12 --keep-yearly 75`` @@ -345,34 +346,46 @@ could specify: Security considerations in append-only mode =========================================== -To prevent data from being deleted by a compromised backup client (for example -due to a ransomware infection), a so-called append-only mode can be used. This -requires the server to deny delete and overwrite operations, which is not -possible on many standard back-ends. Software such as `rest-server`_ or -`rclone`_ can be used instead or in addition. +To prevent a compromised backup client from deleting its backups (for example +due to a ransomware infection), a repository service/backend can serve the +repository in a so-called append-only mode. This means that the repository can +only be written to and read from, while delete and overwrite operations are +denied. Restic's `rest-server`_ features the append-only mode, but few other +standard backends do. To support append-only with such a backend, one can use +`rclone`_ as a complement in between the backup client and the backend service. .. _rest-server: https://github.com/restic/rest-server/ -.. _rclone: https://rclone.org/ +.. _rclone: https://rclone.org/commands/rclone_serve_restic/ -To recover disk space from obsolete snapshots, ``forget`` and ``prune`` must be -run on a repository with full read-write access. If an attacker can do this, -the protection offered by append-only mode is void. However, even if only the -trusted client runs the ``forget`` command, it can be possible for the attacker -to add snapshots in such a pattern that all legitimate snapshots are removed. +To remove snapshots and recover the corresponding disk space, the ``forget`` +and ``prune`` commands must have full read, write and delete access to the +repository. If an attacker has this, the protection offered by append-only +mode is naturally void. -If the ``forget`` policy is to keep three weekly snapshots, the attacker can -add an empty backup for each of the last three weeks with a `time` slightly -newer than the existing snapshots (but still within the target week). The next -time the repository administrator (or cron job) runs the ``forget`` policy, the -legitimate snapshots will be removed. Even without pruning, recovering data -would be messy and some metadata will be lost. +However, even with append-only mode active, an attacker who is able to add +additional and empty or otherwise useless snapshots to the repository can +potentially cause a situation where a trusted client running ``forget`` with +certain ``--keep-*`` options might unknowingly remove legitimate snapshots, +leaving only the attackers useless snapshots. -To avoid this, policies applied to append-only repositories should use the -``--keep-within`` option. If the system time is set correctly when ``forget`` -runs, this will allow you to notice problems with the backup or the compromised -host. This is, of course, limited to the specified duration: if -``restic forget --keep-within 7d`` is run 8 days after the last good snapshot, -then the attacker can still use that opportunity to remove all good snapshots. +For example, if the ``forget`` policy is to keep three weekly snapshots, and +the attacker adds an empty snapshot for each of the last three weeks, all with +a timestamp (see the ``backup`` command's ``-`time`` option) slightly more +recent than the existing snapshots (but still within the target week), then the +next time the repository administrator (or scheduled job) runs the ``forget`` +command with this policy, the legitimate snapshots will be removed (since the +policy will use the most recent snapshot within each week). Even without +running ``prune``, recovering data would be messy and some metadata lost. + +To avoid this, ``forget`` policies applied to append-only repositories should +use the ``--keep-within`` option, as this will keep not only the attacker's +snapshots but also the legitimate ones. Assuming the system time is correctly +set when ``forget`` runs, this will allow the administrator to notice problems +with the backup or the compromised host (e.g. by seeing more snapshots than +usual or snapshots with suspicious timestamps). This is, of course, limited to +the specified duration: if ``forget --keep-within 7d`` is run 8 days after the +last good snapshot, then the attacker can still use that opportunity to remove +all legitimate snapshots. Customize pruning ***************** diff --git a/doc/conf.py b/doc/conf.py index 97c728ac6..3c0af927b 100644 --- a/doc/conf.py +++ b/doc/conf.py @@ -35,7 +35,7 @@ master_doc = 'index' # General information about the project. project = 'restic' -copyright = '2021, restic authors' +copyright = '2018, restic authors' author = 'fd0' # The version info for the project you're documenting, acts as replacement for diff --git a/doc/design.rst b/doc/design.rst index 196ddaa66..e6a751be2 100644 --- a/doc/design.rst +++ b/doc/design.rst @@ -607,7 +607,7 @@ examples of things an adversary could achieve in various circumstances. An adversary with read access to your backup storage location could: - Attempt a brute force password guessing attack against a copy of the - repository (use strong passwords with sufficient entropy). + repository (please use strong passwords with sufficient entropy). - Infer which packs probably contain trees via file access patterns. - Infer the size of backups by using creation timestamps of repository objects. @@ -618,7 +618,7 @@ An adversary with network access could: - Determine from where you create your backups (i.e., the location where the requests originate). - Determine where you store your backups (i.e., which provider/target system). -- Infer the size of backups by using creation timestamps of repository objects. +- Infer the size of backups by observing network traffic. The following are examples of the implications associated with violating some of the aforementioned assumptions. @@ -629,11 +629,11 @@ system making backups could: - Render the entire backup process untrustworthy (e.g., intercept password, copy files, manipulate data). - Create snapshots (containing garbage data) which cover all modified files - and wait until a trusted host has used forget often enough to forget all + and wait until a trusted host has used ``forget`` often enough to remove all correct snapshots. -- Create a garbage snapshot for every existing snapshot with a slightly different - timestamp and wait until forget has run, thereby removing all correct - snapshots at once. +- Create a garbage snapshot for every existing snapshot with a slightly + different timestamp and wait until certain ``forget`` configurations has been + run, thereby removing all correct snapshots at once. An adversary with write access to your files at the storage location could: @@ -645,29 +645,26 @@ An adversary with write access to your files at the storage location could: the snapshot cannot be restored completely. Restic is not designed to detect this attack. -An adversary who compromises a host system with append-only access to the -backup repository could: +An adversary who compromises a host system with append-only (read+write allowed, +delete+overwrite denied) access to the backup repository could: -- Capture the password and decrypt backups from the past and in the future. - See the "leaked key" circumstance below. +- Capture the password and decrypt backups from the past and in the future + (see the "leaked key" example below for related information). - Render new backups untrustworthy *after* the host has been compromised (due to having complete control over new backups). An attacker cannot delete or manipulate old backups. As such, restoring old snapshots created *before* a host compromise remains possible. -- Potentially manipulate the ``restic forget`` command into deleting all +- Potentially manipulate the use of the ``forget`` command into deleting all legitimate snapshots, keeping only bogus snapshots added by the attacker. Ransomware might try this in order to leave only one option to get your data - back: paying the ransom. For safe use of ``restic forget``, see the - documentation on removing backup snapshots. + back: paying the ransom. For safe use of ``forget``, please see the + corresponding documentation on removing backup snapshots and append-only mode. -An adversary who has a leaked key for a repository could: - -- Decrypt existing and future backup data. If multiple hosts backup into the same - repository, an attacker will get access to the backup data of every host. - Since the local encryption key gives access to the master key, a password - change will not prevent this. Changing the master key can currently be done - using ``restic copy`` which moves the data into a new repository with a new - master key, or by making a completely new repository and new backup. - Re-encrypting all data without creating a new repository is tracked in - :issue:`1602`. +An adversary who has a leaked (decrypted) key for a repository could: +- Decrypt existing and future backup data. If multiple hosts backup into the + same repository, an attacker will get access to the backup data of every host. + Note that since the local encryption key gives access to the master key, a + password change will not prevent this. Changing the master key can currently + be done using the ``copy`` command, which moves the data into a new repository + with a new master key, or by making a completely new repository and new backup. From 80969a634729b8573bd5cb3d9c2e42c7548d8f84 Mon Sep 17 00:00:00 2001 From: Luc Gommans Date: Sun, 20 Feb 2022 21:44:18 +0100 Subject: [PATCH 3/4] Update docs according to comments from MichaelEischer in PR #3656 --- doc/060_forget.rst | 26 +++++++++++++++----------- doc/design.rst | 7 ++++--- 2 files changed, 19 insertions(+), 14 deletions(-) diff --git a/doc/060_forget.rst b/doc/060_forget.rst index 5bf87f024..96c9a61c0 100644 --- a/doc/060_forget.rst +++ b/doc/060_forget.rst @@ -319,9 +319,8 @@ four Sundays, but remove the rest: 8 snapshots The result of the ``forget --keep-daily`` operation only partially depends on -when it is run; it will only count the days for which a snapshot exists, -although snapshots with a `time` lying in the future are ignored and never -removed. This is a safety feature: it prevents restic from removing snapshots +when it is run, counting only the days for which a snapshot exists. +This is a safety feature: it prevents restic from removing snapshots when no new ones are created. Otherwise, running ``forget --keep-daily 4`` on a Friday (without any snapshot Monday to Thursday) would remove all snapshots! @@ -346,6 +345,10 @@ could specify: Security considerations in append-only mode =========================================== +.. note:: TL;DR: append-only repositories should use the ``--keep-within`` + option. This will allow you to notice problems with the backup or the + compromised host during the specified duration. + To prevent a compromised backup client from deleting its backups (for example due to a ransomware infection), a repository service/backend can serve the repository in a so-called append-only mode. This means that the repository can @@ -358,23 +361,24 @@ standard backends do. To support append-only with such a backend, one can use .. _rclone: https://rclone.org/commands/rclone_serve_restic/ To remove snapshots and recover the corresponding disk space, the ``forget`` -and ``prune`` commands must have full read, write and delete access to the +and ``prune`` commands require full read, write and delete access to the repository. If an attacker has this, the protection offered by append-only mode is naturally void. However, even with append-only mode active, an attacker who is able to add -additional and empty or otherwise useless snapshots to the repository can -potentially cause a situation where a trusted client running ``forget`` with -certain ``--keep-*`` options might unknowingly remove legitimate snapshots, -leaving only the attackers useless snapshots. +garbage snapshots to the repository could bring the snapshot list into a state +where all legitimate snapshots are deleted by a client with full access. By +running ``forget`` with certain ``--keep-*`` options as repository +administrator, legitimate snapshots might be unknowingly removed, leaving only +the attacker's useless snapshots. For example, if the ``forget`` policy is to keep three weekly snapshots, and the attacker adds an empty snapshot for each of the last three weeks, all with -a timestamp (see the ``backup`` command's ``-`time`` option) slightly more +a timestamp (see the ``backup`` command's ``--time`` option) slightly more recent than the existing snapshots (but still within the target week), then the -next time the repository administrator (or scheduled job) runs the ``forget`` +next time the repository administrator (or a scheduled job) runs the ``forget`` command with this policy, the legitimate snapshots will be removed (since the -policy will use the most recent snapshot within each week). Even without +policy will keep only the most recent snapshot within each week). Even without running ``prune``, recovering data would be messy and some metadata lost. To avoid this, ``forget`` policies applied to append-only repositories should diff --git a/doc/design.rst b/doc/design.rst index e6a751be2..11c304684 100644 --- a/doc/design.rst +++ b/doc/design.rst @@ -632,7 +632,7 @@ system making backups could: and wait until a trusted host has used ``forget`` often enough to remove all correct snapshots. - Create a garbage snapshot for every existing snapshot with a slightly - different timestamp and wait until certain ``forget`` configurations has been + different timestamp and wait until certain ``forget`` configurations have been run, thereby removing all correct snapshots at once. An adversary with write access to your files at the storage location could: @@ -666,5 +666,6 @@ An adversary who has a leaked (decrypted) key for a repository could: same repository, an attacker will get access to the backup data of every host. Note that since the local encryption key gives access to the master key, a password change will not prevent this. Changing the master key can currently - be done using the ``copy`` command, which moves the data into a new repository - with a new master key, or by making a completely new repository and new backup. + only be done using the ``copy`` command, which moves the data into a new + repository with a new master key, or by making a completely new repository + and new backup. From cdf478c8f46a742b73278926dabc8db144da47c5 Mon Sep 17 00:00:00 2001 From: "Leo R. Lundgren" Date: Wed, 23 Mar 2022 23:11:16 +0100 Subject: [PATCH 4/4] doc: More updates to forget documentation and security considerations --- doc/060_forget.rst | 180 ++++++++++++++++++++++----------------------- 1 file changed, 88 insertions(+), 92 deletions(-) diff --git a/doc/060_forget.rst b/doc/060_forget.rst index 96c9a61c0..259371610 100644 --- a/doc/060_forget.rst +++ b/doc/060_forget.rst @@ -155,74 +155,75 @@ to ``forget``: Removing snapshots according to a policy **************************************** -Removing snapshots manually is tedious and error-prone, therefore restic -allows specifying which snapshots should be removed automatically -according to a policy. You can specify how many hourly, daily, weekly, -monthly and yearly snapshots to keep, any other snapshots are removed. -The most important command-line parameter here is ``--dry-run`` which -instructs restic to not remove anything but print which snapshots would -be removed. +Removing snapshots manually is tedious and error-prone, therefore restic allows +specifying a policy (one or more ``--keep-*`` options) for which snapshots to +keep. You can for example define how many hourly, daily, weekly, monthly and +yearly snapshots to keep, and any other snapshots will be removed. -.. note:: If you use an append-only repository, some best practices apply. - Please refer to the security considerations below for more information. +.. warning:: If you use an append-only repository with policy-based snapshot + removal, some security considerations are important. Please refer to the + section below for more information. -When ``forget`` is run with a policy, restic loads the list of all -snapshots, then groups these by host name and list of directories. The grouping -options can be set with ``--group-by``, to only group snapshots by paths and -tags use ``--group-by paths,tags``. The policy is then applied to each group of -snapshots separately. This is a safety feature. +.. note:: You can always use the ``--dry-run`` option of the ``forget`` command, + which instructs restic to not remove anything but instead just print what + actions would be performed. -The ``forget`` command accepts the following parameters: +The ``forget`` command accepts the following policy options: - ``--keep-last n`` keep the ``n`` last (most recent) snapshots. - ``--keep-hourly n`` for the last ``n`` hours which have one or more snapshots, keep only the most recent one for each hour. - ``--keep-daily n`` for the last ``n`` days which have one or more - snapshots, keep only the most recent one for that day. + snapshots, keep only the most recent one for each day. - ``--keep-weekly n`` for the last ``n`` weeks which have one or more - snapshots, keep only the most recent one for that week. + snapshots, keep only the most recent one for each week. - ``--keep-monthly n`` for the last ``n`` months which have one or more - snapshots, keep only the most recent one for that month. + snapshots, keep only the most recent one for each month. - ``--keep-yearly n`` for the last ``n`` years which have one or more - snapshots, keep only the most recent one for that year. + snapshots, keep only the most recent one for each year. - ``--keep-tag`` keep all snapshots which have all tags specified by this option (can be specified multiple times). - ``--keep-within duration`` keep all snapshots having a timestamp within the specified duration of the latest snapshot, where ``duration`` is a number of years, months, days, and hours. E.g. ``2y5m7d3h`` will keep all - snapshots made in the two years, five months, seven days, and three hours + snapshots made in the two years, five months, seven days and three hours before the latest (most recent) snapshot. -- ``--keep-within-hourly duration`` keep all hourly snapshots made within - the specified duration of the latest snapshot. The ``duration`` is specified - in the same way as for ``--keep-within`` and the method for determining - hourly snapshots is the same as for ``--keep-hourly``. -- ``--keep-within-daily duration`` keep all daily snapshots made within +- ``--keep-within-hourly duration`` keep all hourly snapshots made within the + specified duration of the latest snapshot. The ``duration`` is specified in + the same way as for ``--keep-within`` and the method for determining hourly + snapshots is the same as for ``--keep-hourly``. +- ``--keep-within-daily duration`` keep all daily snapshots made within the specified duration of the latest snapshot. -- ``--keep-within-weekly duration`` keep all weekly snapshots made within +- ``--keep-within-weekly duration`` keep all weekly snapshots made within the specified duration of the latest snapshot. -- ``--keep-within-monthly duration`` keep all monthly snapshots made within +- ``--keep-within-monthly duration`` keep all monthly snapshots made within the specified duration of the latest snapshot. -- ``--keep-within-yearly duration`` keep all yearly snapshots made within +- ``--keep-within-yearly duration`` keep all yearly snapshots made within the specified duration of the latest snapshot. -.. note:: All calendar related options (``--keep-*``) work on the natural time - boundaries and not relative to when you run the ``forget`` command. Weeks +.. note:: All calendar related options (``--keep-{hourly,daily,...}``) work on + natural time boundaries and *not* relative to when you run ``forget``. Weeks are Monday 00:00 to Sunday 23:59, days 00:00 to 23:59, hours :00 to :59, etc. + They also only count hours/days/weeks/etc which have one or more snapshots. -.. note:: All duration related options (``--keep-within`` and ``--keep-within-*``) - ignore snapshots with a timestamp in the future (relative to when the - ``forget`` command is run) and these snapshots will hence not be removed. +.. note:: All duration related options (``--keep-{within,-*}``) ignore snapshots + with a timestamp in the future (relative to when the ``forget`` command is + run) and these snapshots will hence not be removed. .. note:: Specifying ``--keep-tag ''`` will match untagged snapshots only. -Multiple policies will be ORed together so as to be as inclusive as possible -for keeping snapshots. +When ``forget`` is run with a policy, restic loads the list of all snapshots, +then groups these by host name and list of directories. The grouping options can +be set with ``--group-by``, to e.g. group snapshots by only paths and tags use +``--group-by paths,tags``. The policy is then applied to each group of snapshots +separately. This is a safety feature to prevent accidental removal of unrelated +backup sets. -Additionally, you can restrict removing snapshots to those which have a -particular hostname with the ``--host`` parameter, or tags with the -``--tag`` option. When multiple tags are specified, only the snapshots -which have all the tags are considered. For example, the following command -removes all but the latest snapshot of all snapshots that have the tag ``foo``: +Additionally, you can restrict the policy to only process snapshots which have a +particular hostname with the ``--host`` parameter, or tags with the ``--tag`` +option. When multiple tags are specified, only the snapshots which have all the +tags are considered. For example, the following command removes all but the +latest snapshot of all snapshots that have the tag ``foo``: .. code-block:: console @@ -249,21 +250,8 @@ the tag. $ restic forget --tag '' --keep-last 1 -All the ``--keep-*`` options above only count -hours/days/weeks/months/years which have a snapshot, so those without a -snapshot are ignored. - -For safety reasons, restic refuses to act on an "empty" policy. For example, -if one were to specify ``--keep-last 0`` to forget *all* snapshots in the -repository, restic will respond that no snapshots will be removed. To delete -all snapshots, use ``--keep-last 1`` and then finally remove the last -snapshot ID manually (by passing the ID to ``forget``). - -All snapshots are evaluated against all matching ``--keep-*`` counts. A -single snapshot on 2017-09-30 (Sat) will count as a daily, weekly and monthly. - -Let's explain this with an example: Suppose you have only made a backup -on each Sunday for 12 weeks: +Let's look at a simple example: Suppose you have only made one backup every +Sunday for 12 weeks: .. code-block:: console @@ -286,8 +274,8 @@ on each Sunday for 12 weeks: --------------------------------------------------------------- 12 snapshots -Then ``forget --keep-daily 4`` will keep the last four snapshots for the last -four Sundays, but remove the rest: +Then ``forget --keep-daily 4`` will keep the last four snapshots, for the last +four Sundays, and remove the other snapshots: .. code-block:: console @@ -318,44 +306,49 @@ four Sundays, but remove the rest: --------------------------------------------------------------- 8 snapshots -The result of the ``forget --keep-daily`` operation only partially depends on -when it is run, counting only the days for which a snapshot exists. -This is a safety feature: it prevents restic from removing snapshots -when no new ones are created. Otherwise, running ``forget --keep-daily 4`` on -a Friday (without any snapshot Monday to Thursday) would remove all snapshots! +The processed snapshots are evaluated against all ``--keep-*`` options but a +snapshot only need to match a single option to be kept (the results are ORed). +This means that the most recent snapshot on a Sunday would match both hourly, +daily and weekly ``--keep-*`` options, and possibly more depending on calendar. -Another example: Suppose you make daily backups for 100 years. Then -``forget --keep-daily 7 --keep-weekly 5 --keep-monthly 12 --keep-yearly 75`` -will keep the most recent 7 daily snapshots, then 4 (remember, 7 dailies -already include a week!) last-day-of-the-weeks and 11 or 12 -last-day-of-the-months (11 or 12 depends if the 5 weeklies cross a month). -And finally 75 last-day-of-the-year snapshots. All other snapshots are -removed. +For example, suppose you make one backup every day for 100 years. Then ``forget +--keep-daily 7 --keep-weekly 5 --keep-monthly 12 --keep-yearly 75`` would keep +the most recent 7 daily snapshots and 4 last-day-of-the-week ones (since the 7 +dailies already include 1 weekly). Additionally, 12 or 11 last-day-of-the-month +snapshots will be kept (depending on whether one of them ends up being the same +as a daily or weekly). And finally 75 or 74 last-day-of-the-year snapshots are +kept, depending on whether one of them ends up being the same as an already kept +snapshot. All other snapshots are removed. -You might want to maintain the same policy as for the example above, but have +You might want to maintain the same policy as in the example above, but have irregular backups. For example, the 7 snapshots specified with ``--keep-daily 7`` -might be spread over a longer period. If what you want is to keep daily snapshots -for a week, weekly for a month, monthly for a year and yearly for 75 years, you -could specify: -``forget --keep-within-daily 7d --keep-within-weekly 1m --keep-within-monthly 1y ---keep-within-yearly 75y`` -(Note that `1w` is not a recognized duration, so you will have to specify -`7d` instead) +might be spread over a longer period. If what you want is to keep daily +snapshots for the last week, weekly for the last month, monthly for the last +year and yearly for the last 75 years, you can instead specify ``forget +--keep-within-daily 7d --keep-within-weekly 1m --keep-within-monthly 1y +--keep-within-yearly 75y`` (note that `1w` is not a recognized duration, so +you will have to specify `7d` instead). + +For safety reasons, restic refuses to act on an "empty" policy. For example, +if one were to specify ``--keep-last 0`` to forget *all* snapshots in the +repository, restic will respond that no snapshots will be removed. To delete +all snapshots, use ``--keep-last 1`` and then finally remove the last snapshot +manually (by passing the ID to ``forget``). Security considerations in append-only mode =========================================== -.. note:: TL;DR: append-only repositories should use the ``--keep-within`` - option. This will allow you to notice problems with the backup or the - compromised host during the specified duration. +.. note:: TL;DR: With append-only repositories, one should specifically use the + ``--keep-within`` option of the ``forget`` command when removing snapshots. To prevent a compromised backup client from deleting its backups (for example due to a ransomware infection), a repository service/backend can serve the -repository in a so-called append-only mode. This means that the repository can -only be written to and read from, while delete and overwrite operations are -denied. Restic's `rest-server`_ features the append-only mode, but few other -standard backends do. To support append-only with such a backend, one can use -`rclone`_ as a complement in between the backup client and the backend service. +repository in a so-called append-only mode. This means that the repository is +served in such a way that it can only be written to and read from, while delete +and overwrite operations are denied. Restic's `rest-server`_ features an +append-only mode, but few other standard backends do. To support append-only +with such backends, one can use `rclone`_ as a complement in between the backup +client and the backend service. .. _rest-server: https://github.com/restic/rest-server/ .. _rclone: https://rclone.org/commands/rclone_serve_restic/ @@ -363,14 +356,17 @@ standard backends do. To support append-only with such a backend, one can use To remove snapshots and recover the corresponding disk space, the ``forget`` and ``prune`` commands require full read, write and delete access to the repository. If an attacker has this, the protection offered by append-only -mode is naturally void. +mode is naturally void. The usual and recommended setup with append-only +repositories is therefore to use a separate and well-secured client whenever +full access to the repository is needed, e.g. for administrative tasks such +as running ``forget``, ``prune`` and other maintenance commands. -However, even with append-only mode active, an attacker who is able to add -garbage snapshots to the repository could bring the snapshot list into a state -where all legitimate snapshots are deleted by a client with full access. By -running ``forget`` with certain ``--keep-*`` options as repository -administrator, legitimate snapshots might be unknowingly removed, leaving only -the attacker's useless snapshots. +However, even with append-only mode active and a separate, well-secured client +used for administrative tasks, an attacker who is able to add garbage snapshots +to the repository could bring the snapshot list into a state where all the +legitimate snapshots risk being deleted by an unsuspecting administrator that +runs the ``forget`` command with certain ``--keep-*`` options, leaving only the +attacker's useless snapshots. For example, if the ``forget`` policy is to keep three weekly snapshots, and the attacker adds an empty snapshot for each of the last three weeks, all with