Generate CHANGELOG.md for 0.14.0

2022-08-25 19:54:02 +02:00 · 2022-08-25 19:54:02 +02:00 · 193c62dfc3
commit 193c62dfc3
parent a825e0d409
1 changed files with 450 additions and 0 deletions
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@ -1,3 +1,453 @@
 Changelog for restic 0.14.0 (2022-08-25)
 =======================================
 The following sections list the changes in restic 0.14.0 relevant to
 restic users. The changes are ordered by importance.
 Summary
 -------
 * Fix #2248: Support `self-update` on Windows
 * Fix #3428: List snapshots in backend at most once to resolve snapshot IDs
 * Fix #3432: Fix rare 'not found in repository' error for `copy` command
 * Fix #3685: The `diff` command incorrectly listed some files as added
 * Fix #3681: Fix rclone (shimmed by Scoop) and sftp not working on Windows
 * Fix #3720: Directory sync errors for repositories accessed via SMB
 * Fix #3736: The `stats` command miscalculated restore size for multiple snapshots
 * Fix #3861: Yield error on invalid policy to `forget`
 * Fix #3716: Print "wrong password" to stderr instead of stdout
 * Fix #3772: Correctly rebuild index for legacy repositories
 * Fix #3776: Limit number of key files tested while opening a repository
 * Chg #1842: Support debug log creation in release builds
 * Chg #3295: Deprecate `check --check-unused` and add further checks
 * Chg #3680: Update dependencies and require Go 1.15 or newer
 * Chg #3742: Replace `--repo2` option used by `init`/`copy` with `--from-repo`
 * Enh #1153: Support pruning even when the disk is full
 * Enh #21: Add compression support
 * Enh #2162: Adaptive IO concurrency based on backend connections
 * Enh #2291: Allow pack size customization
 * Enh #2295: Allow use of SAS token to authenticate to Azure
 * Enh #2696: Improve backup speed with many small files
 * Enh #2907: Make snapshot directory structure of `mount` command customizable
 * Enh #3114: Optimize handling of duplicate blobs in `prune`
 * Enh #3465: Improve handling of temporary files on Windows
 * Enh #3709: Validate exclude patterns before backing up
 * Enh #3837: Improve SFTP repository initialization over slow links
 * Enh #2351: Use config file permissions to control file group access
 * Enh #3475: Allow limiting IO concurrency for local and SFTP backend
 * Enh #3484: Stream data in `check` and `prune` commands
 * Enh #2923: Improve speed of `copy` command
 * Enh #3729: Display full IDs in `check` warnings
 * Enh #3773: Optimize memory usage for directories with many files
 * Enh #3819: Validate include/exclude patterns before restoring
 Details
 -------
 * Bugfix #2248: Support `self-update` on Windows
   Restic `self-update` would fail in situations where the operating system locks running
   binaries, including Windows. The new behavior works around this by renaming the running file
   and swapping the updated file in place.
   https://github.com/restic/restic/issues/2248
   https://github.com/restic/restic/pull/3675
 * Bugfix #3428: List snapshots in backend at most once to resolve snapshot IDs
   Many commands support specifying a list of snapshot IDs which are then used to determine the
   snapshots to be processed by the command. To resolve snapshot IDs or `latest`, and check that
   these exist, restic previously listed all snapshots stored in the repository. Depending on
   the backend this could be a slow and/or expensive operation.
   Restic now lists the snapshots only once and remembers the result in order to resolve all
   further snapshot IDs swiftly.
   https://github.com/restic/restic/issues/3428
   https://github.com/restic/restic/pull/3570
   https://github.com/restic/restic/pull/3395
 * Bugfix #3432: Fix rare 'not found in repository' error for `copy` command
   In rare cases `copy` (and other commands) would report that `LoadTree(...)` returned an `id
   [...] not found in repository` error. This could be caused by a backup or copy command running
   concurrently. The error was only temporary; running the failed restic command a second time as
   a workaround did resolve the error.
   This issue has now been fixed by correcting the order in which restic reads data from the
   repository. It is now guaranteed that restic only loads snapshots for which all necessary data
   is already available.
   https://github.com/restic/restic/issues/3432
   https://github.com/restic/restic/pull/3570
 * Bugfix #3685: The `diff` command incorrectly listed some files as added
   There was a bug in the `diff` command, causing it to always show files in a removed directory as
   added. This has now been fixed.
   https://github.com/restic/restic/issues/3685
   https://github.com/restic/restic/pull/3686
 * Bugfix #3681: Fix rclone (shimmed by Scoop) and sftp not working on Windows
   In #3602 a fix was introduced to address the problem of `rclone` prematurely exiting when
   Ctrl+C is pressed on Windows. The solution was to create the subprocess with its console
   detached from the restic console.
   However, this solution failed when using `rclone` installed by Scoop or using `sftp` with a
   passphrase-protected private key. We've now fixed this by using a different approach to
   prevent Ctrl-C from passing down too early.
   https://github.com/restic/restic/issues/3681
   https://github.com/restic/restic/issues/3692
   https://github.com/restic/restic/pull/3696
 * Bugfix #3720: Directory sync errors for repositories accessed via SMB
   On Linux and macOS, accessing a repository via a SMB/CIFS mount resulted in restic failing to
   save the lock file, yielding the following errors:
   Save(<lock/071fe833f0>) returned error, retrying after 552.330144ms: sync /repo/locks:
   no such file or directory Save(<lock/bf789d7343>) returned error, retrying after
   552.330144ms: sync /repo/locks: invalid argument
   This has now been fixed by ignoring the relevant error codes.
   https://github.com/restic/restic/issues/3720
   https://github.com/restic/restic/issues/3751
   https://github.com/restic/restic/pull/3752
 * Bugfix #3736: The `stats` command miscalculated restore size for multiple snapshots
   Since restic 0.10.0 the restore size calculated by the `stats` command for multiple snapshots
   was too low. The hardlink detection was accidentally applied across multiple snapshots and
   thus ignored many files. This has now been fixed.
   https://github.com/restic/restic/issues/3736
   https://github.com/restic/restic/pull/3740
 * Bugfix #3861: Yield error on invalid policy to `forget`
   The `forget` command previously silently ignored invalid/unsupported units in the duration
   options, such as e.g. `--keep-within-daily 2w`.
   Specifying an invalid/unsupported duration unit now results in an error.
   https://github.com/restic/restic/issues/3861
   https://github.com/restic/restic/pull/3862
 * Bugfix #3716: Print "wrong password" to stderr instead of stdout
   If an invalid password was entered, the error message was printed on stdout and not on stderr as
   intended. This has now been fixed.
   https://github.com/restic/restic/pull/3716
   https://forum.restic.net/t/4965
 * Bugfix #3772: Correctly rebuild index for legacy repositories
   After running `rebuild-index` on a legacy repository containing mixed pack files (that is,
   pack files which store both metadata and file data), `check` printed warnings like `pack
   12345678 contained in several indexes: ...`. This warning was not critical, but has now
   nonetheless been fixed by properly handling mixed pack files while rebuilding the index.
   Running `prune` for such legacy repositories will also fix the warning by reorganizing the
   pack files which caused it.
   https://github.com/restic/restic/pull/3772
   https://github.com/restic/restic/pull/3884
   https://forum.restic.net/t/5044/13
 * Bugfix #3776: Limit number of key files tested while opening a repository
   Previously, restic tested the password against every key in the repository when opening a
   repository. The more keys there were in the repository, the slower this operation became.
   Restic now tests the password against up to 20 key files in the repository. Alternatively, you
   can use the `--key-hint=<key ID>` option to specify a specific key file to use instead.
   https://github.com/restic/restic/pull/3776
 * Change #1842: Support debug log creation in release builds
   Creating a debug log was only possible in debug builds which required users to manually build
   restic. We changed the release builds to allow creating debug logs by simply setting the
   environment variable `DEBUG_LOG=logname.log`.
   https://github.com/restic/restic/issues/1842
   https://github.com/restic/restic/pull/3826
 * Change #3295: Deprecate `check --check-unused` and add further checks
   Since restic 0.12.0, it is expected to still have unused blobs after running `prune`. This made
   the `--check-unused` option of the `check` command rather useless and tended to confuse
   users. This option has been deprecated and is now ignored.
   The `check` command now also warns if a repository is using either the legacy S3 layout or mixed
   pack files with both tree and data blobs. The latter is known to cause performance problems.
   https://github.com/restic/restic/issues/3295
   https://github.com/restic/restic/pull/3730
 * Change #3680: Update dependencies and require Go 1.15 or newer
   We've updated most dependencies. Since some libraries require newer language features we're
   dropping support for Go 1.14, which means that restic now requires at least Go 1.15 to build.
   https://github.com/restic/restic/issues/3680
   https://github.com/restic/restic/issues/3883
 * Change #3742: Replace `--repo2` option used by `init`/`copy` with `--from-repo`
   The `init` and `copy` commands can read data from another repository. However, confusingly
   `--repo2` referred to the repository *from* which the `init` command copies parameters, but
   for the `copy` command `--repo2` referred to the copy *destination*.
   We've introduced a new option, `--from-repo`, which always refers to the source repository
   for both commands. The old parameter names have been deprecated but still work. To create a new
   repository and copy all snapshots to it, the commands are now as follows:
   ``` restic -r /srv/restic-repo-copy init --from-repo /srv/restic-repo
   --copy-chunker-params restic -r /srv/restic-repo-copy copy --from-repo
   /srv/restic-repo ```
   https://github.com/restic/restic/pull/3742
   https://forum.restic.net/t/5017
 * Enhancement #1153: Support pruning even when the disk is full
   When running out of disk space it was no longer possible to add or remove data from a repository.
   To help with recovering from such a deadlock, the prune command now supports an
   `--unsafe-recover-no-free-space` option to recover from these situations. Make sure to
   read the documentation first!
   https://github.com/restic/restic/issues/1153
   https://github.com/restic/restic/pull/3481
 * Enhancement #21: Add compression support
   We've added compression support to the restic repository format. To create a repository using
   the new format run `init --repository-version 2`. Please note that the repository cannot be
   read by restic versions prior to 0.14.0.
   You can configure whether data is compressed with the option `--compression`. It can be set to
   `auto` (the default, which will compress very fast), `max` (which will trade backup speed and
   CPU usage for better compression), or `off` (which disables compression). Each setting is
   only applied for the current run of restic and does *not* apply to future runs. The option can
   also be set via the environment variable `RESTIC_COMPRESSION`.
   To upgrade in place run `migrate upgrade_repo_v2` followed by `prune`. See the documentation
   for more details. The migration checks the repository integrity and upgrades the repository
   format, but will not change any data. Afterwards, prune will rewrite the metadata to make use of
   compression.
   As an alternative you can use the `copy` command to migrate snapshots; First create a new
   repository using `init --repository-version 2 --copy-chunker-params --repo2
   path/to/old/repo`, and then use the `copy` command to copy all snapshots to the new
   repository.
   https://github.com/restic/restic/issues/21
   https://github.com/restic/restic/issues/3779
   https://github.com/restic/restic/pull/3666
   https://github.com/restic/restic/pull/3704
   https://github.com/restic/restic/pull/3733
 * Enhancement #2162: Adaptive IO concurrency based on backend connections
   Many commands used hard-coded limits for the number of concurrent operations. This prevented
   speed improvements by increasing the number of connections used by a backend.
   These limits have now been replaced by using the configured number of backend connections
   instead, which can be controlled using the `-o <backend-name>.connections=5` option.
   Commands will then automatically scale their parallelism accordingly.
   To limit the number of CPU cores used by restic, you can set the environment variable
   `GOMAXPROCS` accordingly. For example to use a single CPU core, use `GOMAXPROCS=1`.
   https://github.com/restic/restic/issues/2162
   https://github.com/restic/restic/issues/1467
   https://github.com/restic/restic/pull/3611
 * Enhancement #2291: Allow pack size customization
   Restic now uses a target pack size of 16 MiB by default. This can be customized using the
   `--pack-size size` option. Supported pack sizes range between 4 and 128 MiB.
   It is possible to migrate an existing repository to _larger_ pack files using `prune
   --repack-small`. This will rewrite every pack file which is significantly smaller than the
   target size.
   https://github.com/restic/restic/issues/2291
   https://github.com/restic/restic/pull/3731
 * Enhancement #2295: Allow use of SAS token to authenticate to Azure
   Previously restic only supported AccountKeys to authenticate to Azure storage accounts,
   which necessitates giving a significant amount of access.
   We added support for Azure SAS tokens which are a more fine-grained and time-limited manner of
   granting access. Set the `AZURE_ACCOUNT_NAME` and `AZURE_ACCOUNT_SAS` environment
   variables to use a SAS token for authentication. Note that if `AZURE_ACCOUNT_KEY` is set, it
   will take precedence.
   https://github.com/restic/restic/issues/2295
   https://github.com/restic/restic/pull/3661
 * Enhancement #2696: Improve backup speed with many small files
   We have restructured the backup pipeline to continue reading files while all upload
   connections are busy. This allows the backup to already prepare the next data file such that the
   upload can continue as soon as a connection becomes available. This can especially improve the
   backup performance for high latency backends.
   The upload concurrency is now controlled using the `-o <backend-name>.connections=5`
   option.
   https://github.com/restic/restic/issues/2696
   https://github.com/restic/restic/pull/3489
 * Enhancement #2907: Make snapshot directory structure of `mount` command customizable
   We've added the possibility to customize the snapshot directory structure of the `mount`
   command using templates passed to the `--snapshot-template` option. The formatting of
   snapshots' timestamps is now controlled using `--time-template` and supports
   subdirectories to for example group snapshots by year. Please see `restic help mount` for
   further details.
   Characters in tag names which are not allowed in a filename are replaced by underscores `_`. For
   example a tag `foo/bar` will result in a directory name of `foo_bar`.
   https://github.com/restic/restic/issues/2907
   https://github.com/restic/restic/pull/2913
   https://github.com/restic/restic/pull/3691
 * Enhancement #3114: Optimize handling of duplicate blobs in `prune`
   Restic `prune` always used to repack all data files containing duplicate blobs. This
   effectively removed all duplicates during prune. However, as a consequence all these data
   files were repacked even if the unused repository space threshold could be reached with less
   work.
   This is now changed and `prune` works nice and fast even when there are lots of duplicate blobs.
   https://github.com/restic/restic/issues/3114
   https://github.com/restic/restic/pull/3290
 * Enhancement #3465: Improve handling of temporary files on Windows
   In some cases restic failed to delete temporary files, causing the current command to fail.
   This has now been fixed by ensuring that Windows automatically deletes the file. In addition,
   temporary files are only written to disk when necessary, reducing disk writes.
   https://github.com/restic/restic/issues/3465
   https://github.com/restic/restic/issues/1551
   https://github.com/restic/restic/pull/3610
 * Enhancement #3709: Validate exclude patterns before backing up
   Exclude patterns provided via `--exclude`, `--iexclude`, `--exclude-file` or
   `--iexclude-file` previously weren't validated. As a consequence, invalid patterns
   resulted in files that were meant to be excluded being backed up.
   Restic now validates all patterns before running the backup and aborts with a fatal error if an
   invalid pattern is detected.
   https://github.com/restic/restic/issues/3709
   https://github.com/restic/restic/pull/3734
 * Enhancement #3837: Improve SFTP repository initialization over slow links
   The `init` command, when used on an SFTP backend, now sends multiple `mkdir` commands to the
   backend concurrently. This reduces the waiting times when creating a repository over a very
   slow connection.
   https://github.com/restic/restic/issues/3837
   https://github.com/restic/restic/pull/3840
 * Enhancement #2351: Use config file permissions to control file group access
   Previously files in a local/SFTP repository would always end up with very restrictive access
   permissions, allowing access only to the owner. This prevented a number of valid use-cases
   involving groups and ACLs.
   We now use the permissions of the config file in the repository to decide whether group access
   should be given to newly created repository files or not. We arrange for repository files to be
   created group readable exactly when the repository config file is group readable.
   To opt-in to group readable repositories, a simple `chmod -R g+r` or equivalent on the config
   file can be used. For repositories that should be writable by group members a tad more setup is
   required, see the docs.
   Posix ACLs can also be used now that the group permissions being forced to zero no longer masks
   the effect of ACL entries.
   https://github.com/restic/restic/issues/2351
   https://github.com/restic/restic/pull/3419
   https://forum.restic.net/t/1391
 * Enhancement #3475: Allow limiting IO concurrency for local and SFTP backend
   Restic did not support limiting the IO concurrency / number of connections for accessing
   repositories stored using the local or SFTP backends. The number of connections is now limited
   as for other backends, and can be configured via the the `-o local.connections=2` and `-o
   sftp.connections=5` options. This ensures that restic does not overwhelm the backend with
   concurrent IO operations.
   https://github.com/restic/restic/pull/3475
 * Enhancement #3484: Stream data in `check` and `prune` commands
   The commands `check --read-data` and `prune` previously downloaded data files into
   temporary files which could end up being written to disk. This could cause a large amount of data
   being written to disk.
   The pack files are now instead streamed, which removes the need for temporary files. Please
   note that *uploads* during `backup` and `prune` still require temporary files.
   https://github.com/restic/restic/issues/3710
   https://github.com/restic/restic/pull/3484
   https://github.com/restic/restic/pull/3717
 * Enhancement #2923: Improve speed of `copy` command
   The `copy` command could require a long time to copy snapshots for non-local backends. This has
   been improved to provide a throughput comparable to the `restore` command.
   Additionally, `copy` now displays a progress bar.
   https://github.com/restic/restic/issues/2923
   https://github.com/restic/restic/pull/3513
 * Enhancement #3729: Display full IDs in `check` warnings
   When running commands to inspect or repair a damaged repository, it is often necessary to
   supply the full IDs of objects stored in the repository.
   The output of `check` now includes full IDs instead of their shortened variant.
   https://github.com/restic/restic/pull/3729
 * Enhancement #3773: Optimize memory usage for directories with many files
   Backing up a directory with hundreds of thousands or more files caused restic to require large
   amounts of memory. We've now optimized the `backup` command such that it requires up to 30% less
   memory.
   https://github.com/restic/restic/pull/3773
 * Enhancement #3819: Validate include/exclude patterns before restoring
   Patterns provided to `restore` via `--exclude`, `--iexclude`, `--include` and
   `--iinclude` weren't validated before running the restore. Invalid patterns would result in
   error messages being printed repeatedly, and possibly unwanted files being restored.
   Restic now validates all patterns before running the restore, and aborts with a fatal error if
   an invalid pattern is detected.
   https://github.com/restic/restic/pull/3819
 Changelog for restic 0.13.0 (2022-03-26)
 =======================================