Generate CHANGELOG.md for 0.13.0

CHANGELOG.md

@@ -1,3 +1,408 @@
+Changelog for restic 0.13.0 (2022-03-26)
+=======================================
+
+The following sections list the changes in restic 0.13.0 relevant to
+restic users. The changes are ordered by importance.
+
+Summary
+-------
+
+ * Fix #1106: Never lock repository for `list locks`
+ * Fix #2345: Make cache crash-resistant and usable by multiple concurrent processes
+ * Fix #2452: Improve error handling of repository locking
+ * Fix #2738: Don't print progress for `backup --json --quiet`
+ * Fix #3382: Make `check` command honor `RESTIC_CACHE_DIR` environment variable
+ * Fix #3518: Make `copy` command honor `--no-lock` for source repository
+ * Fix #3556: Fix hang with Backblaze B2 on SSL certificate authority error
+ * Fix #3601: Fix rclone backend prematurely exiting when receiving SIGINT on Windows
+ * Fix #3667: The `mount` command now reports symlinks sizes
+ * Fix #3488: `rebuild-index` failed if an index file was damaged
+ * Fix #3591: Fix handling of `prune --max-repack-size=0`
+ * Fix #3619: Avoid choosing parent snapshots newer than time of new snapshot
+ * Chg #3641: Ignore parent snapshot for `backup --stdin`
+ * Chg #3519: Require Go 1.14 or newer
+ * Enh #1542: Add `--dry-run`/`-n` option to `backup` command
+ * Enh #2202: Add upload checksum for Azure, GS, S3 and Swift backends
+ * Enh #233: Support negative include/exclude patterns
+ * Enh #2388: Add warning for S3 if partial credentials are provided
+ * Enh #2508: Support JSON output and quiet mode for the `diff` command
+ * Enh #2656: Add flag to disable TLS verification for self-signed certificates
+ * Enh #3003: Atomic uploads for the SFTP backend
+ * Enh #3127: Add xattr (extended attributes) support for Solaris
+ * Enh #3464: Skip lock creation on `forget` if `--no-lock` and `--dry-run`
+ * Enh #3490: Support random subset by size in `check --read-data-subset`
+ * Enh #3541: Improve handling of temporary B2 delete errors
+ * Enh #3542: Add file mode in symbolic notation to `ls --json`
+ * Enh #2594: Speed up the `restore --verify` command
+ * Enh #2816: The `backup` command no longer updates file access times on Linux
+ * Enh #2880: Make `recover` collect only unreferenced trees
+ * Enh #3429: Verify that new or modified keys are stored correctly
+ * Enh #3436: Improve local backend's resilience to (system) crashes
+ * Enh #3508: Cache blobs read by the `dump` command
+ * Enh #3511: Support configurable timeout for the rclone backend
+ * Enh #3593: Improve `copy` performance by parallelizing IO
+
+Details
+-------
+
+ * Bugfix #1106: Never lock repository for `list locks`
+
+   The `list locks` command previously locked to the repository by default. This had the problem
+   that it wouldn't work for an exclusively locked repository and that the command would also
+   display its own lock file which can be confusing.
+
+   Now, the `list locks` command never locks the repository.
+
+   https://github.com/restic/restic/issues/1106
+   https://github.com/restic/restic/pull/3665
+
+ * Bugfix #2345: Make cache crash-resistant and usable by multiple concurrent processes
+
+   The restic cache directory (`RESTIC_CACHE_DIR`) could end up in a broken state in the event of
+   restic (or the OS) crashing. This is now less likely to occur as files are downloaded to a
+   temporary location before being moved to their proper location.
+
+   This also allows multiple concurrent restic processes to operate on a single repository
+   without conflicts. Previously, concurrent operations could cause segfaults because the
+   processes saw each other's partially downloaded files.
+
+   https://github.com/restic/restic/issues/2345
+   https://github.com/restic/restic/pull/2838
+
+ * Bugfix #2452: Improve error handling of repository locking
+
+   Previously, when the lock refresh failed to delete the old lock file, it forgot about the newly
+   created one. Instead it continued trying to delete the old (usually no longer existing) lock
+   file and thus over time lots of lock files accumulated. This has now been fixed.
+
+   https://github.com/restic/restic/issues/2452
+   https://github.com/restic/restic/issues/2473
+   https://github.com/restic/restic/issues/2562
+   https://github.com/restic/restic/pull/3512
+
+ * Bugfix #2738: Don't print progress for `backup --json --quiet`
+
+   Unlike the text output, the `--json` output format still printed progress information even in
+   `--quiet` mode. This has now been fixed by always disabling the progress output in quiet mode.
+
+   https://github.com/restic/restic/issues/2738
+   https://github.com/restic/restic/pull/3264
+
+ * Bugfix #3382: Make `check` command honor `RESTIC_CACHE_DIR` environment variable
+
+   Previously, the `check` command didn't honor the `RESTIC_CACHE_DIR` environment variable,
+   which caused problems in certain system/usage configurations. This has now been fixed.
+
+   https://github.com/restic/restic/issues/3382
+   https://github.com/restic/restic/pull/3474
+
+ * Bugfix #3518: Make `copy` command honor `--no-lock` for source repository
+
+   The `copy` command previously did not respect the `--no-lock` option for the source
+   repository, causing failures with read-only storage backends. This has now been fixed such
+   that the option is now respected.
+
+   https://github.com/restic/restic/issues/3518
+   https://github.com/restic/restic/pull/3589
+
+ * Bugfix #3556: Fix hang with Backblaze B2 on SSL certificate authority error
+
+   Previously, if a request failed with an SSL unknown certificate authority error, the B2
+   backend retried indefinitely and restic would appear to hang.
+
+   This has now been fixed and restic instead fails with an error message.
+
+   https://github.com/restic/restic/issues/3556
+   https://github.com/restic/restic/issues/2355
+   https://github.com/restic/restic/pull/3571
+
+ * Bugfix #3601: Fix rclone backend prematurely exiting when receiving SIGINT on Windows
+
+   Previously, pressing Ctrl+C in a Windows console where restic was running with rclone as the
+   backend would cause rclone to exit prematurely due to getting a `SIGINT` signal at the same time
+   as restic. Restic would then wait for a long time for time with "unexpected EOF" and "rclone
+   stdio connection already closed" errors.
+
+   This has now been fixed by restic starting the rclone process detached from the console restic
+   runs in (similar to starting processes in a new process group on Linux), which enables restic to
+   gracefully clean up rclone (which now never gets the `SIGINT`).
+
+   https://github.com/restic/restic/issues/3601
+   https://github.com/restic/restic/pull/3602
+
+ * Bugfix #3667: The `mount` command now reports symlinks sizes
+
+   Symlinks used to have size zero in restic mountpoints, confusing some third-party tools. They
+   now have a size equal to the byte length of their target path, as required by POSIX.
+
+   https://github.com/restic/restic/issues/3667
+   https://github.com/restic/restic/pull/3668
+
+ * Bugfix #3488: `rebuild-index` failed if an index file was damaged
+
+   Previously, the `rebuild-index` command would fail with an error if an index file was damaged
+   or truncated. This has now been fixed.
+
+   On older restic versions, a (slow) workaround is to use `rebuild-index --read-all-packs` or
+   to manually delete the damaged index.
+
+   https://github.com/restic/restic/pull/3488
+
+ * Bugfix #3591: Fix handling of `prune --max-repack-size=0`
+
+   Restic ignored the `--max-repack-size` option when passing a value of 0. This has now been
+   fixed.
+
+   As a workaround, `--max-repack-size=1` can be used with older versions of restic.
+
+   https://github.com/restic/restic/pull/3591
+
+ * Bugfix #3619: Avoid choosing parent snapshots newer than time of new snapshot
+
+   The `backup` command, when a `--parent` was not provided, previously chose the most recent
+   matching snapshot as the parent snapshot. However, this didn't make sense when the user passed
+   `--time` to create a new snapshot older than the most recent snapshot.
+
+   Instead, `backup` now chooses the most recent snapshot which is not newer than the
+   snapshot-being-created's timestamp, to avoid any time travel.
+
+   https://github.com/restic/restic/pull/3619
+
+ * Change #3641: Ignore parent snapshot for `backup --stdin`
+
+   Restic uses a parent snapshot to speed up directory scanning when performing backups, but this
+   only wasted time and memory when the backup source is stdin (using the `--stdin` option of the
+   `backup` command), since no directory scanning is performed in this case.
+
+   Snapshots made with `backup --stdin` no longer have a parent snapshot, which allows restic to
+   skip some startup operations and saves a bit of resources.
+
+   The `--parent` option is still available for `backup --stdin`, but is now ignored.
+
+   https://github.com/restic/restic/issues/3641
+   https://github.com/restic/restic/pull/3645
+
+ * Change #3519: Require Go 1.14 or newer
+
+   Restic now requires Go 1.14 to build. This allows it to use new standard library features
+   instead of an external dependency.
+
+   https://github.com/restic/restic/issues/3519
+
+ * Enhancement #1542: Add `--dry-run`/`-n` option to `backup` command
+
+   Testing exclude filters and other configuration options was error prone as wrong filters
+   could cause files to be uploaded unintentionally. It was also not possible to estimate
+   beforehand how much data would be uploaded.
+
+   The `backup` command now has a `--dry-run`/`-n` option, which performs all the normal steps of
+   a backup without actually writing anything to the repository.
+
+   Passing -vv will log information about files that would be added, allowing for verification of
+   source and exclusion options before running the real backup.
+
+   https://github.com/restic/restic/issues/1542
+   https://github.com/restic/restic/pull/2308
+   https://github.com/restic/restic/pull/3210
+   https://github.com/restic/restic/pull/3300
+
+ * Enhancement #2202: Add upload checksum for Azure, GS, S3 and Swift backends
+
+   Previously only the B2 and partially the Swift backends verified the integrity of uploaded
+   (encrypted) files. The verification works by informing the backend about the expected hash of
+   the uploaded file. The backend then verifies the upload and thereby rules out any data
+   corruption during upload.
+
+   We have now added upload checksums for the Azure, GS, S3 and Swift backends, which besides
+   integrity checking for uploads also means that restic can now be used to store backups in S3
+   buckets which have Object Lock enabled.
+
+   https://github.com/restic/restic/issues/2202
+   https://github.com/restic/restic/issues/2700
+   https://github.com/restic/restic/issues/3023
+   https://github.com/restic/restic/pull/3246
+
+ * Enhancement #233: Support negative include/exclude patterns
+
+   If a pattern starts with an exclamation mark and it matches a file that was previously matched by
+   a regular pattern, the match is cancelled. Notably, this can be used with `--exclude-file` to
+   cancel the exclusion of some files.
+
+   It works similarly to `.gitignore`, with the same limitation; Once a directory is excluded, it
+   is not possible to include files inside the directory.
+
+   Example of use as an exclude pattern for the `backup` command:
+
+   $HOME/**/* !$HOME/Documents !$HOME/code !$HOME/.emacs.d !$HOME/games # [...]
+   node_modules *~ *.o *.lo *.pyc # [...] $HOME/code/linux/* !$HOME/code/linux/.git # [...]
+
+   https://github.com/restic/restic/issues/233
+   https://github.com/restic/restic/pull/2311
+
+ * Enhancement #2388: Add warning for S3 if partial credentials are provided
+
+   Previously restic did not notify about incomplete credentials when using the S3 backend,
+   instead just reporting access denied.
+
+   Restic now checks that both the AWS key ID and secret environment variables are set before
+   connecting to the remote server, and reports an error if not.
+
+   https://github.com/restic/restic/issues/2388
+   https://github.com/restic/restic/pull/3532
+
+ * Enhancement #2508: Support JSON output and quiet mode for the `diff` command
+
+   The `diff` command now supports outputting machine-readable output in JSON format. To enable
+   this, pass the `--json` option to the command. To only print the summary and suppress detailed
+   output, pass the `--quiet` option.
+
+   https://github.com/restic/restic/issues/2508
+   https://github.com/restic/restic/pull/3592
+
+ * Enhancement #2656: Add flag to disable TLS verification for self-signed certificates
+
+   There is now an `--insecure-tls` global option in restic, which disables TLS verification for
+   self-signed certificates in order to support some development workflows.
+
+   https://github.com/restic/restic/issues/2656
+   https://github.com/restic/restic/pull/2657
+
+ * Enhancement #3003: Atomic uploads for the SFTP backend
+
+   The SFTP backend did not upload files atomically. An interrupted upload could leave an
+   incomplete file behind which could prevent restic from accessing the repository. This has now
+   been fixed and uploads in the SFTP backend are done atomically.
+
+   https://github.com/restic/restic/issues/3003
+   https://github.com/restic/restic/pull/3524
+
+ * Enhancement #3127: Add xattr (extended attributes) support for Solaris
+
+   Restic now supports xattr for the Solaris operating system.
+
+   https://github.com/restic/restic/issues/3127
+   https://github.com/restic/restic/pull/3628
+
+ * Enhancement #3464: Skip lock creation on `forget` if `--no-lock` and `--dry-run`
+
+   Restic used to silently ignore the `--no-lock` option of the `forget` command.
+
+   It now skips creation of lock file in case both `--dry-run` and `--no-lock` are specified. If
+   `--no-lock` option is specified without `--dry-run`, restic prints a warning message to
+   stderr.
+
+   https://github.com/restic/restic/issues/3464
+   https://github.com/restic/restic/pull/3623
+
+ * Enhancement #3490: Support random subset by size in `check --read-data-subset`
+
+   The `--read-data-subset` option of the `check` command now supports a third way of specifying
+   the subset to check, namely `nS` where `n` is a size in bytes with suffix `S` as k/K, m/M, g/G or
+   t/T.
+
+   https://github.com/restic/restic/issues/3490
+   https://github.com/restic/restic/pull/3548
+
+ * Enhancement #3541: Improve handling of temporary B2 delete errors
+
+   Deleting files on B2 could sometimes fail temporarily, which required restic to retry the
+   delete operation. In some cases the file was deleted nevertheless, causing the retries and
+   ultimately the restic command to fail. This has now been fixed.
+
+   https://github.com/restic/restic/issues/3541
+   https://github.com/restic/restic/pull/3544
+
+ * Enhancement #3542: Add file mode in symbolic notation to `ls --json`
+
+   The `ls --json` command now provides the file mode in symbolic notation (using the
+   `permissions` key), aligned with `find --json`.
+
+   https://github.com/restic/restic/issues/3542
+   https://github.com/restic/restic/pull/3573
+   https://forum.restic.net/t/restic-ls-understanding-file-mode-with-json/4371
+
+ * Enhancement #2594: Speed up the `restore --verify` command
+
+   The `--verify` option lets the `restore` command verify the file content after it has restored
+   a snapshot. The performance of this operation has now been improved by up to a factor of two.
+
+   https://github.com/restic/restic/pull/2594
+
+ * Enhancement #2816: The `backup` command no longer updates file access times on Linux
+
+   When reading files during backup, restic used to cause the operating system to update the
+   files' access times. Note that this did not apply to filesystems with disabled file access
+   times.
+
+   Restic now instructs the operating system not to update the file access time, if the user
+   running restic is the file owner or has root permissions.
+
+   https://github.com/restic/restic/pull/2816
+
+ * Enhancement #2880: Make `recover` collect only unreferenced trees
+
+   Previously, the `recover` command used to generate a snapshot containing *all* root trees,
+   even those which were already referenced by a snapshot.
+
+   This has been improved such that it now only processes trees not already referenced by any
+   snapshot.
+
+   https://github.com/restic/restic/pull/2880
+
+ * Enhancement #3429: Verify that new or modified keys are stored correctly
+
+   When adding a new key or changing the password of a key, restic used to just create the new key (and
+   remove the old one, when changing the password). There was no verification that the new key was
+   stored correctly and works properly. As the repository cannot be decrypted without a valid key
+   file, this could in rare cases cause the repository to become inaccessible.
+
+   Restic now checks that new key files actually work before continuing. This can protect against
+   some (rare) cases of hardware or storage problems.
+
+   https://github.com/restic/restic/pull/3429
+
+ * Enhancement #3436: Improve local backend's resilience to (system) crashes
+
+   Restic now ensures that files stored using the `local` backend are created atomically (that
+   is, files are either stored completely or not at all). This ensures that no incomplete files are
+   left behind even if restic is terminated while writing a file.
+
+   In addition, restic now tries to ensure that the directory in the repository which contains a
+   newly uploaded file is also written to disk. This can prevent missing files if the system
+   crashes or the disk is not properly unmounted.
+
+   https://github.com/restic/restic/pull/3436
+
+ * Enhancement #3508: Cache blobs read by the `dump` command
+
+   When dumping a file using the `dump` command, restic did not cache blobs in any way, so even
+   consecutive runs of the same blob were loaded from the repository again and again, slowing down
+   the dump.
+
+   Now, the caching mechanism already used by the `fuse` command is also used by the `dump`
+   command. This makes dumping much faster, especially for sparse files.
+
+   https://github.com/restic/restic/pull/3508
+
+ * Enhancement #3511: Support configurable timeout for the rclone backend
+
+   A slow rclone backend could cause restic to time out while waiting for the repository to open.
+   Restic now offers an `-o rclone.timeout` option to make this timeout configurable.
+
+   https://github.com/restic/restic/issues/3511
+   https://github.com/restic/restic/pull/3514
+
+ * Enhancement #3593: Improve `copy` performance by parallelizing IO
+
+   Restic copy previously only used a single thread for copying blobs between repositories,
+   which resulted in limited performance when copying small blobs to/from a high latency backend
+   (i.e. any remote backend, especially b2).
+
+   Copying will now use 8 parallel threads to increase the throughput of the copy operation.
+
+   https://github.com/restic/restic/pull/3593
+
+
 Changelog for restic 0.12.1 (2021-08-03)
 =======================================
 
@@ -604,7 +1009,7 @@ Details
    reduced.
 
    Restic used to rebuild the index from scratch after pruning. This could lead to missing packs in
-   the index in some cases for eventually consistent backends such as e.g. Amazon S3. This behavior is
+   the index in some cases for eventually consistent backends such as e.g. AWS S3. This behavior is
    now changed and the index rebuilding uses the information already known by `prune`.
 
    By default, the `prune` command no longer removes all unused data. This behavior can be
@@ -3083,7 +3488,7 @@ Summary
  * Enh #1055: Create subdirs below `data/` for local/sftp backends
  * Enh #1067: Allow loading credentials for s3 from IAM
  * Enh #1073: Add `migrate` cmd to migrate from `s3legacy` to `default` layout
- * Enh #1081: Clarify semantic for `--tasg` for the `forget` command
+ * Enh #1081: Clarify semantic for `--tag` for the `forget` command
  * Enh #1080: Ignore chmod() errors on filesystems which do not support it
  * Enh #1082: Print stats on SIGINFO on Darwin and FreeBSD (ctrl+t)
 
@@ -3127,7 +3532,7 @@ Details
    https://github.com/restic/restic/issues/1073
    https://github.com/restic/restic/pull/1075
 
- * Enhancement #1081: Clarify semantic for `--tasg` for the `forget` command
+ * Enhancement #1081: Clarify semantic for `--tag` for the `forget` command
 
    https://github.com/restic/restic/issues/1081
    https://github.com/restic/restic/pull/1090