From 828c8bc1e8f7732ab6777c314ba5333abc15cf0c Mon Sep 17 00:00:00 2001 From: Michael Eischer Date: Mon, 26 Aug 2024 14:38:32 +0200 Subject: [PATCH 1/4] doc: describe how to handle rewrite encoding error --- doc/045_working_with_repos.rst | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/doc/045_working_with_repos.rst b/doc/045_working_with_repos.rst index 8dba8439f..f31e75c84 100644 --- a/doc/045_working_with_repos.rst +++ b/doc/045_working_with_repos.rst @@ -305,6 +305,13 @@ In order to preview the changes which ``rewrite`` would make, you can use the modifying the repository. Instead restic will only print the actions it would perform. +.. note:: The ``rewrite`` command verifies that it does not modify snapshots in + unexpected ways and fails with an ``cannot encode tree at "[...]" without loosing information`` + error otherwise. This can occur when rewriting a snapshot created by a newer + version of restic or some third-party implementation. + + To convert a snapshot into the format expected by the ``rewrite`` command + use ``restic repair snapshots ``. Modifying metadata of snapshots =============================== From 908d097904f153b5d7e6fcf13e798b2be2bbcc85 Mon Sep 17 00:00:00 2001 From: Michael Eischer Date: Mon, 26 Aug 2024 14:38:45 +0200 Subject: [PATCH 2/4] doc: mark S3 layout as deprecated --- doc/design.rst | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/doc/design.rst b/doc/design.rst index 7fb8b71b2..26f1f333f 100644 --- a/doc/design.rst +++ b/doc/design.rst @@ -126,8 +126,8 @@ the option ``-o local.layout=default``, valid values are ``default`` and ``s3legacy``. The option for the sftp backend is named ``sftp.layout``, for the s3 backend ``s3.layout``. -S3 Legacy Layout ----------------- +S3 Legacy Layout (deprecated) +----------------------------- Unfortunately during development the Amazon S3 backend uses slightly different paths (directory names use singular instead of plural for ``key``, @@ -152,8 +152,7 @@ the ``data`` directory. The S3 Legacy repository layout looks like this: /snapshot └── 22a5af1bdc6e616f8a29579458c49627e01b32210d09adb288d1ecda7c5711ec -The S3 backend understands and accepts both forms, new backends are -always created with the default layout for compatibility reasons. +Restic 0.17 is the last version that supports the legacy layout. Pack Format =========== From 05077eaa204a315de6776722e38bbd1764431b69 Mon Sep 17 00:00:00 2001 From: Michael Eischer Date: Mon, 26 Aug 2024 14:40:04 +0200 Subject: [PATCH 3/4] doc: JSON encoder must be deterministic --- doc/design.rst | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/doc/design.rst b/doc/design.rst index 26f1f333f..d83ac8b91 100644 --- a/doc/design.rst +++ b/doc/design.rst @@ -233,7 +233,9 @@ Individual files for the index, locks or snapshots are encrypted and authenticated like Data and Tree Blobs, so the outer structure is ``IV || Ciphertext || MAC`` again. In repository format version 1 the plaintext always consists of a JSON document which must either be an -object or an array. +object or an array. The JSON encoder must deterministically encode the +document and should match the behavior of the Go standard library implementation +in ``encoding/json``. Repository format version 2 adds support for compression. The plaintext now starts with a header to indicate the encoding version to distinguish @@ -472,6 +474,10 @@ A snapshot references a tree by the SHA-256 hash of the JSON string representation of its contents. Trees and data are saved in pack files in a subdirectory of the directory ``data``. +The JSON encoder must deterministically encode the document and should +match the behavior of the Go standard library implementation in ``encoding/json``. +This ensures that trees can be properly deduplicated. + The command ``restic cat blob`` can be used to inspect the tree referenced above (piping the output of the command to ``jq .`` so that the JSON is indented): From dc5d3fc4732bc1c0c20769b62084b963801245b4 Mon Sep 17 00:00:00 2001 From: Michael Eischer Date: Mon, 26 Aug 2024 14:40:38 +0200 Subject: [PATCH 4/4] doc: full tree blob data structure is in the code --- doc/design.rst | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/doc/design.rst b/doc/design.rst index d83ac8b91..c974e997a 100644 --- a/doc/design.rst +++ b/doc/design.rst @@ -512,12 +512,11 @@ this metadata is generated: - The name is quoted using `strconv.Quote `__ before being saved. This handles non-unicode names, but also changes the representation of names containing ``"`` or ``\``. - - The filemode saved is the mode defined by `fs.FileMode `__ masked by ``os.ModePerm | os.ModeType | os.ModeSetuid | os.ModeSetgid | os.ModeSticky`` - -When the entry references a directory, the field ``subtree`` contains the plain text -ID of another tree object. +- When the entry references a directory, the field ``subtree`` contains the plain text + ID of another tree object. +- Check the implementation for a full struct definition. When the command ``restic cat blob`` is used, the plaintext ID is needed to print a tree. The tree referenced above can be dumped as follows: