KurlesHS/restic
1
0
Fork 0
forked from TrueCloudLab/restic
Code Pull requests Activity

Merge pull request #2740 from metalsp0rk/document-json

Browse source 
Add JSON documentation
This commit is contained in:
Michael Eischer 2023-07-08 21:34:02 +00:00 committed by GitHub
commit 41395e83c5
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
1 changed files with 535 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

535
doc/075_scripting.rst
View file

@ -37,3 +37,538 @@ exit code if a different error is encountered (e.g.: incorrect password
to ``cat config``) and it may print a different error message. If there to ``cat config``) and it may print a different error message. If there
are no errors, restic will return a zero exit code and print the repository are no errors, restic will return a zero exit code and print the repository
metadata. metadata.
JSON output
***********
Restic outputs JSON data to ``stdout`` if requested with the ``--json`` flag.
The structure of that data varies depending on the circumstance. The
JSON output of most restic commands are documented here.
.. note::
Not all commands support JSON output. If a command does not support JSON output,
feel free to submit a pull request!
.. warning::
We try to keep the JSON output backwards compatible. However, new message types
or fields may be added at any time. Similarly, enum-like fields for which a fixed
list of allowed values is documented may be extended at any time.
Output formats
--------------
Currently only the output on ``stdout`` is JSON formatted. Errors printed on ``stderr``
are still printed as plain text messages. The generated JSON output uses one of the
following two formats.
Single JSON document
^^^^^^^^^^^^^^^^^^^^
Several commands output a single JSON document that can be parsed in its entirety.
Depending on the command, the output consists of either a single or multiple lines.
JSON lines
^^^^^^^^^^
Several commands, in particular long running ones or those that generate a large output,
use a format also known as JSON lines. It consists of a stream of new-line separated JSON
messages. You can determine the nature of the message using the ``message_type`` field.
As an exception, the ``ls`` command uses the field ``struct_type`` instead.
backup
------
The ``backup`` command uses the JSON lines format with the following message types.
Status
^^^^^^
+----------------------+------------------------------------------------------------+
|``message_type`` | Always "status" |
+----------------------+------------------------------------------------------------+
|``seconds_elapsed`` | Time since backup started |
+----------------------+------------------------------------------------------------+
|``seconds_remaining`` | Estimated time remaining |
+----------------------+------------------------------------------------------------+
|``percent_done`` | Percentage of data backed up (bytes_done/total_bytes) |
+----------------------+------------------------------------------------------------+
|``total_files`` | Total number of files detected |
+----------------------+------------------------------------------------------------+
|``files_done`` | Files completed (backed up to repo) |
+----------------------+------------------------------------------------------------+
|``total_bytes`` | Total number of bytes in backup set |
+----------------------+------------------------------------------------------------+
|``bytes_done`` | Number of bytes completed (backed up to repo) |
+----------------------+------------------------------------------------------------+
|``error_count`` | Number of errors |
+----------------------+------------------------------------------------------------+
|``current_files`` | List of files currently being backed up |
+----------------------+------------------------------------------------------------+
Error
^^^^^
+----------------------+-------------------------------------------+
| ``message_type`` | Always "error" |
+----------------------+-------------------------------------------+
| ``error`` | Error message |
+----------------------+-------------------------------------------+
| ``during`` | What restic was trying to do |
+----------------------+-------------------------------------------+
| ``item`` | Usually, the path of the problematic file |
+----------------------+-------------------------------------------+
Verbose Status
^^^^^^^^^^^^^^
Verbose status provides details about the progress, including details about backed up files.
+----------------------+-----------------------------------------------------------+
| ``message_type`` | Always "verbose_status" |
+----------------------+-----------------------------------------------------------+
| ``action`` | Either "new", "unchanged", "modified" or "scan_finished" |
+----------------------+-----------------------------------------------------------+
| ``item`` | The item in question |
+----------------------+-----------------------------------------------------------+
| ``duration`` | How long it took, in seconds |
+----------------------+-----------------------------------------------------------+
| ``data_size`` | How big the item is |
+----------------------+-----------------------------------------------------------+
| ``metadata_size`` | How big the metadata is |
+----------------------+-----------------------------------------------------------+
| ``total_files`` | Total number of files |
+----------------------+-----------------------------------------------------------+
Summary
^^^^^^^
Summary is the last output line in a successful backup.
+---------------------------+---------------------------------------------------------+
| ``message_type`` | Always "summary" |
+---------------------------+---------------------------------------------------------+
| ``files_new`` | Number of new files |
+---------------------------+---------------------------------------------------------+
| ``files_changed`` | Number of files that changed |
+---------------------------+---------------------------------------------------------+
| ``files_unmodified`` | Number of files that did not change |
+---------------------------+---------------------------------------------------------+
| ``dirs_new`` | Number of new directories |
+---------------------------+---------------------------------------------------------+
| ``dirs_changed`` | Number of directories that changed |
+---------------------------+---------------------------------------------------------+
| ``dirs_unmodified`` | Number of directories that did not change |
+---------------------------+---------------------------------------------------------+
| ``data_blobs`` | Number of data blobs |
+---------------------------+---------------------------------------------------------+
| ``tree_blobs`` | Number of tree blobs |
+---------------------------+---------------------------------------------------------+
| ``data_added`` | Amount of data added, in bytes |
+---------------------------+---------------------------------------------------------+
| ``total_files_processed`` | Total number of files processed |
+---------------------------+---------------------------------------------------------+
| ``total_bytes_processed`` | Total number of bytes processed |
+---------------------------+---------------------------------------------------------+
| ``total_duration`` | Total time it took for the operation to complete |
+---------------------------+---------------------------------------------------------+
| ``snapshot_id`` | ID of the new snapshot |
+---------------------------+---------------------------------------------------------+
cat
---
The ``cat`` command returns data about various objects in the repository, which
are stored in JSON form. Specifying ``--json`` or ``--quiet`` will suppress any
non-JSON messages the command generates.
diff
----
The ``diff`` command uses the JSON lines format with the following message types.
change
^^^^^^
+------------------+--------------------------------------------------------------+
| ``message_type`` | Always "change" |
+------------------+--------------------------------------------------------------+
| ``path`` | Path that has changed |
+------------------+--------------------------------------------------------------+
| ``modifier`` | Type of change, a concatenation of the following characters: |
| | "+" = added, "-" = removed, "T" = entry type changed, |
| | "M" = file content changed, "U" = metadata changed |
+------------------+--------------------------------------------------------------+
statistics
^^^^^^^^^^
+---------------------+----------------------------+
| ``message_type`` | Always "statistics" |
+---------------------+----------------------------+
| ``source_snapshot`` | ID of first snapshot |
+---------------------+----------------------------+
| ``target_snapshot`` | ID of second snapshot |
+---------------------+----------------------------+
| ``changed_files`` | Number of changed files |
+---------------------+----------------------------+
| ``added`` | DiffStat object, see below |
+---------------------+----------------------------+
| ``removed`` | DiffStat object, see below |
+---------------------+----------------------------+
DiffStat object
+----------------+-------------------------------------------+
| ``files`` | Number of changed files |
+----------------+-------------------------------------------+
| ``dirs`` | Number of changed directories |
+----------------+-------------------------------------------+
| ``others`` | Number of changed other directory entries |
+----------------+-------------------------------------------+
| ``data_blobs`` | Number of data blobs |
+----------------+-------------------------------------------+
| ``tree_blobs`` | Number of tree blobs |
+----------------+-------------------------------------------+
| ``bytes`` | Number of bytes |
+----------------+-------------------------------------------+
find
----
The ``find`` command outputs a single JSON document containing an array of JSON
objects with matches for your search term. These matches are organized by snapshot.
If the ``--blob`` or ``--tree`` option is passed, then the output is an array of
Blob objects.
+-----------------+----------------------------------------------+
| ``hits`` | Number of matches in the snapshot |
+-----------------+----------------------------------------------+
| ``snapshot`` | ID of the snapshot |
+-----------------+----------------------------------------------+
| ``matches`` | Array of Match objects detailing a match |
+-----------------+----------------------------------------------+
Match object
+-----------------+----------------------------------------------+
| ``path`` | Object path |
+-----------------+----------------------------------------------+
| ``permissions`` | UNIX permissions |
+-----------------+----------------------------------------------+
| ``type`` | Object type e.g. file, dir, etc... |
+-----------------+----------------------------------------------+
| ``atime`` | Access time |
+-----------------+----------------------------------------------+
| ``mtime`` | Modification time |
+-----------------+----------------------------------------------+
| ``ctime`` | Change time |
+-----------------+----------------------------------------------+
| ``name`` | Object name |
+-----------------+----------------------------------------------+
| ``user`` | Name of owner |
+-----------------+----------------------------------------------+
| ``group`` | Name of group |
+-----------------+----------------------------------------------+
| ``mode`` | UNIX file mode, shorthand of ``permissions`` |
+-----------------+----------------------------------------------+
| ``device_id`` | OS specific device identifier |
+-----------------+----------------------------------------------+
| ``links`` | Number of hardlinks |
+-----------------+----------------------------------------------+
| ``uid`` | ID of owner |
+-----------------+----------------------------------------------+
| ``gid`` | ID of group |
+-----------------+----------------------------------------------+
| ``size`` | Size of object in bytes |
+-----------------+----------------------------------------------+
Blob object
+-----------------+--------------------------------------------+
| ``object_type`` | Either "blob" or "tree" |
+-----------------+--------------------------------------------+
| ``id`` | ID of found blob |
+-----------------+--------------------------------------------+
| ``path`` | Path in snapshot |
+-----------------+--------------------------------------------+
| ``parent_tree`` | Parent tree blob, only set for type "blob" |
+-----------------+--------------------------------------------+
| ``snapshot`` | Snapshot ID |
+-----------------+--------------------------------------------+
| ``time`` | Snapshot timestamp |
+-----------------+--------------------------------------------+
forget
------
The ``forget`` command prints a single JSON document containing an array of
ForgetGroups. If specific snapshot IDs are specified, then no output is generated.
The ``prune`` command does not yet support JSON such that ``forget --prune``
results in a mix of JSON and text output.
ForgetGroup
^^^^^^^^^^^
+-------------+-----------------------------------------------------------+
| ``tags`` | Tags identifying the snapshot group |
+-------------+-----------------------------------------------------------+
| ``host`` | Host identifying the snapshot group |
+-------------+-----------------------------------------------------------+
| ``paths`` | Paths identifying the snapshot group |
+-------------+-----------------------------------------------------------+
| ``keep`` | Array of Snapshot objects that are kept |
+-------------+-----------------------------------------------------------+
| ``remove`` | Array of Snapshot objects that were removed |
+-------------+-----------------------------------------------------------+
| ``reasons`` | Array of Reason objects describing why a snapshot is kept |
+-------------+-----------------------------------------------------------+
Snapshot object
+---------------------+--------------------------------------------------+
| ``time`` | Timestamp of when the backup was started |
+---------------------+--------------------------------------------------+
| ``parent`` | ID of the parent snapshot |
+---------------------+--------------------------------------------------+
| ``tree`` | ID of the root tree blob |
+---------------------+--------------------------------------------------+
| ``paths`` | List of paths included in the backup |
+---------------------+--------------------------------------------------+
| ``hostname`` | Hostname of the backed up machine |
+---------------------+--------------------------------------------------+
| ``username`` | Username the backup command was run as |
+---------------------+--------------------------------------------------+
| ``uid`` | ID of owner |
+---------------------+--------------------------------------------------+
| ``gid`` | ID of group |
+---------------------+--------------------------------------------------+
| ``excludes`` | List of paths and globs excluded from the backup |
+---------------------+--------------------------------------------------+
| ``tags`` | List of tags for the snapshot in question |
+---------------------+--------------------------------------------------+
| ``program_version`` | restic version used to create snapshot |
+---------------------+--------------------------------------------------+
| ``id`` | Snapshot ID |
+---------------------+--------------------------------------------------+
| ``short_id`` | Snapshot ID, short form |
+---------------------+--------------------------------------------------+
Reason object
+----------------+---------------------------------------------------------+
| ``snapshot`` | Snapshot object, without ``id`` and ``short_id`` fields |
+----------------+---------------------------------------------------------+
| ``matches`` | Array containing descriptions of the matching criteria |
+----------------+---------------------------------------------------------+
| ``counters`` | Object containing counters used by the policies |
+----------------+---------------------------------------------------------+
init
----
The ``init`` command uses the JSON lines format, but only outputs a single message.
+------------------+--------------------------------+
| ``message_type`` | Always "initialized" |
+------------------+--------------------------------+
| ``id`` | ID of the created repository |
+------------------+--------------------------------+
| ``repository`` | URL of the repository |
+------------------+--------------------------------+
key list
--------
The ``key list`` command returns an array of objects with the following structure.
+--------------+------------------------------------+
| ``current`` | Is currently used key? |
+--------------+------------------------------------+
| ``id`` | Unique key ID |
+--------------+------------------------------------+
| ``userName`` | User who created it |
+--------------+------------------------------------+
| ``hostName`` | Name of machine it was created on |
+--------------+------------------------------------+
| ``created`` | Timestamp when it was created |
+--------------+------------------------------------+
ls
--
The ``ls`` command uses the JSON lines format with the following message types.
As an exception, the ``struct_type`` field is used to determine the message type.
snapshot
^^^^^^^^
+----------------+--------------------------------------------------+
| ``struct_type``| Always "snapshot" |
+----------------+--------------------------------------------------+
| ``time`` | Timestamp of when the backup was started |
+----------------+--------------------------------------------------+
| ``parent`` | ID of the parent snapshot |
+----------------+--------------------------------------------------+
| ``tree`` | ID of the root tree blob |
+----------------+--------------------------------------------------+
| ``paths`` | List of paths included in the backup |
+----------------+--------------------------------------------------+
| ``hostname`` | Hostname of the backed up machine |
+----------------+--------------------------------------------------+
| ``username`` | Username the backup command was run as |
+----------------+--------------------------------------------------+
| ``uid`` | ID of owner |
+----------------+--------------------------------------------------+
| ``gid`` | ID of group |
+----------------+--------------------------------------------------+
| ``excludes`` | List of paths and globs excluded from the backup |
+----------------+--------------------------------------------------+
| ``tags`` | List of tags for the snapshot in question |
+----------------+--------------------------------------------------+
| ``id`` | Snapshot ID |
+----------------+--------------------------------------------------+
| ``short_id`` | Snapshot ID, short form |
+----------------+--------------------------------------------------+
node
^^^^
+-----------------+--------------------------+
| ``struct_type`` | Always "node" |
+-----------------+--------------------------+
| ``name`` | Node name |
+-----------------+--------------------------+
| ``type`` | Node type |
+-----------------+--------------------------+
| ``path`` | Node path |
+-----------------+--------------------------+
| ``uid`` | UID of node |
+-----------------+--------------------------+
| ``gid`` | GID of node |
+-----------------+--------------------------+
| ``size`` | Size in bytes |
+-----------------+--------------------------+
| ``mode`` | Node mode |
+-----------------+--------------------------+
| ``atime`` | Node access time |
+-----------------+--------------------------+
| ``mtime`` | Node modification time |
+-----------------+--------------------------+
| ``ctime`` | Node creation time |
+-----------------+--------------------------+
restore
-------
The ``restore`` command uses the JSON lines format with the following message types.
Status
^^^^^^
+----------------------+------------------------------------------------------------+
|``message_type`` | Always "status" |
+----------------------+------------------------------------------------------------+
|``seconds_elapsed`` | Time since restore started |
+----------------------+------------------------------------------------------------+
|``percent_done`` | Percentage of data backed up (bytes_restored/total_bytes) |
+----------------------+------------------------------------------------------------+
|``total_files`` | Total number of files detected |
+----------------------+------------------------------------------------------------+
|``files_restored`` | Files restored |
+----------------------+------------------------------------------------------------+
|``total_bytes`` | Total number of bytes in restore set |
+----------------------+------------------------------------------------------------+
|``bytes_restored`` | Number of bytes restored |
+----------------------+------------------------------------------------------------+
Summary
^^^^^^^
+----------------------+------------------------------------------------------------+
|``message_type`` | Always "summary" |
+----------------------+------------------------------------------------------------+
|``seconds_elapsed`` | Time since restore started |
+----------------------+------------------------------------------------------------+
|``total_files`` | Total number of files detected |
+----------------------+------------------------------------------------------------+
|``files_restored`` | Files restored |
+----------------------+------------------------------------------------------------+
|``total_bytes`` | Total number of bytes in restore set |
+----------------------+------------------------------------------------------------+
|``bytes_restored`` | Number of bytes restored |
+----------------------+------------------------------------------------------------+
snapshots
---------
The snapshots command returns a single JSON object, an array with objects of the structure outlined below.
+---------------------+--------------------------------------------------+
| ``time`` | Timestamp of when the backup was started |
+---------------------+--------------------------------------------------+
| ``parent`` | ID of the parent snapshot |
+---------------------+--------------------------------------------------+
| ``tree`` | ID of the root tree blob |
+---------------------+--------------------------------------------------+
| ``paths`` | List of paths included in the backup |
+---------------------+--------------------------------------------------+
| ``hostname`` | Hostname of the backed up machine |
+---------------------+--------------------------------------------------+
| ``username`` | Username the backup command was run as |
+---------------------+--------------------------------------------------+
| ``uid`` | ID of owner |
+---------------------+--------------------------------------------------+
| ``gid`` | ID of group |
+---------------------+--------------------------------------------------+
| ``excludes`` | List of paths and globs excluded from the backup |
+---------------------+--------------------------------------------------+
| ``tags`` | List of tags for the snapshot in question |
+---------------------+--------------------------------------------------+
| ``program_version`` | restic version used to create snapshot |
+---------------------+--------------------------------------------------+
| ``id`` | Snapshot ID |
+---------------------+--------------------------------------------------+
| ``short_id`` | Snapshot ID, short form |
+---------------------+--------------------------------------------------+
stats
-----
The snapshots command returns a single JSON object.
+------------------------------+-----------------------------------------------------+
| ``total_size`` | Repository size in bytes |
+------------------------------+-----------------------------------------------------+
| ``total_file_count`` | Number of files backed up in the repository |
+------------------------------+-----------------------------------------------------+
| ``total_blob_count`` | Number of blobs in the repository |
+------------------------------+-----------------------------------------------------+
| ``snapshots_count`` | Number of processed snapshots |
+------------------------------+-----------------------------------------------------+
| ``total_uncompressed_size`` | Repository size in bytes if blobs were uncompressed |
+------------------------------+-----------------------------------------------------+
| ``compression_ratio`` | Factor by which the already compressed data |
| | has shrunk due to compression |
+------------------------------+-----------------------------------------------------+
| ``compression_progress`` | Percentage of already compressed data |
+------------------------------+-----------------------------------------------------+
| ``compression_space_saving`` | Overall space saving due to compression |
+------------------------------+-----------------------------------------------------+