From baa02d45703043ded4f95066f88bdbb5b98829f7 Mon Sep 17 00:00:00 2001 From: Stanislav Bogatyrev Date: Fri, 11 Dec 2020 10:20:21 +0300 Subject: [PATCH] Update auto-generated documentation Signed-off-by: Stanislav Bogatyrev --- proto-docs/acl.md | 4 ++- proto-docs/container.md | 10 ++++-- proto-docs/netmap.md | 5 ++- proto-docs/object.md | 74 ++++++++++++++++++++++++++++++++--------- proto-docs/refs.md | 39 +++++++++++++++++----- proto-docs/session.md | 2 +- proto-docs/tombstone.md | 63 +++++++++++++++++++++++++++++++++++ 7 files changed, 168 insertions(+), 29 deletions(-) create mode 100644 proto-docs/tombstone.md diff --git a/proto-docs/acl.md b/proto-docs/acl.md index da4f946..6bf0176 100644 --- a/proto-docs/acl.md +++ b/proto-docs/acl.md @@ -65,7 +65,7 @@ owner with additional information preventing token's abuse. ### Message BearerToken.Body.TokenLifetime -Lifetime parameters of the token. Filed names taken from +Lifetime parameters of the token. Field names taken from [rfc7519](https://tools.ietf.org/html/rfc7519). @@ -101,6 +101,8 @@ prefix to the name. Here is the list of fields available via this prefix: * $Object:version \ version +* $Object:objectID \ + object_id * $Object:containerID \ container_id * $Object:ownerID \ diff --git a/proto-docs/container.md b/proto-docs/container.md index 25ac837..c5dbe82 100644 --- a/proto-docs/container.md +++ b/proto-docs/container.md @@ -453,7 +453,7 @@ of stable-marshalled container message. | ----- | ---- | ----- | ----------- | | version | [neo.fs.v2.refs.Version](#neo.fs.v2.refs.Version) | | Container format version. Effectively the version of API library used to create container. | | owner_id | [neo.fs.v2.refs.OwnerID](#neo.fs.v2.refs.OwnerID) | | Identifier of the container owner | -| nonce | [bytes](#bytes) | | Nonce is a 16 byte UUID, used to avoid collisions of container id | +| nonce | [bytes](#bytes) | | Nonce is a 16 byte UUIDv4, used to avoid collisions of `ContainerID`s | | basic_acl | [uint32](#uint32) | | `BasicACL` contains access control rules for owner, system, others groups and permission bits for `BearerToken` and `Extended ACL` | | attributes | [Container.Attribute](#neo.fs.v2.container.Container.Attribute) | repeated | Attributes represent immutable container's meta data | | placement_policy | [neo.fs.v2.netmap.PlacementPolicy](#neo.fs.v2.netmap.PlacementPolicy) | | Placement policy for the object inside the container | @@ -463,8 +463,12 @@ of stable-marshalled container message. ### Message Container.Attribute `Attribute` is a user-defined Key-Value metadata pair attached to the -container. Container attribute are immutable. They are set at container -creation and cna never be added or updated. +container. Container attributes are immutable. They are set at container +creation and can never be added or updated. + +Key name must be a container-unique valid UTF-8 string. Value can't be +empty. Containers with duplicated attribute names or attributes with empty +values will be considered invalid. There are some "well-known" attributes affecting system behaviour: diff --git a/proto-docs/netmap.md b/proto-docs/netmap.md index 30386ec..cc27c5a 100644 --- a/proto-docs/netmap.md +++ b/proto-docs/netmap.md @@ -149,7 +149,7 @@ NeoFS node description | ----- | ---- | ----- | ----------- | | public_key | [bytes](#bytes) | | Public key of the NeoFS node in a binary format. | | address | [string](#string) | | Ways to connect to a node | -| attributes | [NodeInfo.Attribute](#neo.fs.v2.netmap.NodeInfo.Attribute) | repeated | Carries list of the NeoFS node attributes in a string key-value format. | +| attributes | [NodeInfo.Attribute](#neo.fs.v2.netmap.NodeInfo.Attribute) | repeated | Carries list of the NeoFS node attributes in a key-value form. Key name must be a node-unique valid UTF-8 string. Value can't be empty. NodeInfo structures with duplicated attribute names or attributes with empty values will be considered invalid. | | state | [NodeInfo.State](#neo.fs.v2.netmap.NodeInfo.State) | | Carries state of the NeoFS node. | @@ -158,6 +158,9 @@ NeoFS node description ### Message NodeInfo.Attribute Administrator-defined Attributes of the NeoFS Storage Node. +`Attribute` is a Key-Value metadata pair. Key name must be a valid UTF-8 +string. Value can't be empty. + Node's attributes are mostly used during Storage Policy evaluation to calculate object's placement and find a set of nodes satisfying policy requirements. There are some "well-known" node attributes common to all the diff --git a/proto-docs/object.md b/proto-docs/object.md index 4072fa4..1b0433f 100644 --- a/proto-docs/object.md +++ b/proto-docs/object.md @@ -51,6 +51,7 @@ - [Header.Split](#neo.fs.v2.object.Header.Split) - [Object](#neo.fs.v2.object.Object) - [ShortHeader](#neo.fs.v2.object.ShortHeader) + - [SplitInfo](#neo.fs.v2.object.SplitInfo) - [Scalar Value Types](#scalar-value-types) @@ -199,6 +200,10 @@ removal in distributed system. Object DELETE Response has an empty body. +| Field | Type | Label | Description | +| ----- | ---- | ----- | ----------- | +| tombstone | [neo.fs.v2.refs.Address](#neo.fs.v2.refs.Address) | | Address of the tombstone created for the deleted object | + @@ -275,6 +280,7 @@ Byte range of object's payload request body | ----- | ---- | ----- | ----------- | | address | [neo.fs.v2.refs.Address](#neo.fs.v2.refs.Address) | | Address of the object containing the requested payload range | | range | [Range](#neo.fs.v2.object.Range) | | Requested payload range | +| raw | [bool](#bool) | | If `raw` flag is set, request will work only with objects that are physically stored on the peer node. | @@ -301,7 +307,8 @@ chunks. | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | -| chunk | [bytes](#bytes) | | Chunked object payload's range | +| chunk | [bytes](#bytes) | | Chunked object payload's range. | +| split_info | [SplitInfo](#neo.fs.v2.object.SplitInfo) | | Meta information of split hierarchy. | @@ -352,6 +359,7 @@ GET Object Response body | ----- | ---- | ----- | ----------- | | init | [GetResponse.Body.Init](#neo.fs.v2.object.GetResponse.Body.Init) | | Initial part of the object stream | | chunk | [bytes](#bytes) | | Chunked object payload | +| split_info | [SplitInfo](#neo.fs.v2.object.SplitInfo) | | Meta information of split hierarchy for object assembly. | @@ -417,6 +425,7 @@ Object HEAD response body | ----- | ---- | ----- | ----------- | | header | [HeaderWithSignature](#neo.fs.v2.object.HeaderWithSignature) | | Full object's `Header` with `ObjectID` signature | | short_header | [ShortHeader](#neo.fs.v2.object.ShortHeader) | | Short object header | +| split_info | [SplitInfo](#neo.fs.v2.object.SplitInfo) | | Meta information of split hierarchy. | @@ -551,6 +560,8 @@ prefix to the name. Here is the list of fields available via this prefix: * $Object:version \ version +* $Object:objectID \ + object_id * $Object:containerID \ container_id * $Object:ownerID \ @@ -567,20 +578,24 @@ prefix to the name. Here is the list of fields available via this prefix: homomorphic_hash * $Object:split.parent \ object_id of parent +* $Object:split.splitID \ + 16 byte UUIDv4 used to identify the split object hierarchy parts There are some well-known filter aliases to match objects by certain properties: * $Object:ROOT \ - With the `value` set to `true` checks if an object is a top object in a - split hierarchy. With other values returns non-root objects. -* $Object:LEAF \ - With the `value` set to `true` checks if an object is a leaf in a split - hierarchy. With other values returns non-leaf objects. -* $Object:CHILDFREE \ - With the `value` set to `true` checks if an object has empty - children list in `Split` header. With other values returns - non-childfree objects. + Returns only `REGULAR` type objects that are not split or are the top + level root objects in a split hierarchy. This includes objects not + present physically, like large objects split into smaller objects + without separate top-level root object. Other type objects like + StorageGroups and Tombstones will not be shown. This filter may be + useful for listing objects like `ls` command of some virtual file + system. This filter is activated if the `key` exists, disregarding the + value and matcher type. +* $Object:PHY \ + Returns only objects physically stored in the system. This filter is + activated if the `key` exists, disregarding the value and matcher type. | Field | Type | Label | Description | @@ -640,10 +655,10 @@ Object Header | container_id | [neo.fs.v2.refs.ContainerID](#neo.fs.v2.refs.ContainerID) | | Object's container | | owner_id | [neo.fs.v2.refs.OwnerID](#neo.fs.v2.refs.OwnerID) | | Object's owner | | creation_epoch | [uint64](#uint64) | | Object creation Epoch | -| payload_length | [uint64](#uint64) | | Size of payload in bytes. `0xFFFFFFFFFFFFFFFF` means `payload_length` is unknown | +| payload_length | [uint64](#uint64) | | Size of payload in bytes. `0xFFFFFFFFFFFFFFFF` means `payload_length` is unknown. | | payload_hash | [neo.fs.v2.refs.Checksum](#neo.fs.v2.refs.Checksum) | | Hash of payload bytes | | object_type | [ObjectType](#neo.fs.v2.object.ObjectType) | | Type of the object payload content | -| homomorphic_hash | [neo.fs.v2.refs.Checksum](#neo.fs.v2.refs.Checksum) | | Homomorphic hash of the object payload. | +| homomorphic_hash | [neo.fs.v2.refs.Checksum](#neo.fs.v2.refs.Checksum) | | Homomorphic hash of the object payload | | session_token | [neo.fs.v2.session.SessionToken](#neo.fs.v2.session.SessionToken) | | Session token, if it was used during Object creation. Need it to verify integrity and authenticity out of Request scope. | | attributes | [Header.Attribute](#neo.fs.v2.object.Header.Attribute) | repeated | User-defined object attributes | | split | [Header.Split](#neo.fs.v2.object.Header.Split) | | Position of the object in the split hierarchy | @@ -655,6 +670,10 @@ Object Header `Attribute` is a user-defined Key-Value metadata pair attached to the object. +Key name must be a object-unique valid UTF-8 string. Value can't be empty. +Objects with duplicated attribute names or attributes with empty values +will be considered invalid. + There are some "well-known" attributes starting with `__NEOFS__` prefix that affect system behaviour: @@ -698,6 +717,7 @@ must be within the same container. | parent_signature | [neo.fs.v2.refs.Signature](#neo.fs.v2.refs.Signature) | | `signature` field of the parent object. Used to reconstruct parent. | | parent_header | [Header](#neo.fs.v2.object.Header) | | `header` field of the parent object. Used to reconstruct parent. | | children | [neo.fs.v2.refs.ObjectID](#neo.fs.v2.refs.ObjectID) | repeated | List of identifiers of the objects generated by splitting current one. | +| split_id | [bytes](#bytes) | | 16 byte UUIDv4 used to identify the split object hierarchy parts. Must be unique inside container. All objects participating in the split must have the same `split_id` value. | @@ -713,7 +733,7 @@ hash of header field, which contains hash of object's payload. | object_id | [neo.fs.v2.refs.ObjectID](#neo.fs.v2.refs.ObjectID) | | Object's unique identifier. | | signature | [neo.fs.v2.refs.Signature](#neo.fs.v2.refs.Signature) | | Signed object_id | | header | [Header](#neo.fs.v2.object.Header) | | Object metadata headers | -| payload | [bytes](#bytes) | | Payload bytes. | +| payload | [bytes](#bytes) | | Payload bytes | @@ -724,12 +744,28 @@ Short header fields | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | -| version | [neo.fs.v2.refs.Version](#neo.fs.v2.refs.Version) | | Object format version. Effectively the version of API library used to create particular object | +| version | [neo.fs.v2.refs.Version](#neo.fs.v2.refs.Version) | | Object format version. Effectively the version of API library used to create particular object. | | creation_epoch | [uint64](#uint64) | | Epoch when the object was created | | owner_id | [neo.fs.v2.refs.OwnerID](#neo.fs.v2.refs.OwnerID) | | Object's owner | | object_type | [ObjectType](#neo.fs.v2.object.ObjectType) | | Type of the object payload content | | payload_length | [uint64](#uint64) | | Size of payload in bytes. `0xFFFFFFFFFFFFFFFF` means `payload_length` is unknown | + + + +### Message SplitInfo +Meta information of split hierarchy for object assembly. With last part +one can traverse linked list of split hierarchy back to first part and +assemble original object. With linking object one can assembly object +straight away from the object parts. + + +| Field | Type | Label | Description | +| ----- | ---- | ----- | ----------- | +| split_id | [bytes](#bytes) | | 16 byte UUID used to identify the split object hierarchy parts. | +| last_part | [neo.fs.v2.refs.ObjectID](#neo.fs.v2.refs.ObjectID) | | Identifier of the last object in split hierarchy parts. It contains split header with original object header. | +| link | [neo.fs.v2.refs.ObjectID](#neo.fs.v2.refs.ObjectID) | | Identifier of linking object for split hierarchy parts. It contains split header with original object header and sorted list of object parts. | + @@ -748,7 +784,15 @@ Type of match expression ### ObjectType -Type of the object payload content. +Type of the object payload content. Only `REGULAR` type objects can be split, +hence `TOMBSTONE` and `STORAGEGROUP` payload is limited by maximal object +size. + +String presentation of object type is PascalCased `ObjectType` enumeration +item name: +* Regular +* Tombstone +* StorageGroup | Name | Number | Description | | ---- | ------ | ----------- | diff --git a/proto-docs/refs.md b/proto-docs/refs.md index f53d36a..5280c23 100644 --- a/proto-docs/refs.md +++ b/proto-docs/refs.md @@ -31,8 +31,10 @@ ### Message Address -Object in NeoFS can be addressed by it's ContainerID and ObjectID. In string -format there MUST be a '/' delimeter between them. +Objects in NeoFS are addressed by their ContainerID and ObjectID. + +String presentation of `Address` is the concatenation of string encoded +`ContainerID` and `ObjectID` delimited by '/' character. | Field | Type | Label | Description | @@ -45,6 +47,12 @@ format there MUST be a '/' delimeter between them. ### Message Checksum Checksum message. +Depending on checksum algorithm type the string presentation may vary: + +* TZ \ + Hex encoded string without `0x` prefix +* SHA256 \ + Hex encoded string without `0x` prefix | Field | Type | Label | Description | @@ -57,8 +65,12 @@ Checksum message. ### Message ContainerID NeoFS container identifier. Container structures are immutable and -content-addressed. `ContainerID` is a 32 byte long SHA256 hash of -stable-marshalled container message. +content-addressed. + +`ContainerID` is a 32 byte long SHA256 hash of stable-marshalled container +message. + +String presentation is base58 encoded string. | Field | Type | Label | Description | @@ -70,9 +82,12 @@ stable-marshalled container message. ### Message ObjectID NeoFS Object unique identifier. Objects are immutable and content-addressed. -It means `ObjectID` will change if `header` or `payload` changes. `ObjectID` -is calculated as a hash of `header` field, which contains hash of object's -payload. +It means `ObjectID` will change if `header` or `payload` changes. + +`ObjectID` is a 32 byte long SHA256 hash of object's `header` field, which, +in it's turn, contains hash of object's payload. + +String presentation is base58 encoded string. | Field | Type | Label | Description | @@ -83,10 +98,15 @@ payload. ### Message OwnerID -OwnerID is a derivative of a user's main public key. The transformation +`OwnerID` is a derivative of a user's main public key. The transformation algorithm is the same as for Neo3 wallet addresses. Neo3 wallet address can be directly used as `OwnerID`. +`OwnerID` is a 25 bytes sequence starting with Neo version prefix byte +followed by 20 bytes of ScrptHash and 4 bytes of checksum. + +String presentation is Base58 Check Encoded string. + | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | @@ -110,6 +130,9 @@ Signature of something in NeoFS. ### Message Version API version used by a node. +String presentation is a Semantic Versioning 2.0.0 compatible version string +with 'v' prefix. I.e. `vX.Y`, where `X` - major number, `Y` - minor number. + | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | diff --git a/proto-docs/session.md b/proto-docs/session.md index c61855d..e1340c9 100644 --- a/proto-docs/session.md +++ b/proto-docs/session.md @@ -230,7 +230,7 @@ Session Token body ### Message SessionToken.Body.TokenLifetime -Lifetime parameters of the token. Filed names taken from rfc7519. +Lifetime parameters of the token. Field names taken from rfc7519. | Field | Type | Label | Description | diff --git a/proto-docs/tombstone.md b/proto-docs/tombstone.md new file mode 100644 index 0000000..e43a212 --- /dev/null +++ b/proto-docs/tombstone.md @@ -0,0 +1,63 @@ +# Protocol Documentation + + +## Table of Contents + +- [tombstone/types.proto](#tombstone/types.proto) + + - Messages + - [Tombstone](#neo.fs.v2.tombstone.Tombstone) + + +- [Scalar Value Types](#scalar-value-types) + + + + +

Top

+ +## tombstone/types.proto + + + + + + + +### Message Tombstone +Tombstone keeps record of deleted objects for few epochs until they are +purged from the NeoFS network. + + +| Field | Type | Label | Description | +| ----- | ---- | ----- | ----------- | +| expiration_epoch | [uint64](#uint64) | | Last NeoFS epoch number of the tombstone lifetime. It's set by tombstone creator depending on current NeoFS network settings. | +| split_id | [bytes](#bytes) | | 16 byte UUID used to identify the split object hierarchy parts. Must be unique inside container. All objects participating in the split must have the same `split_id` value. | +| members | [neo.fs.v2.refs.ObjectID](#neo.fs.v2.refs.ObjectID) | repeated | List of objects to be deleted. | + + + + + + + +## Scalar Value Types + +| .proto Type | Notes | C++ Type | Java Type | Python Type | +| ----------- | ----- | -------- | --------- | ----------- | +| double | | double | double | float | +| float | | float | float | float | +| int32 | Uses variable-length encoding. Inefficient for encoding negative numbers – if your field is likely to have negative values, use sint32 instead. | int32 | int | int | +| int64 | Uses variable-length encoding. Inefficient for encoding negative numbers – if your field is likely to have negative values, use sint64 instead. | int64 | long | int/long | +| uint32 | Uses variable-length encoding. | uint32 | int | int/long | +| uint64 | Uses variable-length encoding. | uint64 | long | int/long | +| sint32 | Uses variable-length encoding. Signed int value. These more efficiently encode negative numbers than regular int32s. | int32 | int | int | +| sint64 | Uses variable-length encoding. Signed int value. These more efficiently encode negative numbers than regular int64s. | int64 | long | int/long | +| fixed32 | Always four bytes. More efficient than uint32 if values are often greater than 2^28. | uint32 | int | int | +| fixed64 | Always eight bytes. More efficient than uint64 if values are often greater than 2^56. | uint64 | long | int/long | +| sfixed32 | Always four bytes. | int32 | int | int | +| sfixed64 | Always eight bytes. | int64 | long | int/long | +| bool | | bool | boolean | boolean | +| string | A string must always contain UTF-8 encoded or 7-bit ASCII text. | string | String | str/unicode | +| bytes | May contain any arbitrary sequence of bytes. | string | ByteString | str | +