frostfs-api/proto-docs/object.md
Alex Vanin 5916eb1170 [#80] Clarify well-known object filters parent and childfree
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-23 18:17:28 +03:00

34 KiB

Protocol Documentation

Table of Contents

Top

object/service.proto

Service "neo.fs.v2.object.ObjectService"

ObjectService provides API for manipulating objects. Object operations do not interact with sidechain and are only served by nodes in p2p style.

rpc Get(GetRequest) returns (stream GetResponse);
rpc Put(stream PutRequest) returns (PutResponse);
rpc Delete(DeleteRequest) returns (DeleteResponse);
rpc Head(HeadRequest) returns (HeadResponse);
rpc Search(SearchRequest) returns (stream SearchResponse);
rpc GetRange(GetRangeRequest) returns (stream GetRangeResponse);
rpc GetRangeHash(GetRangeHashRequest) returns (GetRangeHashResponse);

Method Get

Receive full object structure, including Headers and payload. Response uses gRPC stream. First response message carries object with requested address. Chunk messages are parts of the object's payload if it is needed. All messages, except the first one, carry payload chunks. Requested object can be restored by concatenation of object message payload and all chunks keeping receiving order.

Name Input Output
Get GetRequest GetResponse

Method Put

Put the object into container. Request uses gRPC stream. First message SHOULD be of PutHeader type. ContainerID and OwnerID of an object SHOULD be set. Session token SHOULD be obtained before PUT operation (see session package). Chunk messages are considered by server as a part of an object payload. All messages, except first one, SHOULD be payload chunks. Chunk messages SHOULD be sent in direct order of fragmentation.

Name Input Output
Put PutRequest PutResponse

Method Delete

Delete the object from a container. There is no immediate removal guarantee. Object will be marked for removal and deleted eventually.

Name Input Output
Delete DeleteRequest DeleteResponse

Method Head

Returns the object Headers without data payload. By default full header is returned. If main_only request field is set, the short header with only the very minimal information would be returned instead.

Name Input Output
Head HeadRequest HeadResponse

Search objects in container. Search query allows to match by Object Header's filed values. Please see the corresponding NeoFS Technical Specification section for more details.

Name Input Output
Search SearchRequest SearchResponse

Method GetRange

Get byte range of data payload. Range is set as an (offset, length) tuple. Like in Get method, the response uses gRPC stream. Requested range can be restored by concatenation of all received payload chunks keeping receiving order.

Name Input Output
GetRange GetRangeRequest GetRangeResponse

Method GetRangeHash

Returns homomorphic or regular hash of object's payload range after applying XOR operation with the provided salt. Ranges are set of (offset, length) tuples. Hashes order in response corresponds to ranges order in request. Note that hash is calculated for XORed data.

Name Input Output
GetRangeHash GetRangeHashRequest GetRangeHashResponse

Message DeleteRequest

Object DELETE request

Field Type Label Description
body DeleteRequest.Body Body of delete object request message.
meta_header neo.fs.v2.session.RequestMetaHeader Carries request meta information. Header data is used only to regulate message transport and does not affect request execution.
verify_header neo.fs.v2.session.RequestVerificationHeader Carries request verification information. This header is used to authenticate the nodes of the message route and check the correctness of transmission.

Message DeleteRequest.Body

Object DELETE request body

Field Type Label Description
address neo.fs.v2.refs.Address Address of the object to be deleted

Message DeleteResponse

DeleteResponse body is empty because we cannot guarantee permanent object removal in distributed system.

Field Type Label Description
body DeleteResponse.Body Body of delete object response message.
meta_header neo.fs.v2.session.ResponseMetaHeader Carries response meta information. Header data is used only to regulate message transport and does not affect request execution.
verify_header neo.fs.v2.session.ResponseVerificationHeader Carries response verification information. This header is used to authenticate the nodes of the message route and check the correctness of transmission.

Message DeleteResponse.Body

Object DELETE Response has an empty body.

Message GetRangeHashRequest

Get hash of object's payload part

Field Type Label Description
body GetRangeHashRequest.Body Body of get range hash object request message.
meta_header neo.fs.v2.session.RequestMetaHeader Carries request meta information. Header data is used only to regulate message transport and does not affect request execution.
verify_header neo.fs.v2.session.RequestVerificationHeader Carries request verification information. This header is used to authenticate the nodes of the message route and check the correctness of transmission.

Message GetRangeHashRequest.Body

Get hash of object's payload part request body.

Field Type Label Description
address neo.fs.v2.refs.Address Address of the object that containing the requested payload range
ranges Range repeated List of object's payload ranges to calculate homomorphic hash
salt bytes Binary salt to XOR object's payload ranges before hash calculation
type neo.fs.v2.refs.ChecksumType Checksum algorithm type

Message GetRangeHashResponse

Get hash of object's payload part

Field Type Label Description
body GetRangeHashResponse.Body Body of get range hash object response message.
meta_header neo.fs.v2.session.ResponseMetaHeader Carries response meta information. Header data is used only to regulate message transport and does not affect request execution.
verify_header neo.fs.v2.session.ResponseVerificationHeader Carries response verification information. This header is used to authenticate the nodes of the message route and check the correctness of transmission.

Message GetRangeHashResponse.Body

Get hash of object's payload part response body.

Field Type Label Description
type neo.fs.v2.refs.ChecksumType Checksum algorithm type
hash_list bytes repeated List of range hashes in a binary format

Message GetRangeRequest

Request part of object's payload

Field Type Label Description
body GetRangeRequest.Body Body of get range object request message.
meta_header neo.fs.v2.session.RequestMetaHeader Carries request meta information. Header data is used only to regulate message transport and does not affect request execution.
verify_header neo.fs.v2.session.RequestVerificationHeader Carries request verification information. This header is used to authenticate the nodes of the message route and check the correctness of transmission.

Message GetRangeRequest.Body

Byte range of object's payload request body

Field Type Label Description
address neo.fs.v2.refs.Address Address of the object containing the requested payload range
range Range Requested payload range

Message GetRangeResponse

Get part of object's payload

Field Type Label Description
body GetRangeResponse.Body Body of get range object response message.
meta_header neo.fs.v2.session.ResponseMetaHeader Carries response meta information. Header data is used only to regulate message transport and does not affect request execution.
verify_header neo.fs.v2.session.ResponseVerificationHeader Carries response verification information. This header is used to authenticate the nodes of the message route and check the correctness of transmission.

Message GetRangeResponse.Body

Get Range response body uses streams to transfer the response. Because object payload considered a byte sequence, there is no need to have some initial preamble message. The requested byte range is sent as a series chunks.

Field Type Label Description
chunk bytes Chunked object payload's range

Message GetRequest

GET object request

Field Type Label Description
body GetRequest.Body Body of get object request message.
meta_header neo.fs.v2.session.RequestMetaHeader Carries request meta information. Header data is used only to regulate message transport and does not affect request execution.
verify_header neo.fs.v2.session.RequestVerificationHeader Carries request verification information. This header is used to authenticate the nodes of the message route and check the correctness of transmission.

Message GetRequest.Body

GET Object request body

Field Type Label Description
address neo.fs.v2.refs.Address Address of the requested object
raw bool If raw flag is set, request will work only with objects that are physically stored on the peer node

Message GetResponse

GET object response

Field Type Label Description
body GetResponse.Body Body of get object response message.
meta_header neo.fs.v2.session.ResponseMetaHeader Carries response meta information. Header data is used only to regulate message transport and does not affect request execution.
verify_header neo.fs.v2.session.ResponseVerificationHeader Carries response verification information. This header is used to authenticate the nodes of the message route and check the correctness of transmission.

Message GetResponse.Body

GET Object Response body

Field Type Label Description
init GetResponse.Body.Init Initial part of the object stream
chunk bytes Chunked object payload

Message GetResponse.Body.Init

Initial part of the Object structure stream. Technically it's a set of all Object structure's fields except payload.

Field Type Label Description
object_id neo.fs.v2.refs.ObjectID Object's unique identifier.
signature neo.fs.v2.refs.Signature Signed ObjectID
header Header Object metadata headers

Message HeadRequest

Object HEAD request

Field Type Label Description
body HeadRequest.Body Body of head object request message.
meta_header neo.fs.v2.session.RequestMetaHeader Carries request meta information. Header data is used only to regulate message transport and does not affect request execution.
verify_header neo.fs.v2.session.RequestVerificationHeader Carries request verification information. This header is used to authenticate the nodes of the message route and check the correctness of transmission.

Message HeadRequest.Body

Object HEAD request body

Field Type Label Description
address neo.fs.v2.refs.Address Address of the object with the requested Header
main_only bool Return only minimal header subset
raw bool If raw flag is set, request will work only with objects that are physically stored on the peer node

Message HeadResponse

Object HEAD response

Field Type Label Description
body HeadResponse.Body Body of head object response message.
meta_header neo.fs.v2.session.ResponseMetaHeader Carries response meta information. Header data is used only to regulate message transport and does not affect request execution.
verify_header neo.fs.v2.session.ResponseVerificationHeader Carries response verification information. This header is used to authenticate the nodes of the message route and check the correctness of transmission.

Message HeadResponse.Body

Object HEAD response body

Field Type Label Description
header HeaderWithSignature Full object's Header with ObjectID signature
short_header ShortHeader Short object header

Message HeaderWithSignature

Tuple of full object header and signature of ObjectID.
Signed ObjectID is present to verify full header's authenticity through the following steps:

  1. Calculate SHA-256 of marshalled Header structure
  2. Check if the resulting hash matched ObjectID
  3. Check if ObjectID signature in signature field is correct
Field Type Label Description
header Header Full object header
signature neo.fs.v2.refs.Signature Signed ObjectID to verify full header's authenticity

Message PutRequest

PUT object request

Field Type Label Description
body PutRequest.Body Body of put object request message.
meta_header neo.fs.v2.session.RequestMetaHeader Carries request meta information. Header data is used only to regulate message transport and does not affect request execution.
verify_header neo.fs.v2.session.RequestVerificationHeader Carries request verification information. This header is used to authenticate the nodes of the message route and check the correctness of transmission.

Message PutRequest.Body

PUT request body

Field Type Label Description
init PutRequest.Body.Init Initial part of the object stream
chunk bytes Chunked object payload

Message PutRequest.Body.Init

Newly created object structure parameters. If some optional parameters are not set, they will be calculated by a peer node.

Field Type Label Description
object_id neo.fs.v2.refs.ObjectID ObjectID if available.
signature neo.fs.v2.refs.Signature Object signature if available
header Header Object's Header
copies_number uint32 Number of the object copies to store within the RPC call. By default object is processed according to the container's placement policy.

Message PutResponse

PUT Object response

Field Type Label Description
body PutResponse.Body Body of put object response message.
meta_header neo.fs.v2.session.ResponseMetaHeader Carries response meta information. Header data is used only to regulate message transport and does not affect request execution.
verify_header neo.fs.v2.session.ResponseVerificationHeader Carries response verification information. This header is used to authenticate the nodes of the message route and check the correctness of transmission.

Message PutResponse.Body

PUT Object response body

Field Type Label Description
object_id neo.fs.v2.refs.ObjectID Identifier of the saved object

Message Range

Object payload range.Ranges of zero length SHOULD be considered as invalid.

Field Type Label Description
offset uint64 Offset of the range from the object payload start
length uint64 Length in bytes of the object payload range

Message SearchRequest

Object Search request

Field Type Label Description
body SearchRequest.Body Body of search object request message.
meta_header neo.fs.v2.session.RequestMetaHeader Carries request meta information. Header data is used only to regulate message transport and does not affect request execution.
verify_header neo.fs.v2.session.RequestVerificationHeader Carries request verification information. This header is used to authenticate the nodes of the message route and check the correctness of transmission.

Message SearchRequest.Body

Object Search request body

Field Type Label Description
container_id neo.fs.v2.refs.ContainerID Container identifier were to search
version uint32 Version of the Query Language used
filters SearchRequest.Body.Filter repeated List of search expressions

Message SearchRequest.Body.Filter

Filter structure checks if object header field or attribute content matches a value.

By default key field refers to the corresponding object's Attribute. Some Object's header fields can also be accessed by adding $Object: prefix to the name. Here is the list of fields available via this prefix:

  • $Object:version
    version
  • $Object:containerID
    container_id
  • $Object:ownerID
    owner_id
  • $Object:creationEpoch
    creation_epoch
  • $Object:payloadLength
    payload_length
  • $Object:payloadHash
    payload_hash
  • $Object:objectType
    object_type
  • $Object:homomorphicHash
    homomorphic_hash
  • $Object:split.parent
    object_id of parent

There are some well-known filter aliases to match objects by certain properties:

  • $Object:ROOT
    With the value set to true checks if an object is a top object in a split hierarchy. With other values returns non-root objects.
  • $Object:LEAF
    With the value set to true checks if an object is a leaf in a split hierarchy. With other values returns non-leaf objects.
  • $Object:CHILDFREE
    With the value set to true checks if an object has empty children list in Split header. With other values returns non-childfree objects.
Field Type Label Description
match_type MatchType Match type to use
key string Attribute or Header fields to match
value string Value to match

Message SearchResponse

Search response

Field Type Label Description
body SearchResponse.Body Body of search object response message.
meta_header neo.fs.v2.session.ResponseMetaHeader Carries response meta information. Header data is used only to regulate message transport and does not affect request execution.
verify_header neo.fs.v2.session.ResponseVerificationHeader Carries response verification information. This header is used to authenticate the nodes of the message route and check the correctness of transmission.

Message SearchResponse.Body

Object Search response body

Field Type Label Description
id_list neo.fs.v2.refs.ObjectID repeated List of ObjectIDs that match the search query

Top

object/types.proto

Message Header

Object Header

Field Type Label Description
version neo.fs.v2.refs.Version Object format version. Effectively the version of API library used to create particular object
container_id neo.fs.v2.refs.ContainerID Object's container
owner_id neo.fs.v2.refs.OwnerID Object's owner
creation_epoch uint64 Object creation Epoch
payload_length uint64 Size of payload in bytes. 0xFFFFFFFFFFFFFFFF means payload_length is unknown
payload_hash neo.fs.v2.refs.Checksum Hash of payload bytes
object_type ObjectType Type of the object payload content
homomorphic_hash neo.fs.v2.refs.Checksum Homomorphic hash of the object payload.
session_token neo.fs.v2.session.SessionToken Session token, if it was used during Object creation. Need it to verify integrity and authenticity out of Request scope.
attributes Header.Attribute repeated User-defined object attributes
split Header.Split Position of the object in the split hierarchy

Message Header.Attribute

Attribute is a user-defined Key-Value metadata pair attached to the object.

There are some "well-known" attributes starting with __NEOFS__ prefix that affect system behaviour:

  • __NEOFS__UPLOAD_ID
    Marks smaller parts of a split bigger object
  • __NEOFS__EXPIRATION_EPOCH
    Tells GC to delete object after that epoch

And some well-known attributes used by applications only:

  • Name
    Human-friendly name
  • FileName
    File name to be associated with the object on saving
  • Timestamp
    User-defined local time of object creation in Unix Timestamp format

For detailed description of each well-known attribute please see the corresponding section in NeoFS Technical specification.

Field Type Label Description
key string string key to the object attribute
value string string value of the object attribute

Message Header.Split

Bigger objects can be split into a chain of smaller objects. Information about inter-dependencies between spawned objects and how to re-construct the original one is in the Split headers. Parent and children objects must be within the same container.

Field Type Label Description
parent neo.fs.v2.refs.ObjectID Identifier of the origin object. Known only to the minor child.
previous neo.fs.v2.refs.ObjectID Identifier of the left split neighbor
parent_signature neo.fs.v2.refs.Signature signature field of the parent object. Used to reconstruct parent.
parent_header Header header field of the parent object. Used to reconstruct parent.
children neo.fs.v2.refs.ObjectID repeated List of identifiers of the objects generated by splitting current one.

Message Object

Object structure. Object is immutable and content-addressed. It means ObjectID will change if header or payload changes. It's calculated as a hash of header field, which contains hash of object's payload.

Field Type Label Description
object_id neo.fs.v2.refs.ObjectID Object's unique identifier.
signature neo.fs.v2.refs.Signature Signed object_id
header Header Object metadata headers
payload bytes Payload bytes.

Message ShortHeader

Short header fields

Field Type Label Description
version neo.fs.v2.refs.Version Object format version. Effectively the version of API library used to create particular object
creation_epoch uint64 Epoch when the object was created
owner_id neo.fs.v2.refs.OwnerID Object's owner
object_type ObjectType Type of the object payload content
payload_length uint64 Size of payload in bytes. 0xFFFFFFFFFFFFFFFF means payload_length is unknown

MatchType

Type of match expression

Name Number Description
MATCH_TYPE_UNSPECIFIED 0 Unknown. Not used
STRING_EQUAL 1 Full string match

ObjectType

Type of the object payload content.

Name Number Description
REGULAR 0 Just a normal object
TOMBSTONE 1 Used internally to identify deleted objects
STORAGE_GROUP 2 StorageGroup information

Scalar Value Types

.proto Type Notes C++ Type Java Type Python Type
double double double float
float float float float
int32 Uses variable-length encoding. Inefficient for encoding negative numbers – if your field is likely to have negative values, use sint32 instead. int32 int int
int64 Uses variable-length encoding. Inefficient for encoding negative numbers – if your field is likely to have negative values, use sint64 instead. int64 long int/long
uint32 Uses variable-length encoding. uint32 int int/long
uint64 Uses variable-length encoding. uint64 long int/long
sint32 Uses variable-length encoding. Signed int value. These more efficiently encode negative numbers than regular int32s. int32 int int
sint64 Uses variable-length encoding. Signed int value. These more efficiently encode negative numbers than regular int64s. int64 long int/long
fixed32 Always four bytes. More efficient than uint32 if values are often greater than 2^28. uint32 int int
fixed64 Always eight bytes. More efficient than uint64 if values are often greater than 2^56. uint64 long int/long
sfixed32 Always four bytes. int32 int int
sfixed64 Always eight bytes. int64 long int/long
bool bool boolean boolean
string A string must always contain UTF-8 encoded or 7-bit ASCII text. string String str/unicode
bytes May contain any arbitrary sequence of bytes. string ByteString str