frostfs-api/proto-docs/object.md
Stanislav Bogatyrev 4e908a17b1 [#50] Add version fields where it makes sense
Some pieces of data (container, object, etc) may be stored for a long time and
there will be a need in the future to understand which obscure format from the
past was used to create it.

Signed-off-by: Stanislav Bogatyrev <stanislav@nspcc.ru>
2020-08-17 12:12:30 +03:00

30 KiB

Protocol Documentation

Table of Contents

Top

object/service.proto

Service "neo.fs.v2.object.ObjectService"

Object service provides API for manipulating with the object.

rpc Get(GetRequest) returns (stream GetResponse);
rpc Put(stream PutRequest) returns (PutResponse);
rpc Delete(DeleteRequest) returns (DeleteResponse);
rpc Head(HeadRequest) returns (HeadResponse);
rpc Search(SearchRequest) returns (stream SearchResponse);
rpc GetRange(GetRangeRequest) returns (stream GetRangeResponse);
rpc GetRangeHash(GetRangeHashRequest) returns (GetRangeHashResponse);

Method Get

Get the object from container. Response uses gRPC stream. First response message carry object of requested address. Chunk messages are parts of the object's payload if it is needed. All messages except first carry chunks. Requested object can be restored by concatenation of object message payload and all chunks keeping receiving order.

Name Input Output
Get GetRequest GetResponse

Method Put

Put the object into container. Request uses gRPC stream. First message SHOULD BE type of PutHeader. Container id and Owner id of object SHOULD BE set. Session token SHOULD BE obtained before put operation (see session package). Chunk messages considered by server as part of object payload. All messages except first SHOULD BE chunks. Chunk messages SHOULD BE sent in direct order of fragmentation.

Name Input Output
Put PutRequest PutResponse

Method Delete

Delete the object from a container

Name Input Output
Delete DeleteRequest DeleteResponse

Method Head

Head returns the object without data payload. Object in the response has system header only. If full headers flag is set, extended headers are also present.

Name Input Output
Head HeadRequest HeadResponse

Search objects in container. Version of query language format SHOULD BE set to 1. Search query represented in serialized format (see query package).

Name Input Output
Search SearchRequest SearchResponse

Method GetRange

GetRange of data payload. Range is a pair (offset, length). Requested range can be restored by concatenation of all chunks keeping receiving order.

Name Input Output
GetRange GetRangeRequest GetRangeResponse

Method GetRangeHash

GetRangeHash returns homomorphic hash of object payload range after XOR operation. Ranges are set of pairs (offset, length). Hashes order in response corresponds to ranges order in request. Homomorphic hash is calculated for XORed data.

Name Input Output
GetRangeHash GetRangeHashRequest GetRangeHashResponse

Message DeleteRequest

Object Delete request

Field Type Label Description
body DeleteRequest.Body Body of delete object request message.
meta_header neo.fs.v2.service.RequestMetaHeader Carries request meta information. Header data is used only to regulate message transport and does not affect request execution.
verify_header neo.fs.v2.service.RequestVerificationHeader Carries request verification information. This header is used to authenticate the nodes of the message route and check the correctness of transmission.

Message DeleteRequest.Body

Request body

Field Type Label Description
address neo.fs.v2.refs.Address Carries the address of the object to be deleted.
owner_id neo.fs.v2.refs.OwnerID Carries identifier the object owner.

Message DeleteResponse

DeleteResponse is empty because we cannot guarantee permanent object removal in distributed system.

Field Type Label Description
body DeleteResponse.Body Body of delete object response message.
meta_header neo.fs.v2.service.ResponseMetaHeader Carries response meta information. Header data is used only to regulate message transport and does not affect request execution.
verify_header neo.fs.v2.service.ResponseVerificationHeader Carries response verification information. This header is used to authenticate the nodes of the message route and check the correctness of transmission.

Message DeleteResponse.Body

Response body

Message GetRangeHashRequest

Get hash of object's payload part

Field Type Label Description
body GetRangeHashRequest.Body Body of get range hash object request message.
meta_header neo.fs.v2.service.RequestMetaHeader Carries request meta information. Header data is used only to regulate message transport and does not affect request execution.
verify_header neo.fs.v2.service.RequestVerificationHeader Carries request verification information. This header is used to authenticate the nodes of the message route and check the correctness of transmission.

Message GetRangeHashRequest.Body

Request body

Field Type Label Description
address neo.fs.v2.refs.Address Carries address of the object that contains the requested payload range.
ranges Range repeated Carries the list of object payload range to calculate homomorphic hash.
salt bytes Carries binary salt to XOR object payload ranges before hash calculation.

Message GetRangeHashResponse

Get hash of object's payload part

Field Type Label Description
body GetRangeHashResponse.Body Body of get range hash object response message.
meta_header neo.fs.v2.service.ResponseMetaHeader Carries response meta information. Header data is used only to regulate message transport and does not affect request execution.
verify_header neo.fs.v2.service.ResponseVerificationHeader Carries response verification information. This header is used to authenticate the nodes of the message route and check the correctness of transmission.

Message GetRangeHashResponse.Body

Response body

Field Type Label Description
hash_list bytes repeated Carries list of homomorphic hashes in a binary format.

Message GetRangeRequest

Request to get part of object's payload

Field Type Label Description
body GetRangeRequest.Body Body of get range object request message.
meta_header neo.fs.v2.service.RequestMetaHeader Carries request meta information. Header data is used only to regulate message transport and does not affect request execution.
verify_header neo.fs.v2.service.RequestVerificationHeader Carries request verification information. This header is used to authenticate the nodes of the message route and check the correctness of transmission.

Message GetRangeRequest.Body

Request Body

Field Type Label Description
address neo.fs.v2.refs.Address Address carries address of the object that contains the requested payload range.
range Range Range carries the parameters of the requested payload range.

Message GetRangeResponse

Get part of object's payload

Field Type Label Description
body GetRangeResponse.Body Body of get range object response message.
meta_header neo.fs.v2.service.ResponseMetaHeader Carries response meta information. Header data is used only to regulate message transport and does not affect request execution.
verify_header neo.fs.v2.service.ResponseVerificationHeader Carries response verification information. This header is used to authenticate the nodes of the message route and check the correctness of transmission.

Message GetRangeResponse.Body

Response body

Field Type Label Description
chunk bytes Carries part of the object payload.

Message GetRequest

Get object request

Field Type Label Description
body GetRequest.Body Body of get object request message.
meta_header neo.fs.v2.service.RequestMetaHeader Carries request meta information. Header data is used only to regulate message transport and does not affect request execution.
verify_header neo.fs.v2.service.RequestVerificationHeader Carries request verification information. This header is used to authenticate the nodes of the message route and check the correctness of transmission.

Message GetRequest.Body

Request body

Field Type Label Description
address neo.fs.v2.refs.Address Address of the requested object.
raw bool Carries the raw option flag of the request. Raw request is sent to receive only the objects that are physically stored on the server.

Message GetResponse

get object response

Field Type Label Description
body GetResponse.Body Body of get object response message.
meta_header neo.fs.v2.service.ResponseMetaHeader Carries response meta information. Header data is used only to regulate message transport and does not affect request execution.
verify_header neo.fs.v2.service.ResponseVerificationHeader Carries response verification information. This header is used to authenticate the nodes of the message route and check the correctness of transmission.

Message GetResponse.Body

Response body

Field Type Label Description
init GetResponse.Body.Init Initialization parameters of the object stream.
chunk bytes Part of the object payload.

Message GetResponse.Body.Init

Initialization parameters of the object got from NeoFS.

Field Type Label Description
object_id neo.fs.v2.refs.ObjectID Object ID
signature neo.fs.v2.service.Signature Object signature
header Header Object header.

Message HeadRequest

Object head request

Field Type Label Description
body HeadRequest.Body Body of head object request message.
meta_header neo.fs.v2.service.RequestMetaHeader Carries request meta information. Header data is used only to regulate message transport and does not affect request execution.
verify_header neo.fs.v2.service.RequestVerificationHeader Carries request verification information. This header is used to authenticate the nodes of the message route and check the correctness of transmission.

Message HeadRequest.Body

Request body

Field Type Label Description
address neo.fs.v2.refs.Address Address of the object with the requested header.
main_only bool Return only minimal header subset
raw bool Carries the raw option flag of the request. Raw request is sent to receive only the headers of the objects that are physically stored on the server.

Message HeadResponse

Head response

Field Type Label Description
body HeadResponse.Body Body of head object response message.
meta_header neo.fs.v2.service.ResponseMetaHeader Carries response meta information. Header data is used only to regulate message transport and does not affect request execution.
verify_header neo.fs.v2.service.ResponseVerificationHeader Carries response verification information. This header is used to authenticate the nodes of the message route and check the correctness of transmission.

Message HeadResponse.Body

Response body

Field Type Label Description
header Header Full object header
short_header ShortHeader Short object header

Message PutRequest

Put object request

Field Type Label Description
body PutRequest.Body Body of put object request message.
meta_header neo.fs.v2.service.RequestMetaHeader Carries request meta information. Header data is used only to regulate message transport and does not affect request execution.
verify_header neo.fs.v2.service.RequestVerificationHeader Carries request verification information. This header is used to authenticate the nodes of the message route and check the correctness of transmission.

Message PutRequest.Body

Request body

Field Type Label Description
init PutRequest.Body.Init Carries the initialization parameters of the object stream.
chunk bytes Carries part of the object payload.

Message PutRequest.Body.Init

Groups initialization parameters of object placement in NeoFS.

Field Type Label Description
object_id neo.fs.v2.refs.ObjectID Object ID, where available
signature neo.fs.v2.service.Signature Object signature, were available
header Header Header of the object to save in the system.
copies_number uint32 Number of the object copies to store within the RPC call. Default zero value is processed according to the container placement rules.

Message PutResponse

Put object response

Field Type Label Description
body PutResponse.Body Body of put object response message.
meta_header neo.fs.v2.service.ResponseMetaHeader Carries response meta information. Header data is used only to regulate message transport and does not affect request execution.
verify_header neo.fs.v2.service.ResponseVerificationHeader Carries response verification information. This header is used to authenticate the nodes of the message route and check the correctness of transmission.

Message PutResponse.Body

Response body

Field Type Label Description
object_id neo.fs.v2.refs.ObjectID Carries identifier of the saved object. It is used to access an object in the container.

Message Range

Range groups the parameters of object payload range.

Field Type Label Description
offset uint64 Carries the offset of the range from the object payload start.
length uint64 Carries the length of the object payload range.

Message SearchRequest

Search objects request

Field Type Label Description
body SearchRequest.Body Body of search object request message.
meta_header neo.fs.v2.service.RequestMetaHeader Carries request meta information. Header data is used only to regulate message transport and does not affect request execution.
verify_header neo.fs.v2.service.RequestVerificationHeader Carries request verification information. This header is used to authenticate the nodes of the message route and check the correctness of transmission.

Message SearchRequest.Body

Request body

Field Type Label Description
container_id neo.fs.v2.refs.ContainerID Carries search container identifier.
version uint32 Version of the Query Language used
filters SearchRequest.Body.Filter repeated List of search expressions

Message SearchRequest.Body.Filter

Filter structure

Field Type Label Description
match_type MatchType Match type to use
name string Header name to match
value string Header value to match

Message SearchResponse

Search response

Field Type Label Description
body SearchResponse.Body Body of search object response message.
meta_header neo.fs.v2.service.ResponseMetaHeader Carries response meta information. Header data is used only to regulate message transport and does not affect request execution.
verify_header neo.fs.v2.service.ResponseVerificationHeader Carries response verification information. This header is used to authenticate the nodes of the message route and check the correctness of transmission.

Message SearchResponse.Body

Response body

Field Type Label Description
id_list neo.fs.v2.refs.ObjectID repeated Carries list of object identifiers that match the search query

Top

object/types.proto

Message Header

Object Headers

Field Type Label Description
version neo.fs.v2.service.Version Object format version. Effectively the version of API library used to create particular object
container_id neo.fs.v2.refs.ContainerID Object's container
owner_id neo.fs.v2.refs.OwnerID Object's owner
creation_epoch uint64 Object creation Epoch
payload_length uint64 Size of payload in bytes. 0xFFFFFFFFFFFFFFFF means payload_length is unknown
payload_hash bytes Hash of payload bytes
object_type ObjectType Special object type
homomorphic_hash bytes Homomorphic hash of the object payload.
session_token neo.fs.v2.service.SessionToken Session token, if it was used during Object creation. Need it to verify integrity and authenticity out of Request scope.
attributes Header.Attribute repeated User-defined object attributes
split Header.Split Position of the object in the split hierarchy.

Message Header.Attribute

Attribute groups the user-defined Key-Value pairs attached to the object

Field Type Label Description
key string string key to the object attribute
value string string value of the object attribute

Message Header.Split

Information about spawning the objects through a payload splitting.

Field Type Label Description
parent neo.fs.v2.refs.ObjectID Identifier of the origin object. Parent and children objects must be within the same container. Parent object_id is known only to the minor child.
previous neo.fs.v2.refs.ObjectID Previous carries identifier of the left split neighbor.
parent_signature neo.fs.v2.service.Signature signature field of the parent object. Used to reconstruct parent.
parent_header Header header field of the parent object. Used to reconstruct parent.
children neo.fs.v2.refs.ObjectID repeated Children carries list of identifiers of the objects generated by splitting the current.

Message Object

Object structure.

Field Type Label Description
object_id neo.fs.v2.refs.ObjectID Object's unique identifier. Object is content-addressed. It means id will change if header or payload changes. It's calculated as a hash of header field, which contains hash of object's payload
signature neo.fs.v2.service.Signature Signed object_id
header Header Object metadata headers
payload bytes Payload bytes.

Message ShortHeader

Short header fields

Field Type Label Description
version neo.fs.v2.service.Version Object format version.
creation_epoch uint64 Epoch when the object was created
owner_id neo.fs.v2.refs.OwnerID Object's owner
object_type ObjectType Type of the object payload content
payload_length uint64 Size of payload in bytes. 0xFFFFFFFFFFFFFFFF means payload_length is unknown

MatchType

Type of match expression

Name Number Description
MATCH_TYPE_UNSPECIFIED 0 Unknown. Not used
STRING_EQUAL 1 Full string match

ObjectType

Type of the object payload content

Name Number Description
REGULAR 0 Just a normal object
TOMBSTONE 1 Used internally to identify deleted objects
STORAGE_GROUP 2 Identifies that the object holds StorageGroup information

Scalar Value Types

.proto Type Notes C++ Type Java Type Python Type
double double double float
float float float float
int32 Uses variable-length encoding. Inefficient for encoding negative numbers – if your field is likely to have negative values, use sint32 instead. int32 int int
int64 Uses variable-length encoding. Inefficient for encoding negative numbers – if your field is likely to have negative values, use sint64 instead. int64 long int/long
uint32 Uses variable-length encoding. uint32 int int/long
uint64 Uses variable-length encoding. uint64 long int/long
sint32 Uses variable-length encoding. Signed int value. These more efficiently encode negative numbers than regular int32s. int32 int int
sint64 Uses variable-length encoding. Signed int value. These more efficiently encode negative numbers than regular int64s. int64 long int/long
fixed32 Always four bytes. More efficient than uint32 if values are often greater than 2^28. uint32 int int
fixed64 Always eight bytes. More efficient than uint64 if values are often greater than 2^56. uint64 long int/long
sfixed32 Always four bytes. int32 int int
sfixed64 Always eight bytes. int64 long int/long
bool bool boolean boolean
string A string must always contain UTF-8 encoded or 7-bit ASCII text. string String str/unicode
bytes May contain any arbitrary sequence of bytes. string ByteString str