TrueCloudLab/frostfs-api
26
0
Fork 17
Code Issues 11 Pull requests 1 Releases Activity Actions
frostfs-api/proto-docs/object.md
Bruk Ori 527fe93e5d
Some checks failed
Formatters / Run fmt (pull_request) Successful in 33s
Details
DCO action / DCO (pull_request) Successful in 1m0s
Details
Pre-commit hooks / Pre-commit (pull_request) Failing after 1m1s
Details
[#68] Ensure compatibility of different API versions with each other. 
Remove unnecessary language options.
Update linters.
Signed-off-by: Ori Bruk <o.bruk@yadro.com>
2024-10-08 17:30:50 +03:00

58 KiB
Raw Blame History

Protocol Documentation

Table of Contents

Top

object/service.proto

Service "frost.fs.object.ObjectService"

ObjectService provides API for manipulating objects. Object operations do not affect the sidechain and are only served by nodes in p2p style.

rpc Get(GetRequest) returns (stream GetResponse);
rpc Put(stream PutRequest) returns (PutResponse);
rpc Delete(DeleteRequest) returns (DeleteResponse);
rpc Head(HeadRequest) returns (HeadResponse);
rpc Search(SearchRequest) returns (stream SearchResponse);
rpc GetRange(GetRangeRequest) returns (stream GetRangeResponse);
rpc GetRangeHash(GetRangeHashRequest) returns (GetRangeHashResponse);
rpc PutSingle(PutSingleRequest) returns (PutSingleResponse);
rpc Patch(stream PatchRequest) returns (PatchResponse);

Method Get

Receive full object structure, including Headers and payload. Response uses gRPC stream. First response message carries the object with the requested address. Chunk messages are parts of the object's payload if it is needed. All messages, except the first one, carry payload chunks. The requested object can be restored by concatenation of object message payload and all chunks keeping the receiving order.

Extended headers can change Get behaviour:

  • [ __SYSTEM__NETMAP_EPOCH ]
    Will use the requsted version of Network Map for object placement calculation.
  • [ __SYSTEM__NETMAP_LOOKUP_DEPTH ]
    Will try older versions (starting from __SYSTEM__NETMAP_EPOCH if specified or the latest one otherwise) of Network Map to find an object until the depth limit is reached.

Please refer to detailed XHeader description.

Statuses:

  • OK (0, SECTION_SUCCESS):
    object has been successfully read;
  • Common failures (SECTION_FAILURE_COMMON);
  • ACCESS_DENIED (2048, SECTION_OBJECT):
    read access to the object is denied;
  • OBJECT_NOT_FOUND (2049, SECTION_OBJECT):
    object not found in container;
  • OBJECT_ALREADY_REMOVED (2052, SECTION_OBJECT):
    the requested object has been marked as deleted;
  • CONTAINER_NOT_FOUND (3072, SECTION_CONTAINER):
    object container not found;
  • CONTAINER_ACCESS_DENIED (3074, SECTION_CONTAINER):
    access to container is denied;
  • TOKEN_EXPIRED (4097, SECTION_SESSION):
    provided session token has expired.
Name Input Output
Get GetRequest GetResponse

Method Put

Put the object into container. Request uses gRPC stream. First message SHOULD be of PutHeader type. ContainerID and OwnerID of an object SHOULD be set. Session token SHOULD be obtained before PUT operation (see session package). Chunk messages are considered by server as a part of an object payload. All messages, except first one, SHOULD be payload chunks. Chunk messages SHOULD be sent in the direct order of fragmentation.

Extended headers can change Put behaviour:

  • [ __SYSTEM__NETMAP_EPOCH
    Will use the requsted version of Network Map for object placement calculation.

Please refer to detailed XHeader description.

Statuses:

  • OK (0, SECTION_SUCCESS):
    object has been successfully saved in the container;
  • Common failures (SECTION_FAILURE_COMMON);
  • ACCESS_DENIED (2048, SECTION_OBJECT):
    write access to the container is denied;
  • LOCKED (2050, SECTION_OBJECT):
    placement of an object of type TOMBSTONE that includes at least one locked object is prohibited;
  • LOCK_NON_REGULAR_OBJECT (2051, SECTION_OBJECT):
    placement of an object of type LOCK that includes at least one object of type other than REGULAR is prohibited;
  • CONTAINER_NOT_FOUND (3072, SECTION_CONTAINER):
    object storage container not found;
  • CONTAINER_ACCESS_DENIED (3074, SECTION_CONTAINER):
    access to container is denied;
  • TOKEN_NOT_FOUND (4096, SECTION_SESSION):
    (for trusted object preparation) session private key does not exist or has been deleted;
  • TOKEN_EXPIRED (4097, SECTION_SESSION):
    provided session token has expired.
Name Input Output
Put PutRequest PutResponse

Method Delete

Delete the object from a container. There is no immediate removal guarantee. Object will be marked for removal and deleted eventually.

Extended headers can change Delete behaviour:

  • [ __SYSTEM__NETMAP_EPOCH ]
    Will use the requested version of Network Map for object placement calculation.

Please refer to detailed XHeader description.

Statuses:

  • OK (0, SECTION_SUCCESS):
    object has been successfully marked to be removed from the container;
  • Common failures (SECTION_FAILURE_COMMON);
  • ACCESS_DENIED (2048, SECTION_OBJECT):
    delete access to the object is denied;
  • OBJECT_NOT_FOUND (2049, SECTION_OBJECT):
    the object could not be deleted because it has not been
    found within the container;
  • LOCKED (2050, SECTION_OBJECT):
    deleting a locked object is prohibited;
  • CONTAINER_NOT_FOUND (3072, SECTION_CONTAINER):
    object container not found;
  • CONTAINER_ACCESS_DENIED (3074, SECTION_CONTAINER):
    access to container is denied;
  • TOKEN_EXPIRED (4097, SECTION_SESSION):
    provided session token has expired.
Name Input Output
Delete DeleteRequest DeleteResponse

Method Head

Returns the object Headers without data payload. By default full header is returned. If main_only request field is set, the short header with only the very minimal information will be returned instead.

Extended headers can change Head behaviour:

  • [ __SYSTEM__NETMAP_EPOCH ]
    Will use the requested version of Network Map for object placement calculation.

Please refer to detailed XHeader description.

Statuses:

  • OK (0, SECTION_SUCCESS):
    object header has been successfully read;
  • Common failures (SECTION_FAILURE_COMMON);
  • ACCESS_DENIED (2048, SECTION_OBJECT):
    access to operation HEAD of the object is denied;
  • OBJECT_NOT_FOUND (2049, SECTION_OBJECT):
    object not found in container;
  • OBJECT_ALREADY_REMOVED (2052, SECTION_OBJECT):
    the requested object has been marked as deleted;
  • CONTAINER_NOT_FOUND (3072, SECTION_CONTAINER):
    object container not found;
  • CONTAINER_ACCESS_DENIED (3074, SECTION_CONTAINER):
    access to container is denied;
  • TOKEN_EXPIRED (4097, SECTION_SESSION):
    provided session token has expired.
Name Input Output
Head HeadRequest HeadResponse

Method Search

Search objects in container. Search query allows to match by Object Header's filed values. Please see the corresponding FrostFS Technical Specification section for more details.

Extended headers can change Search behaviour:

  • [ __SYSTEM__NETMAP_EPOCH ]
    Will use the requested version of Network Map for object placement calculation.

Please refer to detailed XHeader description.

Statuses:

  • OK (0, SECTION_SUCCESS):
    objects have been successfully selected;
  • Common failures (SECTION_FAILURE_COMMON);
  • ACCESS_DENIED (2048, SECTION_OBJECT):
    access to operation SEARCH of the object is denied;
  • CONTAINER_NOT_FOUND (3072, SECTION_CONTAINER):
    search container not found;
  • CONTAINER_ACCESS_DENIED (3074, SECTION_CONTAINER):
    access to container is denied;
  • TOKEN_EXPIRED (4097, SECTION_SESSION):
    provided session token has expired.
Name Input Output
Search SearchRequest SearchResponse

Method GetRange

Get byte range of data payload. Range is set as an (offset, length) tuple. Like in Get method, the response uses gRPC stream. Requested range can be restored by concatenation of all received payload chunks keeping the receiving order.

Extended headers can change GetRange behaviour:

  • [ __SYSTEM__NETMAP_EPOCH ]
    Will use the requested version of Network Map for object placement calculation.
  • [ __SYSTEM__NETMAP_LOOKUP_DEPTH ]
    Will try older versions of Network Map to find an object until the depth limit is reached.

Please refer to detailed XHeader description.

Statuses:

  • OK (0, SECTION_SUCCESS):
    data range of the object payload has been successfully read;
  • Common failures (SECTION_FAILURE_COMMON);
  • ACCESS_DENIED (2048, SECTION_OBJECT):
    access to operation RANGE of the object is denied;
  • OBJECT_NOT_FOUND (2049, SECTION_OBJECT):
    object not found in container;
  • OBJECT_ALREADY_REMOVED (2052, SECTION_OBJECT):
    the requested object has been marked as deleted.
  • OUT_OF_RANGE (2053, SECTION_OBJECT):
    the requested range is out of bounds;
  • CONTAINER_NOT_FOUND (3072, SECTION_CONTAINER):
    object container not found;
  • CONTAINER_ACCESS_DENIED (3074, SECTION_CONTAINER):
    access to container is denied;
  • TOKEN_EXPIRED (4097, SECTION_SESSION):
    provided session token has expired.
Name Input Output
GetRange GetRangeRequest GetRangeResponse

Method GetRangeHash

Returns homomorphic or regular hash of object's payload range after applying XOR operation with the provided salt. Ranges are set of (offset, length) tuples. Hashes order in response corresponds to the ranges order in the request. Note that hash is calculated for XORed data.

Extended headers can change GetRangeHash behaviour:

  • [ __SYSTEM__NETMAP_EPOCH ]
    Will use the requested version of Network Map for object placement calculation.
  • [ __SYSTEM__NETMAP_LOOKUP_DEPTH ]
    Will try older versions of Network Map to find an object until the depth limit is reached.

Please refer to detailed XHeader description.

Statuses:

  • OK (0, SECTION_SUCCESS):
    data range of the object payload has been successfully hashed;
  • Common failures (SECTION_FAILURE_COMMON);
  • ACCESS_DENIED (2048, SECTION_OBJECT):
    access to operation RANGEHASH of the object is denied;
  • OBJECT_NOT_FOUND (2049, SECTION_OBJECT):
    object not found in container;
  • OUT_OF_RANGE (2053, SECTION_OBJECT):
    the requested range is out of bounds;
  • CONTAINER_NOT_FOUND (3072, SECTION_CONTAINER):
    object container not found;
  • CONTAINER_ACCESS_DENIED (3074, SECTION_CONTAINER):
    access to container is denied;
  • TOKEN_EXPIRED (4097, SECTION_SESSION):
    provided session token has expired.
Name Input Output
GetRangeHash GetRangeHashRequest GetRangeHashResponse

Method PutSingle

Put the prepared object into container. ContainerID, ObjectID, OwnerID, PayloadHash and PayloadLength of an object MUST be set.

Extended headers can change Put behaviour:

  • [ __SYSTEM__NETMAP_EPOCH
    Will use the requested version of Network Map for object placement calculation.

Please refer to detailed XHeader description.

Statuses:

  • OK (0, SECTION_SUCCESS):
    object has been successfully saved in the container;
  • Common failures (SECTION_FAILURE_COMMON);
  • ACCESS_DENIED (2048, SECTION_OBJECT):
    write access to the container is denied;
  • LOCKED (2050, SECTION_OBJECT):
    placement of an object of type TOMBSTONE that includes at least one locked object is prohibited;
  • LOCK_NON_REGULAR_OBJECT (2051, SECTION_OBJECT):
    placement of an object of type LOCK that includes at least one object of type other than REGULAR is prohibited;
  • CONTAINER_NOT_FOUND (3072, SECTION_CONTAINER):
    object storage container not found;
  • CONTAINER_ACCESS_DENIED (3074, SECTION_CONTAINER):
    access to container is denied;
  • TOKEN_NOT_FOUND (4096, SECTION_SESSION):
    (for trusted object preparation) session private key does not exist or has been deleted;
  • TOKEN_EXPIRED (4097, SECTION_SESSION):
    provided session token has expired.
Name Input Output
PutSingle PutSingleRequest PutSingleResponse

Method Patch

Patch the object. Request uses gRPC stream. First message must set the address of the object that is going to get patched. If the object's attributes are patched, then these attrubutes must be set only within the first stream message.

If the patch request is performed by NOT the object's owner but if the actor has the permission to perform the patch, then OwnerID of the object is changed. In this case the object's owner loses the object's ownership after the patch request is successfully done.

As objects are content-addressable the patching causes new object ID generation for the patched object. This object id is set witihn PatchResponse. But the object id may remain unchanged in such cases:

  1. The chunk of the applying patch contains the same value as the object's payload within the same range;
  2. The patch that reverts the changes applied by preceding patch;
  3. The application of the same patches for the object a few times.

Extended headers can change Patch behaviour:

  • [ __SYSTEM__NETMAP_EPOCH
    Will use the requsted version of Network Map for object placement calculation.

Please refer to detailed XHeader description.

Statuses:

  • OK (0, SECTION_SUCCESS):
    object has been successfully patched and saved in the container;
  • Common failures (SECTION_FAILURE_COMMON);
  • ACCESS_DENIED (2048, SECTION_OBJECT):
    write access to the container is denied;
  • OBJECT_NOT_FOUND (2049, SECTION_OBJECT):
    object not found in container;
  • OBJECT_ALREADY_REMOVED (2052, SECTION_OBJECT):
    the requested object has been marked as deleted.
  • OUT_OF_RANGE (2053, SECTION_OBJECT):
    the requested range is out of bounds;
  • CONTAINER_NOT_FOUND (3072, SECTION_CONTAINER):
    object storage container not found;
  • CONTAINER_ACCESS_DENIED (3074, SECTION_CONTAINER):
    access to container is denied;
  • TOKEN_NOT_FOUND (4096, SECTION_SESSION):
    (for trusted object preparation) session private key does not exist or has been deleted;
  • TOKEN_EXPIRED (4097, SECTION_SESSION):
    provided session token has expired.
Name Input Output
Patch PatchRequest PatchResponse

Message DeleteRequest

Object DELETE request

Field Type Label Description
body DeleteRequest.Body Body of delete object request message.
meta_header frost.fs.session.RequestMetaHeader Carries request meta information. Header data is used only to regulate message transport and does not affect request execution.
verify_header frost.fs.session.RequestVerificationHeader Carries request verification information. This header is used to authenticate the nodes of the message route and check the correctness of transmission.

Message DeleteRequest.Body

Object DELETE request body

Field Type Label Description
address frost.fs.refs.Address Address of the object to be deleted

Message DeleteResponse

DeleteResponse body is empty because we cannot guarantee permanent object removal in distributed system.

Field Type Label Description
body DeleteResponse.Body Body of delete object response message.
meta_header frost.fs.session.ResponseMetaHeader Carries response meta information. Header data is used only to regulate message transport and does not affect request execution.
verify_header frost.fs.session.ResponseVerificationHeader Carries response verification information. This header is used to authenticate the nodes of the message route and check the correctness of transmission.

Message DeleteResponse.Body

Object DELETE Response has an empty body.

Field Type Label Description
tombstone frost.fs.refs.Address Address of the tombstone created for the deleted object

Message GetRangeHashRequest

Get hash of object's payload part

Field Type Label Description
body GetRangeHashRequest.Body Body of get range hash object request message.
meta_header frost.fs.session.RequestMetaHeader Carries request meta information. Header data is used only to regulate message transport and does not affect request execution.
verify_header frost.fs.session.RequestVerificationHeader Carries request verification information. This header is used to authenticate the nodes of the message route and check the correctness of transmission.

Message GetRangeHashRequest.Body

Get hash of object's payload part request body.

Field Type Label Description
address frost.fs.refs.Address Address of the object that containing the requested payload range
ranges Range repeated List of object's payload ranges to calculate homomorphic hash
salt bytes Binary salt to XOR object's payload ranges before hash calculation
type frost.fs.refs.ChecksumType Checksum algorithm type

Message GetRangeHashResponse

Get hash of object's payload part

Field Type Label Description
body GetRangeHashResponse.Body Body of get range hash object response message.
meta_header frost.fs.session.ResponseMetaHeader Carries response meta information. Header data is used only to regulate message transport and does not affect request execution.
verify_header frost.fs.session.ResponseVerificationHeader Carries response verification information. This header is used to authenticate the nodes of the message route and check the correctness of transmission.

Message GetRangeHashResponse.Body

Get hash of object's payload part response body.

Field Type Label Description
type frost.fs.refs.ChecksumType Checksum algorithm type
hash_list bytes repeated List of range hashes in a binary format

Message GetRangeRequest

Request part of object's payload

Field Type Label Description
body GetRangeRequest.Body Body of get range object request message.
meta_header frost.fs.session.RequestMetaHeader Carries request meta information. Header data is used only to regulate message transport and does not affect request execution.
verify_header frost.fs.session.RequestVerificationHeader Carries request verification information. This header is used to authenticate the nodes of the message route and check the correctness of transmission.

Message GetRangeRequest.Body

Byte range of object's payload request body

Field Type Label Description
address frost.fs.refs.Address Address of the object containing the requested payload range
range Range Requested payload range
raw bool If raw flag is set, request will work only with objects that are physically stored on the peer node.

Message GetRangeResponse

Get part of object's payload

Field Type Label Description
body GetRangeResponse.Body Body of get range object response message.
meta_header frost.fs.session.ResponseMetaHeader Carries response meta information. Header data is used only to regulate message transport and does not affect request execution.
verify_header frost.fs.session.ResponseVerificationHeader Carries response verification information. This header is used to authenticate the nodes of the message route and check the correctness of transmission.

Message GetRangeResponse.Body

Get Range response body uses streams to transfer the response. Because object payload considered a byte sequence, there is no need to have some initial preamble message. The requested byte range is sent as a series chunks.

Field Type Label Description
chunk bytes Chunked object payload's range.
split_info SplitInfo Meta information of split hierarchy.
ec_info ECInfo Meta information for EC object assembly.

Message GetRequest

GET object request

Field Type Label Description
body GetRequest.Body Body of get object request message.
meta_header frost.fs.session.RequestMetaHeader Carries request meta information. Header data is used only to regulate message transport and does not affect request execution.
verify_header frost.fs.session.RequestVerificationHeader Carries request verification information. This header is used to authenticate the nodes of the message route and check the correctness of transmission.

Message GetRequest.Body

GET Object request body

Field Type Label Description
address frost.fs.refs.Address Address of the requested object
raw bool If raw flag is set, request will work only with objects that are physically stored on the peer node

Message GetResponse

GET object response

Field Type Label Description
body GetResponse.Body Body of get object response message.
meta_header frost.fs.session.ResponseMetaHeader Carries response meta information. Header data is used only to regulate message transport and does not affect request execution.
verify_header frost.fs.session.ResponseVerificationHeader Carries response verification information. This header is used to authenticate the nodes of the message route and check the correctness of transmission.

Message GetResponse.Body

GET Object Response body

Field Type Label Description
init GetResponse.Body.Init Initial part of the object stream
chunk bytes Chunked object payload
split_info SplitInfo Meta information of split hierarchy for object assembly.
ec_info ECInfo Meta information for EC object assembly.

Message GetResponse.Body.Init

Initial part of the Object structure stream. Technically it's a set of all Object structure's fields except payload.

Field Type Label Description
object_id frost.fs.refs.ObjectID Object's unique identifier.
signature frost.fs.refs.Signature Signed ObjectID
header Header Object metadata headers

Message HeadRequest

Object HEAD request

Field Type Label Description
body HeadRequest.Body Body of head object request message.
meta_header frost.fs.session.RequestMetaHeader Carries request meta information. Header data is used only to regulate message transport and does not affect request execution.
verify_header frost.fs.session.RequestVerificationHeader Carries request verification information. This header is used to authenticate the nodes of the message route and check the correctness of transmission.

Message HeadRequest.Body

Object HEAD request body

Field Type Label Description
address frost.fs.refs.Address Address of the object with the requested Header
main_only bool Return only minimal header subset
raw bool If raw flag is set, request will work only with objects that are physically stored on the peer node

Message HeadResponse

Object HEAD response

Field Type Label Description
body HeadResponse.Body Body of head object response message.
meta_header frost.fs.session.ResponseMetaHeader Carries response meta information. Header data is used only to regulate message transport and does not affect request execution.
verify_header frost.fs.session.ResponseVerificationHeader Carries response verification information. This header is used to authenticate the nodes of the message route and check the correctness of transmission.

Message HeadResponse.Body

Object HEAD response body

Field Type Label Description
header HeaderWithSignature Full object's Header with ObjectID signature
short_header ShortHeader Short object header
split_info SplitInfo Meta information of split hierarchy.
ec_info ECInfo Meta information for EC object assembly.

Message HeaderWithSignature

Tuple of a full object header and signature of an ObjectID.
Signed ObjectID is present to verify full header's authenticity through the following steps:

  1. Calculate SHA-256 of the marshalled Header structure
  2. Check if the resulting hash matches ObjectID
  3. Check if ObjectID signature in signature field is correct
Field Type Label Description
header Header Full object header
signature frost.fs.refs.Signature Signed ObjectID to verify full header's authenticity

Message PatchRequest

Object PATCH request

Field Type Label Description
body PatchRequest.Body Body for patch request message.
meta_header frost.fs.session.RequestMetaHeader Carries request meta information. Header data is used only to regulate message transport and does not affect request execution.
verify_header frost.fs.session.RequestVerificationHeader Carries request verification information. This header is used to authenticate the nodes of the message route and check the correctness of transmission.

Message PatchRequest.Body

PATCH request body

Field Type Label Description
address frost.fs.refs.Address The address of the object that is requested to get patched.
new_attributes Header.Attribute repeated New attributes for the object. See replace_attributes flag usage to define how new attributes should be set.
replace_attributes bool If this flag is set, then the object's attributes will be entirely replaced by new_attributes list. The empty new_attributes list with replace_attributes = true just resets attributes list for the object.

Default false value for this flag means the attributes will be just merged. If the incoming new_attributes list contains already existing key, then it just replaces it while merging the lists. | | patch | PatchRequest.Body.Patch | | The patch that is applied for the object. |

Message PatchRequest.Body.Patch

The patch for the object's payload.

Field Type Label Description
source_range Range The range of the source object for which the payload is replaced by the patch's chunk. If the range's length = 0, then the patch's chunk is just appended to the original payload starting from the offest without any replace.
chunk bytes The chunk that is being appended to or that replaces the original payload on the given range.

Message PatchResponse

Object PATCH response

Field Type Label Description
body PatchResponse.Body Body for patch response message.
meta_header frost.fs.session.ResponseMetaHeader Carries response meta information. Header data is used only to regulate message transport and does not affect request execution.
verify_header frost.fs.session.ResponseVerificationHeader Carries response verification information. This header is used to authenticate the nodes of the message route and check the correctness of transmission.

Message PatchResponse.Body

PATCH response body

Field Type Label Description
object_id frost.fs.refs.ObjectID The object ID of the saved patched object.

Message PutRequest

PUT object request

Field Type Label Description
body PutRequest.Body Body of put object request message.
meta_header frost.fs.session.RequestMetaHeader Carries request meta information. Header data is used only to regulate message transport and does not affect request execution.
verify_header frost.fs.session.RequestVerificationHeader Carries request verification information. This header is used to authenticate the nodes of the message route and check the correctness of transmission.

Message PutRequest.Body

PUT request body

Field Type Label Description
init PutRequest.Body.Init Initial part of the object stream
chunk bytes Chunked object payload

Message PutRequest.Body.Init

Newly created object structure parameters. If some optional parameters are not set, they will be calculated by a peer node.

Field Type Label Description
object_id frost.fs.refs.ObjectID ObjectID if available.
signature frost.fs.refs.Signature Object signature if available
header Header Object's Header
copies_number uint32 repeated Number of copies of the object to store within the RPC call. By default, object is processed according to the container's placement policy. Can be one of: 1. A single number; applied to the whole request and is treated as a minimal number of nodes that must store an object to complete the request successfully. 2. An ordered array; every number is treated as a minimal number of nodes in a corresponding placement vector that must store an object to complete the request successfully. The length MUST equal the placement vectors number, otherwise request is considered malformed.

Message PutResponse

PUT Object response

Field Type Label Description
body PutResponse.Body Body of put object response message.
meta_header frost.fs.session.ResponseMetaHeader Carries response meta information. Header data is used only to regulate message transport and does not affect request execution.
verify_header frost.fs.session.ResponseVerificationHeader Carries response verification information. This header is used to authenticate the nodes of the message route and check the correctness of transmission.

Message PutResponse.Body

PUT Object response body

Field Type Label Description
object_id frost.fs.refs.ObjectID Identifier of the saved object

Message PutSingleRequest

Object PUT Single request

Field Type Label Description
body PutSingleRequest.Body Body of put single object request message.
meta_header frost.fs.session.RequestMetaHeader Carries request meta information. Header data is used only to regulate message transport and does not affect request execution.
verify_header frost.fs.session.RequestVerificationHeader Carries request verification information. This header is used to authenticate the nodes of the message route and check the correctness of transmission.

Message PutSingleRequest.Body

PUT Single request body

Field Type Label Description
object Object Prepared object with payload.
copies_number uint32 repeated Number of copies of the object to store within the RPC call. By default, object is processed according to the container's placement policy. Every number is treated as a minimal number of nodes in a corresponding placement vector that must store an object to complete the request successfully. The length MUST equal the placement vectors number, otherwise request is considered malformed.

Message PutSingleResponse

Object PUT Single response

Field Type Label Description
body PutSingleResponse.Body Body of put single object response message.
meta_header frost.fs.session.ResponseMetaHeader Carries response meta information. Header data is used only to regulate message transport and does not affect request execution.
verify_header frost.fs.session.ResponseVerificationHeader Carries response verification information. This header is used to authenticate the nodes of the message route and check the correctness of transmission.

Message PutSingleResponse.Body

PUT Single Object response body

Message Range

Object payload range.Ranges of zero length SHOULD be considered as invalid.

Field Type Label Description
offset uint64 Offset of the range from the object payload start
length uint64 Length in bytes of the object payload range

Message SearchRequest

Object Search request

Field Type Label Description
body SearchRequest.Body Body of search object request message.
meta_header frost.fs.session.RequestMetaHeader Carries request meta information. Header data is used only to regulate message transport and does not affect request execution.
verify_header frost.fs.session.RequestVerificationHeader Carries request verification information. This header is used to authenticate the nodes of the message route and check the correctness of transmission.

Message SearchRequest.Body

Object Search request body

Field Type Label Description
container_id frost.fs.refs.ContainerID Container identifier were to search
version uint32 Version of the Query Language used
filters SearchRequest.Body.Filter repeated List of search expressions

Message SearchRequest.Body.Filter

Filter structure checks if the object header field or the attribute content matches a value.

If no filters are set, search request will return all objects of the container, including Regular object and Tombstone objects. Most human users expect to get only object they can directly work with. In that case, $Object:ROOT filter should be used.

By default key field refers to the corresponding object's Attribute. Some Object's header fields can also be accessed by adding $Object: prefix to the name. Here is the list of fields available via this prefix:

  • $Object:version
    version
  • $Object:objectID
    object_id
  • $Object:containerID
    container_id
  • $Object:ownerID
    owner_id
  • $Object:creationEpoch
    creation_epoch
  • $Object:payloadLength
    payload_length
  • $Object:payloadHash
    payload_hash
  • $Object:objectType
    object_type
  • $Object:homomorphicHash
    homomorphic_hash
  • $Object:split.parent
    object_id of parent
  • $Object:split.splitID
    16 byte UUIDv4 used to identify the split object hierarchy parts
  • $Object:ec.parent
    If the object is stored according to EC policy, then ec_parent attribute is set to return an id list of all related EC chunks.

There are some well-known filter aliases to match objects by certain properties:

  • $Object:ROOT
    Returns only REGULAR type objects that are not split or that are the top level root objects in a split hierarchy. This includes objects not present physically, like large objects split into smaller objects without a separate top-level root object. Objects of other types like Locks and Tombstones will not be shown. This filter may be useful for listing objects like ls command of some virtual file system. This filter is activated if the key exists, disregarding the value and matcher type.
  • $Object:PHY
    Returns only objects physically stored in the system. This filter is activated if the key exists, disregarding the value and matcher type.

Note: using filters with a key with prefix $Object: and match type NOT_PRESENT is not recommended since this is not a cross-version approach. Behavior when processing this kind of filters is undefined.

Field Type Label Description
match_type MatchType Match type to use
key string Attribute or Header fields to match
value string Value to match

Message SearchResponse

Search response

Field Type Label Description
body SearchResponse.Body Body of search object response message.
meta_header frost.fs.session.ResponseMetaHeader Carries response meta information. Header data is used only to regulate message transport and does not affect request execution.
verify_header frost.fs.session.ResponseVerificationHeader Carries response verification information. This header is used to authenticate the nodes of the message route and check the correctness of transmission.

Message SearchResponse.Body

Object Search response body

Field Type Label Description
id_list frost.fs.refs.ObjectID repeated List of ObjectIDs that match the search query

Top

object/types.proto

Message ECInfo

Meta information for the erasure-encoded object.

Field Type Label Description
chunks ECInfo.Chunk repeated Chunk stored on the node.

Message ECInfo.Chunk

Field Type Label Description
id frost.fs.refs.ObjectID Object ID of the chunk.
index uint32 Index of the chunk.
total uint32 Total number of chunks in this split.

Message Header

Object Header

Field Type Label Description
version frost.fs.refs.Version Object format version. Effectively, the version of API library used to create particular object
container_id frost.fs.refs.ContainerID Object's container
owner_id frost.fs.refs.OwnerID Object's owner
creation_epoch uint64 Object creation Epoch
payload_length uint64 Size of payload in bytes. 0xFFFFFFFFFFFFFFFF means payload_length is unknown.
payload_hash frost.fs.refs.Checksum Hash of payload bytes
object_type ObjectType Type of the object payload content
homomorphic_hash frost.fs.refs.Checksum Homomorphic hash of the object payload
session_token frost.fs.session.SessionToken Session token, if it was used during Object creation. Need it to verify integrity and authenticity out of Request scope.
attributes Header.Attribute repeated User-defined object attributes
split Header.Split Position of the object in the split hierarchy
ec Header.EC Erasure code chunk information.

Message Header.Attribute

Attribute is a user-defined Key-Value metadata pair attached to an object.

Key name must be an object-unique valid UTF-8 string. Value can't be empty. Objects with duplicated attribute names or attributes with empty values will be considered invalid.

There are some "well-known" attributes starting with __SYSTEM__ prefix that affect system behaviour:

  • [ __SYSTEM__UPLOAD_ID ]
    Marks smaller parts of a split bigger object
  • [ __SYSTEM__EXPIRATION_EPOCH ]
    The epoch after which object with no LOCKs on it becomes unavailable. Locked object continues to be available until each of the LOCKs expire.
  • [ __SYSTEM__TICK_EPOCH ]
    Decimal number that defines what epoch must produce object notification with UTF-8 object address in a body (0 value produces notification right after object put)
  • [ __SYSTEM__TICK_TOPIC ]
    UTF-8 string topic ID that is used for object notification

And some well-known attributes used by applications only:

  • Name
    Human-friendly name
  • FileName
    File name to be associated with the object on saving
  • FilePath
    Full path to be associated with the object on saving. Should start with a '/' and use '/' as a delimiting symbol. Trailing '/' should be interpreted as a virtual directory marker. If an object has conflicting FilePath and FileName, FilePath should have higher priority, because it is used to construct the directory tree. FilePath with trailing '/' and non-empty FileName attribute should not be used together.
  • Timestamp
    User-defined local time of object creation in Unix Timestamp format
  • Content-Type
    MIME Content Type of object's payload

For detailed description of each well-known attribute please see the corresponding section in FrostFS Technical Specification.

Field Type Label Description
key string string key to the object attribute
value string string value of the object attribute

Message Header.EC

Erasure code can be applied to any object. Information about encoded object structure is stored in EC header. All objects belonging to a single EC group have the same parent field.

Field Type Label Description
parent frost.fs.refs.ObjectID Identifier of the origin object. Known to all chunks.
index uint32 Index of this chunk.
total uint32 Total number of chunks in this split.
header_length uint32 Total length of a parent header. Used to trim padding zeroes.
header bytes Chunk of a parent header.
parent_split_id bytes As the origin object is EC-splitted its identifier is known to all chunks as parent. But parent itself can be a part of Split (does not relate to EC-split). In this case parent_split_id should be set.
parent_split_parent_id frost.fs.refs.ObjectID EC-parent's parent ID. parent_split_parent_id is set if EC-parent, itself, is a part of Split and if an object ID of its parent is presented. The field allows to determine how EC-chunk is placed in Split hierarchy.
parent_attributes Header.Attribute repeated EC parent's attributes.

Message Header.Split

Bigger objects can be split into a chain of smaller objects. Information about inter-dependencies between spawned objects and how to re-construct the original one is in the Split headers. Parent and children objects must be within the same container.

Field Type Label Description
parent frost.fs.refs.ObjectID Identifier of the origin object. Known only to the minor child.
previous frost.fs.refs.ObjectID Identifier of the left split neighbor
parent_signature frost.fs.refs.Signature signature field of the parent object. Used to reconstruct parent.
parent_header Header header field of the parent object. Used to reconstruct parent.
children frost.fs.refs.ObjectID repeated List of identifiers of the objects generated by splitting current one.
split_id bytes 16 byte UUIDv4 used to identify the split object hierarchy parts. Must be unique inside container. All objects participating in the split must have the same split_id value.

Message Object

Object structure. Object is immutable and content-addressed. It means ObjectID will change if the header or the payload changes. It's calculated as a hash of header field which contains hash of the object's payload.

For non-regular object types payload format depends on object type specified in the header.

Field Type Label Description
object_id frost.fs.refs.ObjectID Object's unique identifier.
signature frost.fs.refs.Signature Signed object_id
header Header Object metadata headers
payload bytes Payload bytes

Message ShortHeader

Short header fields

Field Type Label Description
version frost.fs.refs.Version Object format version. Effectively, the version of API library used to create particular object.
creation_epoch uint64 Epoch when the object was created
owner_id frost.fs.refs.OwnerID Object's owner
object_type ObjectType Type of the object payload content
payload_length uint64 Size of payload in bytes. 0xFFFFFFFFFFFFFFFF means payload_length is unknown
payload_hash frost.fs.refs.Checksum Hash of payload bytes
homomorphic_hash frost.fs.refs.Checksum Homomorphic hash of the object payload

Message SplitInfo

Meta information of split hierarchy for object assembly. With the last part one can traverse linked list of split hierarchy back to the first part and assemble the original object. With a linking object one can assemble an object right from the object parts.

Field Type Label Description
split_id bytes 16 byte UUID used to identify the split object hierarchy parts.
last_part frost.fs.refs.ObjectID The identifier of the last object in split hierarchy parts. It contains split header with the original object header.
link frost.fs.refs.ObjectID The identifier of a linking object for split hierarchy parts. It contains split header with the original object header and a sorted list of object parts.

MatchType

Type of match expression

Name Number Description
MATCH_TYPE_UNSPECIFIED 0 Unknown. Not used
STRING_EQUAL 1 Full string match
STRING_NOT_EQUAL 2 Full string mismatch
NOT_PRESENT 3 Lack of key
COMMON_PREFIX 4 String prefix match

ObjectType

Type of the object payload content. Only REGULAR type objects can be split, hence TOMBSTONE and LOCK payload is limited by the maximum object size.

String presentation of object type is the same as definition:

  • REGULAR
  • TOMBSTONE
  • LOCK
Name Number Description
REGULAR 0 Just a normal object
TOMBSTONE 1 Used internally to identify deleted objects
LOCK 3 Object lock

Scalar Value Types

.proto Type Notes C++ Type Java Type Python Type
double double double float
float float float float
int32 Uses variable-length encoding. Inefficient for encoding negative numbers – if your field is likely to have negative values, use sint32 instead. int32 int int
int64 Uses variable-length encoding. Inefficient for encoding negative numbers – if your field is likely to have negative values, use sint64 instead. int64 long int/long
uint32 Uses variable-length encoding. uint32 int int/long
uint64 Uses variable-length encoding. uint64 long int/long
sint32 Uses variable-length encoding. Signed int value. These more efficiently encode negative numbers than regular int32s. int32 int int
sint64 Uses variable-length encoding. Signed int value. These more efficiently encode negative numbers than regular int64s. int64 long int/long
fixed32 Always four bytes. More efficient than uint32 if values are often greater than 2^28. uint32 int int
fixed64 Always eight bytes. More efficient than uint64 if values are often greater than 2^56. uint64 long int/long
sfixed32 Always four bytes. int32 int int
sfixed64 Always eight bytes. int64 long int/long
bool bool boolean boolean
string A string must always contain UTF-8 encoded or 7-bit ASCII text. string String str/unicode
bytes May contain any arbitrary sequence of bytes. string ByteString str