TrueCloudLab/frostfs-api
28
0
Fork 17
Code Issues 11 Pull requests 1 Releases Activity Actions

Release v2.3.0 - Seonyudo (선유도, 仙遊島)

Browse source 
UN-LOCODE support and Container's used space reporting

Signed-off-by: Stanislav Bogatyrev <stanislav@nspcc.ru>
This commit is contained in:
Stanislav Bogatyrev 2021-02-11 16:13:07 +03:00
parent 5152f607f1
commit a37a7c5ef4
5 changed files with 190 additions and 22 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

75
CHANGELOG.md
View file

@ -1,11 +1,47 @@
# Changelog
## [2.3.0] - 2021-02-11 - Seonyudo (선유도, 仙遊島)
This release brings support for nodes to exchange information about disk space
used by each Container. This information will be used by Inner Ring nodes to
calculate basic rewards and payments.
Another significant change is UN/LOCODE support for node's attributes describing
geographical location. From now on, most of the geographical attributes will be
calculated automatically from a single `UN-LOCODE` attribute in a deterministic
manner.
### Added
- Added `container.AnnounceUsedSpace` request for announcing disk space consumed
by container's objects on the node
- Added `Continent` well-known node's attribute
- Added `SubDivCode` well-known node's attribute
- Added `Location` well-known node's attribute
- Added `CounrtyCode` well-known node's attribute
- Added `STRING_NOT_EQUAL` match type
- Added `NOT_PRESENT` match type
- Added JSON names for search request filter fields
### Changed
- `Locode` well-known node's attribute renamed to `UN-LOCODE`. It will be used
as a base for calculating most of the node's geographical attributes.
- `Region` well-known node's attribute renamed to `SubDiv`
### Removed
- Removed `City` well-known node's attribute
- Removed `Region` well-known node's attribute
## [2.2.1] - 2021-01-15
Define "well-known" X-headers
### Added
- Description of the format of "well-known" X-headers that affect system behavior
- Description of the format of "well-known" X-headers that affect system
behavior
- X-header key to netmap epoch value
- X-header key to netmap lookup depth value
@ -14,9 +50,11 @@ Define "well-known" X-headers
Storage Groups based Data Audit updates
### Added
- `audit.DataAuditResult` message for recording audit result is added
### Changed
- `object.ShortHeader` now has `payload_hash` and `homomorphic_hash` fields
## [2.1.1] - 2020-12-17
@ -24,6 +62,7 @@ Storage Groups based Data Audit updates
Minor documentation fixes
### Changed
- Clarify JSON encoding for `ObjectID`, `ContainerID` and `OwnerID`
- Clarify object field usage in some requests' eACL filters
@ -32,6 +71,7 @@ Minor documentation fixes
Object split and deletion improvements, documentation clarifications.
### Added
- `$Object:objectID` added to the list of available ACL and Search filters
- `split_id` field added in `object.Object.header`
- `$Object:split.splitID` search filter added
@ -47,10 +87,12 @@ Object split and deletion improvements, documentation clarifications.
- `raw` flag added in `object.GetRangeRequest.Body`
### Changed
- Clarified special search index descriptions
- Clarified various types encoding formats descriptions
### Removed
- `$Object:CHILDFREE` filter description removed from well-known list
- `$Object:LEAF` filter description removed from well-known list
@ -59,11 +101,14 @@ Object split and deletion improvements, documentation clarifications.
More "well-known" application attributes and documentation updates.
### Added
- Added "well-known" attributes list for extended ACL and object search filters
- Added `Name`, `Timestamp` "well-known" application attributes for container
- Added `Name`, `FileName`, `Timestamp` "well-known" application attributes for object
- Added `Name`, `FileName`, `Timestamp` "well-known" application attributes for
object
### Changed
- `BearerToken.owner_id` field description changed
- `Subnet` container attribute now has `__NEOFS__` prefix
- Search and ACL filters now have `key` and `value` fields
@ -73,16 +118,19 @@ More "well-known" application attributes and documentation updates.
Documentation updates and JSON field names definition
### Added
- Added "well-known" attributes list for `netmap.NodeInfo`
- Added "well-known" attributes list for objects
- Added "well-known" attributes list for containers
- JSON field names defined for most of data structures
### Changed
- Documentation updated for all packages fixing typos and minor inaccuracies
- `acl.EACLRecord.Target.key_list` field renamed to `keys` for consistency
### Removed
- Human-written documentation will now be only in [NeoFS
Specification](https://github.com/nspcc-dev/neofs-spec)
@ -91,6 +139,7 @@ Documentation updates and JSON field names definition
Major API refactoring and simplification.
### Added
- `neo.fs.v2` prefix added to all package names
- `container.Attributes` field added
- `refs.ContainerID` added as a separate type
@ -102,26 +151,31 @@ Major API refactoring and simplification.
- `refs.Version` defined as a separate type
- `refs.Version` field added to all messages stored in SmartContracts
- `refs.Checksum` defined as a separate type
- `netmap.LocalNodeInfo` request added to get actual information from connected peer
- `netmap.LocalNodeInfo` request added to get actual information from connected
peer
### Changed
- Extended ACL Table format changed
- Protobuf definitions style changed to follow Google Style Guide
- `System` and `Extended` Object headers are merged into on `object.Header` type
- `object.UserHeader` renamed to `object.Header.Attribute`
- `refs.ObjectID` is now a hash of the `object.Header` field, which contains hash of payload
- `refs.ObjectID` is now a hash of the `object.Header` field, which contains
hash of payload
- `StorageGroup` information moved to Object's payload
- `netmap.NodeInfo.options` renamed to `netmap.NodeInfo.attributes` and it uses a
separate `netmap.NodeInfo.Attribute` type now.
- `netmap.NodeInfo.options` renamed to `netmap.NodeInfo.attributes` and it uses
a separate `netmap.NodeInfo.Attribute` type now.
- `netmap.NodeInfo.Attribute` type now has a list of parents to construct a tree
- Session Token renamed to `session.SessionToken` from `session.Token`
- All Requests and Responses now have a common "body-meta-verify" structure
- Meta and Verification headers now follow Matryoshka-style composition
- SessionToken and BearerToken are now part of Meta header
- Object placement policy format is simplified and defined in `netmap` package
- `object.Head()` request now returns either short header or full header with a signature
- `object.Head()` request now returns either short header or full header with a
signature
### Removed
- gogoproto is not used anymore
- `decimal` package merged into `accounting` package
- `query` package merged into `object` package
@ -142,7 +196,8 @@ Major API refactoring and simplification.
### Added
- ```acl.EACLRecord```, ```acl.EACLTable``` messages for the table of extended ACL rules.
- ```acl.EACLRecord```, ```acl.EACLTable``` messages for the table of extended
ACL rules.
## [1.1.0] - 2020-06-18
@ -201,7 +256,8 @@ Bump major release
### Removed
- `Raw` field from `object.GetRequest` and `object.HeadRequest` messages.
- `Token` field from `object.PutRequest.PutHeader` and `object.DeleteRequest` messages.
- `Token` field from `object.PutRequest.PutHeader` and `object.DeleteRequest`
messages.
- `VerificationHeader` message.
## [0.7.1] - 2020-04-20
@ -256,3 +312,4 @@ Bump major release
[2.1.1]: https://github.com/nspcc-dev/neofs-api/compare/v2.1.0...v2.1.1
[2.2.0]: https://github.com/nspcc-dev/neofs-api/compare/v2.1.1...v2.2.0
[2.2.1]: https://github.com/nspcc-dev/neofs-api/compare/v2.2.0...v2.2.1
[2.3.0]: https://github.com/nspcc-dev/neofs-api/compare/v2.2.1...v2.3.0

71
proto-docs/container.md
View file

@ -8,6 +8,11 @@
- [ContainerService](#neo.fs.v2.container.ContainerService)
- Messages
- [AnnounceUsedSpaceRequest](#neo.fs.v2.container.AnnounceUsedSpaceRequest)
- [AnnounceUsedSpaceRequest.Body](#neo.fs.v2.container.AnnounceUsedSpaceRequest.Body)
- [AnnounceUsedSpaceRequest.Body.Announcement](#neo.fs.v2.container.AnnounceUsedSpaceRequest.Body.Announcement)
- [AnnounceUsedSpaceResponse](#neo.fs.v2.container.AnnounceUsedSpaceResponse)
- [AnnounceUsedSpaceResponse.Body](#neo.fs.v2.container.AnnounceUsedSpaceResponse.Body)
- [DeleteRequest](#neo.fs.v2.container.DeleteRequest)
- [DeleteRequest.Body](#neo.fs.v2.container.DeleteRequest.Body)
- [DeleteResponse](#neo.fs.v2.container.DeleteResponse)
@ -68,6 +73,7 @@ rpc Get(GetRequest) returns (GetResponse);
rpc List(ListRequest) returns (ListResponse);
rpc SetExtendedACL(SetExtendedACLRequest) returns (SetExtendedACLResponse);
rpc GetExtendedACL(GetExtendedACLRequest) returns (GetExtendedACLResponse);
rpc AnnounceUsedSpace(AnnounceUsedSpaceRequest) returns (AnnounceUsedSpaceResponse);
```
@ -122,9 +128,74 @@ storage.
| Name | Input | Output |
| ---- | ----- | ------ |
| GetExtendedACL | [GetExtendedACLRequest](#neo.fs.v2.container.GetExtendedACLRequest) | [GetExtendedACLResponse](#neo.fs.v2.container.GetExtendedACLResponse) |
#### Method AnnounceUsedSpace
Announce container used space values for P2P synchronization.
| Name | Input | Output |
| ---- | ----- | ------ |
| AnnounceUsedSpace | [AnnounceUsedSpaceRequest](#neo.fs.v2.container.AnnounceUsedSpaceRequest) | [AnnounceUsedSpaceResponse](#neo.fs.v2.container.AnnounceUsedSpaceResponse) |
<!-- end services -->
<a name="neo.fs.v2.container.AnnounceUsedSpaceRequest"></a>
### Message AnnounceUsedSpaceRequest
Announce container used space
| Field | Type | Label | Description |
| ----- | ---- | ----- | ----------- |
| body | [AnnounceUsedSpaceRequest.Body](#neo.fs.v2.container.AnnounceUsedSpaceRequest.Body) | | Body of announce used space request message. |
| meta_header | [neo.fs.v2.session.RequestMetaHeader](#neo.fs.v2.session.RequestMetaHeader) | | Carries request meta information. Header data is used only to regulate message transport and does not affect request execution. |
| verify_header | [neo.fs.v2.session.RequestVerificationHeader](#neo.fs.v2.session.RequestVerificationHeader) | | Carries request verification information. This header is used to authenticate the nodes of the message route and check the correctness of transmission. |
<a name="neo.fs.v2.container.AnnounceUsedSpaceRequest.Body"></a>
### Message AnnounceUsedSpaceRequest.Body
Container used space announcement body.
| Field | Type | Label | Description |
| ----- | ---- | ----- | ----------- |
| announcements | [AnnounceUsedSpaceRequest.Body.Announcement](#neo.fs.v2.container.AnnounceUsedSpaceRequest.Body.Announcement) | repeated | List of announcements. If nodes share several containers, then announcements transferred in a batch. |
<a name="neo.fs.v2.container.AnnounceUsedSpaceRequest.Body.Announcement"></a>
### Message AnnounceUsedSpaceRequest.Body.Announcement
Announcement contains used space information about single container.
| Field | Type | Label | Description |
| ----- | ---- | ----- | ----------- |
| epoch | [uint64](#uint64) | | Epoch number for which container size estimation was produced. |
| container_id | [neo.fs.v2.refs.ContainerID](#neo.fs.v2.refs.ContainerID) | | Identifier of the container. |
| used_space | [uint64](#uint64) | | Used space is a sum of object payload sizes of specified container, stored in the node. It must not include inhumed objects. |
<a name="neo.fs.v2.container.AnnounceUsedSpaceResponse"></a>
### Message AnnounceUsedSpaceResponse
Announce container used space
| Field | Type | Label | Description |
| ----- | ---- | ----- | ----------- |
| body | [AnnounceUsedSpaceResponse.Body](#neo.fs.v2.container.AnnounceUsedSpaceResponse.Body) | | Body of announce used space response message. |
| meta_header | [neo.fs.v2.session.ResponseMetaHeader](#neo.fs.v2.session.ResponseMetaHeader) | | Carries response meta information. Header data is used only to regulate message transport and does not affect request execution. |
| verify_header | [neo.fs.v2.session.ResponseVerificationHeader](#neo.fs.v2.session.ResponseVerificationHeader) | | Carries response verification information. This header is used to authenticate the nodes of the message route and check the correctness of transmission. |
<a name="neo.fs.v2.container.AnnounceUsedSpaceResponse.Body"></a>
### Message AnnounceUsedSpaceResponse.Body
`AnnounceUsedSpaceResponse` has an empty body because announcements are
one way communication.
<a name="neo.fs.v2.container.DeleteRequest"></a>
### Message DeleteRequest

38
proto-docs/netmap.md
View file

@ -177,21 +177,37 @@ explicitly set:
* Subnet \
String ID of Node's storage subnet. There can be only one subnet served
by the Storage Node.
* Locode \
* UN-LOCODE \
Node's geographic location in
[UN/LOCODE](https://www.unece.org/cefact/codesfortrade/codes_index.html)
format approximated to the nearest point defined in standard.
* Country \
format approximated to the nearest point defined in the standard.
* CountryCode \
Country code in
[ISO 3166-1_alpha-2](https://en.wikipedia.org/wiki/ISO_3166-1_alpha-2)
format. Calculated automatically from `Locode` attribute
* Region \
Country's administative subdivision where node is located. Calculated
automatically from `Locode` attribute based on `SubDiv` field. Presented
in [ISO 3166-2](https://en.wikipedia.org/wiki/ISO_3166-2) format.
* City \
City, town, village or rural area name where node is located written
without diacritics . Calculated automatically from `Locode` attribute.
format. Calculated automatically from `UN-LOCODE` attribute.
* Country \
Country short name in English, as defined in
[ISO-3166](https://www.iso.org/obp/ui/#search). Calculated automatically
from `UN-LOCODE` attribute.
* Location \
Place names are given, whenever possible, in their national language
versions as expressed in the Roman alphabet using the 26 characters of
the character set adopted for international trade data interchange,
written without diacritics . Calculated automatically from `UN-LOCODE`
attribute.
* SubDivCode \
Country's administrative subdivision where node is located. Calculated
automatically from `UN-LOCODE` attribute based on `SubDiv` field.
Presented in [ISO 3166-2](https://en.wikipedia.org/wiki/ISO_3166-2)
format.
* SubDiv \
Country's administrative subdivision name, as defined in
[ISO 3166-2](https://en.wikipedia.org/wiki/ISO_3166-2). Calculated
automatically from `UN-LOCODE` attribute.
* Continent \
Node's continent name according to the [Seven-Continent model]
(https://en.wikipedia.org/wiki/Continent#Number). Calculated
automatically from `UN-LOCODE` attribute.
For detailed description of each well-known attribute please see the
corresponding section in NeoFS Technical specification.

6
proto-docs/object.md
View file

@ -597,6 +597,10 @@ properties:
Returns only objects physically stored in the system. This filter is
activated if the `key` exists, disregarding the value and matcher type.
Note: using filters with a key with prefix `$Object:` and match type
`NOT_PRESENT `is not recommended since this is not a cross-version approach.
Behavior when processing this kind of filters is undefined.
| Field | Type | Label | Description |
| ----- | ---- | ----- | ----------- |
@ -780,6 +784,8 @@ Type of match expression
| ---- | ------ | ----------- |
| MATCH_TYPE_UNSPECIFIED | 0 | Unknown. Not used |
| STRING_EQUAL | 1 | Full string match |
| STRING_NOT_EQUAL | 2 | Full string mismatch |
| NOT_PRESENT | 3 | Lack of key |

22
proto-docs/session.md
View file

@ -84,7 +84,7 @@ Session creation request body
| Field | Type | Label | Description |
| ----- | ---- | ----- | ----------- |
| owner_id | [neo.fs.v2.refs.OwnerID](#neo.fs.v2.refs.OwnerID) | | Dession initiating user's or node's key derived `OwnerID`. |
| owner_id | [neo.fs.v2.refs.OwnerID](#neo.fs.v2.refs.OwnerID) | | Session initiating user's or node's key derived `OwnerID` |
| expiration | [uint64](#uint64) | | Session expiration `Epoch` |
@ -243,7 +243,25 @@ Lifetime parameters of the token. Field names taken from rfc7519.
<a name="neo.fs.v2.session.XHeader"></a>
### Message XHeader
Extended headers for Request/Response.
Extended headers for Request/Response. May contain any user-defined headers
to be interpreted on application level.
Key name must be unique valid UTF-8 string. Value can't be empty. Requests or
Responses with duplicated header names or headers with empty values will be
considered invalid.
There are some "well-known" headers starting with `__NEOFS__` prefix that
affect system behaviour:
* __NEOFS__NETMAP_EPOCH \
Netmap epoch to use for object placement calculation. The `value` is string
encoded `uint64` in decimal presentation. If set to '0' or not set, the
current epoch only will be used.
* __NEOFS__NETMAP_LOOKUP_DEPTH \
If object can't be found using current epoch's netmap, this header limits
how many past epochs back the node can lookup. The `value` is string
encoded `uint64` in decimal presentation. If set to '0' or not set, the
current epoch only will be used.
| Field | Type | Label | Description |