Release v2.3.0 - Seonyudo (선유도, 仙遊島)

UN-LOCODE support and Container's used space reporting

Signed-off-by: Stanislav Bogatyrev <stanislav@nspcc.ru>

CHANGELOG.md

@@ -1,11 +1,47 @@
 # Changelog
 
+## [2.3.0] - 2021-02-11 - Seonyudo (선유도, 仙遊島)
+
+This release brings support for nodes to exchange information about disk space
+used by each Container. This information will be used by Inner Ring nodes to
+calculate basic rewards and payments.
+
+Another significant change is UN/LOCODE support for node's attributes describing
+geographical location. From now on, most of the geographical attributes will be
+calculated automatically from a single `UN-LOCODE` attribute in a deterministic
+manner.
+
+### Added
+
+- Added `container.AnnounceUsedSpace` request for announcing disk space consumed
+  by container's objects on the node
+- Added `Continent` well-known node's attribute
+- Added `SubDivCode` well-known node's attribute
+- Added `Location` well-known node's attribute
+- Added `CounrtyCode` well-known node's attribute
+- Added `STRING_NOT_EQUAL` match type
+- Added `NOT_PRESENT` match type
+- Added JSON names for search request filter fields
+
+### Changed
+
+- `Locode` well-known node's attribute renamed to `UN-LOCODE`. It will be used
+  as a base for calculating most of the node's geographical attributes.
+- `Region` well-known node's attribute renamed to `SubDiv`
+
+### Removed
+
+- Removed `City` well-known node's attribute
+- Removed `Region` well-known node's attribute
+
 ## [2.2.1] - 2021-01-15
 
 Define "well-known" X-headers
 
 ### Added
-- Description of the format of "well-known" X-headers that affect system behavior
+
+- Description of the format of "well-known" X-headers that affect system
+  behavior
 - X-header key to netmap epoch value
 - X-header key to netmap lookup depth value
 
@@ -14,9 +50,11 @@ Define "well-known" X-headers
 Storage Groups based Data Audit updates
 
 ### Added
+
 - `audit.DataAuditResult` message for recording audit result is added
 
 ### Changed
+
 - `object.ShortHeader` now has `payload_hash` and `homomorphic_hash` fields
 
 ## [2.1.1] - 2020-12-17
@@ -24,6 +62,7 @@ Storage Groups based Data Audit updates
 Minor documentation fixes
 
 ### Changed
+
 - Clarify JSON encoding for `ObjectID`, `ContainerID` and `OwnerID`
 - Clarify object field usage in some requests' eACL filters
 
@@ -32,6 +71,7 @@ Minor documentation fixes
 Object split and deletion improvements, documentation clarifications.
 
 ### Added
+
 - `$Object:objectID` added to the list of available ACL and Search filters
 - `split_id` field added in `object.Object.header`
 - `$Object:split.splitID` search filter added
@@ -47,10 +87,12 @@ Object split and deletion improvements, documentation clarifications.
 - `raw` flag added in `object.GetRangeRequest.Body`
 
 ### Changed
+
 - Clarified special search index descriptions
 - Clarified various types encoding formats descriptions
 
 ### Removed
+
 - `$Object:CHILDFREE` filter description removed from well-known list
 - `$Object:LEAF` filter description removed from well-known list
 
@@ -59,11 +101,14 @@ Object split and deletion improvements, documentation clarifications.
 More "well-known" application attributes and documentation updates.
 
 ### Added
+
 - Added "well-known" attributes list for extended ACL and object search filters
 - Added `Name`, `Timestamp` "well-known" application attributes for container
-- Added `Name`, `FileName`, `Timestamp` "well-known" application attributes for object
+- Added `Name`, `FileName`, `Timestamp` "well-known" application attributes for
+  object
 
 ### Changed
+
 - `BearerToken.owner_id` field description changed
 - `Subnet` container attribute now has `__NEOFS__` prefix
 - Search and ACL filters now have `key` and `value` fields
@@ -73,16 +118,19 @@ More "well-known" application attributes and documentation updates.
 Documentation updates and JSON field names definition
 
 ### Added
+
 - Added "well-known" attributes list for `netmap.NodeInfo`
 - Added "well-known" attributes list for objects
 - Added "well-known" attributes list for containers
 - JSON field names defined for most of data structures
 
 ### Changed
+
 - Documentation updated for all packages fixing typos and minor inaccuracies
 - `acl.EACLRecord.Target.key_list` field renamed to `keys` for consistency
 
 ### Removed
+
 - Human-written documentation will now be only in [NeoFS
   Specification](https://github.com/nspcc-dev/neofs-spec)
 
@@ -91,6 +139,7 @@ Documentation updates and JSON field names definition
 Major API refactoring and simplification.
 
 ### Added
+
 - `neo.fs.v2` prefix added to all package names
 - `container.Attributes` field added
 - `refs.ContainerID` added as a separate type
@@ -102,26 +151,31 @@ Major API refactoring and simplification.
 - `refs.Version` defined as a separate type
 - `refs.Version` field added to all messages stored in SmartContracts
 - `refs.Checksum` defined as a separate type
-- `netmap.LocalNodeInfo` request added to get actual information from connected peer
+- `netmap.LocalNodeInfo` request added to get actual information from connected
+  peer
 
 ### Changed
+
 - Extended ACL Table format changed
 - Protobuf definitions style changed to follow Google Style Guide
 - `System` and `Extended` Object headers are merged into on `object.Header` type
 - `object.UserHeader` renamed to `object.Header.Attribute`
-- `refs.ObjectID` is now a hash of the `object.Header` field, which contains hash of payload
+- `refs.ObjectID` is now a hash of the `object.Header` field, which contains
+  hash of payload
 - `StorageGroup` information moved to Object's payload
-- `netmap.NodeInfo.options` renamed to `netmap.NodeInfo.attributes` and it uses a
+- `netmap.NodeInfo.options` renamed to `netmap.NodeInfo.attributes` and it uses
-  separate `netmap.NodeInfo.Attribute` type now.
+  a separate `netmap.NodeInfo.Attribute` type now.
 - `netmap.NodeInfo.Attribute` type now has a list of parents to construct a tree
 - Session Token renamed to `session.SessionToken` from `session.Token`
 - All Requests and Responses now have a common "body-meta-verify" structure
 - Meta and Verification headers now follow Matryoshka-style composition
 - SessionToken and BearerToken are now part of Meta header
 - Object placement policy format is simplified and defined in `netmap` package
-- `object.Head()` request now returns either short header or full header with a signature
+- `object.Head()` request now returns either short header or full header with a
+  signature
 
 ### Removed
+
 - gogoproto is not used anymore
 - `decimal` package merged into `accounting` package
 - `query` package merged into `object` package
@@ -142,7 +196,8 @@ Major API refactoring and simplification.
 
 ### Added
 
-- ```acl.EACLRecord```, ```acl.EACLTable``` messages for the table of extended ACL rules.
+- ```acl.EACLRecord```, ```acl.EACLTable``` messages for the table of extended
+  ACL rules.
 
 ## [1.1.0] - 2020-06-18
 
@@ -201,7 +256,8 @@ Bump major release
 ### Removed
 
 - `Raw` field from `object.GetRequest` and `object.HeadRequest` messages.
-- `Token` field from `object.PutRequest.PutHeader` and `object.DeleteRequest` messages.
+- `Token` field from `object.PutRequest.PutHeader` and `object.DeleteRequest`
+  messages.
 - `VerificationHeader` message.
 
 ## [0.7.1] - 2020-04-20
@@ -256,3 +312,4 @@ Bump major release
 [2.1.1]: https://github.com/nspcc-dev/neofs-api/compare/v2.1.0...v2.1.1
 [2.2.0]: https://github.com/nspcc-dev/neofs-api/compare/v2.1.1...v2.2.0
 [2.2.1]: https://github.com/nspcc-dev/neofs-api/compare/v2.2.0...v2.2.1
+[2.3.0]: https://github.com/nspcc-dev/neofs-api/compare/v2.2.1...v2.3.0

proto-docs/container.md

@@ -8,6 +8,11 @@
     - [ContainerService](#neo.fs.v2.container.ContainerService)
     
   - Messages
+    - [AnnounceUsedSpaceRequest](#neo.fs.v2.container.AnnounceUsedSpaceRequest)
+    - [AnnounceUsedSpaceRequest.Body](#neo.fs.v2.container.AnnounceUsedSpaceRequest.Body)
+    - [AnnounceUsedSpaceRequest.Body.Announcement](#neo.fs.v2.container.AnnounceUsedSpaceRequest.Body.Announcement)
+    - [AnnounceUsedSpaceResponse](#neo.fs.v2.container.AnnounceUsedSpaceResponse)
+    - [AnnounceUsedSpaceResponse.Body](#neo.fs.v2.container.AnnounceUsedSpaceResponse.Body)
     - [DeleteRequest](#neo.fs.v2.container.DeleteRequest)
     - [DeleteRequest.Body](#neo.fs.v2.container.DeleteRequest.Body)
     - [DeleteResponse](#neo.fs.v2.container.DeleteResponse)
@@ -68,6 +73,7 @@ rpc Get(GetRequest) returns (GetResponse);
 rpc List(ListRequest) returns (ListResponse);
 rpc SetExtendedACL(SetExtendedACLRequest) returns (SetExtendedACLResponse);
 rpc GetExtendedACL(GetExtendedACLRequest) returns (GetExtendedACLResponse);
+rpc AnnounceUsedSpace(AnnounceUsedSpaceRequest) returns (AnnounceUsedSpaceResponse);
 
 ```
 
@@ -122,9 +128,74 @@ storage.
 | Name | Input | Output |
 | ---- | ----- | ------ |
 | GetExtendedACL | [GetExtendedACLRequest](#neo.fs.v2.container.GetExtendedACLRequest) | [GetExtendedACLResponse](#neo.fs.v2.container.GetExtendedACLResponse) |
+#### Method AnnounceUsedSpace
+
+Announce container used space values for P2P synchronization.
+
+| Name | Input | Output |
+| ---- | ----- | ------ |
+| AnnounceUsedSpace | [AnnounceUsedSpaceRequest](#neo.fs.v2.container.AnnounceUsedSpaceRequest) | [AnnounceUsedSpaceResponse](#neo.fs.v2.container.AnnounceUsedSpaceResponse) |
  <!-- end services -->
 
 
+<a name="neo.fs.v2.container.AnnounceUsedSpaceRequest"></a>
+
+### Message AnnounceUsedSpaceRequest
+Announce container used space
+
+
+| Field | Type | Label | Description |
+| ----- | ---- | ----- | ----------- |
+| body | [AnnounceUsedSpaceRequest.Body](#neo.fs.v2.container.AnnounceUsedSpaceRequest.Body) |  | Body of announce used space request message. |
+| meta_header | [neo.fs.v2.session.RequestMetaHeader](#neo.fs.v2.session.RequestMetaHeader) |  | Carries request meta information. Header data is used only to regulate message transport and does not affect request execution. |
+| verify_header | [neo.fs.v2.session.RequestVerificationHeader](#neo.fs.v2.session.RequestVerificationHeader) |  | Carries request verification information. This header is used to authenticate the nodes of the message route and check the correctness of transmission. |
+
+
+<a name="neo.fs.v2.container.AnnounceUsedSpaceRequest.Body"></a>
+
+### Message AnnounceUsedSpaceRequest.Body
+Container used space announcement body.
+
+
+| Field | Type | Label | Description |
+| ----- | ---- | ----- | ----------- |
+| announcements | [AnnounceUsedSpaceRequest.Body.Announcement](#neo.fs.v2.container.AnnounceUsedSpaceRequest.Body.Announcement) | repeated | List of announcements. If nodes share several containers, then announcements transferred in a batch. |
+
+
+<a name="neo.fs.v2.container.AnnounceUsedSpaceRequest.Body.Announcement"></a>
+
+### Message AnnounceUsedSpaceRequest.Body.Announcement
+Announcement contains used space information about single container.
+
+
+| Field | Type | Label | Description |
+| ----- | ---- | ----- | ----------- |
+| epoch | [uint64](#uint64) |  | Epoch number for which container size estimation was produced. |
+| container_id | [neo.fs.v2.refs.ContainerID](#neo.fs.v2.refs.ContainerID) |  | Identifier of the container. |
+| used_space | [uint64](#uint64) |  | Used space is a sum of object payload sizes of specified container, stored in the node. It must not include inhumed objects. |
+
+
+<a name="neo.fs.v2.container.AnnounceUsedSpaceResponse"></a>
+
+### Message AnnounceUsedSpaceResponse
+Announce container used space
+
+
+| Field | Type | Label | Description |
+| ----- | ---- | ----- | ----------- |
+| body | [AnnounceUsedSpaceResponse.Body](#neo.fs.v2.container.AnnounceUsedSpaceResponse.Body) |  | Body of announce used space response message. |
+| meta_header | [neo.fs.v2.session.ResponseMetaHeader](#neo.fs.v2.session.ResponseMetaHeader) |  | Carries response meta information. Header data is used only to regulate message transport and does not affect request execution. |
+| verify_header | [neo.fs.v2.session.ResponseVerificationHeader](#neo.fs.v2.session.ResponseVerificationHeader) |  | Carries response verification information. This header is used to authenticate the nodes of the message route and check the correctness of transmission. |
+
+
+<a name="neo.fs.v2.container.AnnounceUsedSpaceResponse.Body"></a>
+
+### Message AnnounceUsedSpaceResponse.Body
+`AnnounceUsedSpaceResponse` has an empty body because announcements are
+one way communication.
+
+
+
 <a name="neo.fs.v2.container.DeleteRequest"></a>
 
 ### Message DeleteRequest

proto-docs/netmap.md

@@ -177,21 +177,37 @@ explicitly set:
 * Subnet \
   String ID of Node's storage subnet. There can be only one subnet served
   by the Storage Node.
-* Locode \
+* UN-LOCODE \
   Node's geographic location in
   [UN/LOCODE](https://www.unece.org/cefact/codesfortrade/codes_index.html)
-  format approximated to the nearest point defined in standard.
+  format approximated to the nearest point defined in the standard.
-* Country \
+* CountryCode \
   Country code in
   [ISO 3166-1_alpha-2](https://en.wikipedia.org/wiki/ISO_3166-1_alpha-2)
-  format. Calculated automatically from `Locode` attribute
+  format. Calculated automatically from `UN-LOCODE` attribute.
-* Region \
+* Country \
-  Country's administative subdivision where node is located. Calculated
+  Country short name in English, as defined in
-  automatically from `Locode` attribute based on `SubDiv` field. Presented
+  [ISO-3166](https://www.iso.org/obp/ui/#search). Calculated automatically
-  in [ISO 3166-2](https://en.wikipedia.org/wiki/ISO_3166-2) format.
+  from `UN-LOCODE` attribute.
-* City \
+* Location \
-  City, town, village or rural area name where node is located written
+  Place names are given, whenever possible, in their national language
-  without diacritics . Calculated automatically from `Locode` attribute.
+  versions as expressed in the Roman alphabet using the 26 characters of
+  the character set adopted for international trade data interchange,
+  written without diacritics . Calculated automatically from `UN-LOCODE`
+  attribute.
+* SubDivCode \
+  Country's administrative subdivision where node is located. Calculated
+  automatically from `UN-LOCODE` attribute based on `SubDiv` field.
+  Presented in [ISO 3166-2](https://en.wikipedia.org/wiki/ISO_3166-2)
+  format.
+* SubDiv \
+  Country's administrative subdivision name, as defined in
+  [ISO 3166-2](https://en.wikipedia.org/wiki/ISO_3166-2). Calculated
+  automatically from `UN-LOCODE` attribute.
+* Continent \
+  Node's continent name according to the [Seven-Continent model]
+  (https://en.wikipedia.org/wiki/Continent#Number). Calculated
+  automatically from `UN-LOCODE` attribute.
 
 For detailed description of each well-known attribute please see the
 corresponding section in NeoFS Technical specification.

proto-docs/object.md

@@ -597,6 +597,10 @@ properties:
   Returns only objects physically stored in the system. This filter is
   activated if the `key` exists, disregarding the value and matcher type.
 
+Note: using filters with a key with prefix `$Object:` and match type
+`NOT_PRESENT `is not recommended since this is not a cross-version approach.
+Behavior when processing this kind of filters is undefined.
+
 
 | Field | Type | Label | Description |
 | ----- | ---- | ----- | ----------- |
@@ -780,6 +784,8 @@ Type of match expression
 | ---- | ------ | ----------- |
 | MATCH_TYPE_UNSPECIFIED | 0 | Unknown. Not used |
 | STRING_EQUAL | 1 | Full string match |
+| STRING_NOT_EQUAL | 2 | Full string mismatch |
+| NOT_PRESENT | 3 | Lack of key |
 
 
 

proto-docs/session.md

@@ -84,7 +84,7 @@ Session creation request body
 
 | Field | Type | Label | Description |
 | ----- | ---- | ----- | ----------- |
-| owner_id | [neo.fs.v2.refs.OwnerID](#neo.fs.v2.refs.OwnerID) |  | Dession initiating user's or node's key derived `OwnerID`. |
+| owner_id | [neo.fs.v2.refs.OwnerID](#neo.fs.v2.refs.OwnerID) |  | Session initiating user's or node's key derived `OwnerID` |
 | expiration | [uint64](#uint64) |  | Session expiration `Epoch` |
 
 
@@ -243,7 +243,25 @@ Lifetime parameters of the token. Field names taken from rfc7519.
 <a name="neo.fs.v2.session.XHeader"></a>
 
 ### Message XHeader
-Extended headers for Request/Response.
+Extended headers for Request/Response. May contain any user-defined headers
+to be interpreted on application level.
+
+Key name must be unique valid UTF-8 string. Value can't be empty. Requests or
+Responses with duplicated header names or headers with empty values will be
+considered invalid.
+
+There are some "well-known" headers starting with `__NEOFS__` prefix that
+affect system behaviour:
+
+* __NEOFS__NETMAP_EPOCH \
+  Netmap epoch to use for object placement calculation. The `value` is string
+  encoded `uint64` in decimal presentation. If set to '0' or not set, the
+  current epoch only will be used.
+* __NEOFS__NETMAP_LOOKUP_DEPTH \
+  If object can't be found using current epoch's netmap, this header limits
+  how many past epochs back the node can lookup. The `value` is string
+  encoded `uint64` in decimal presentation. If set to '0' or not set, the
+  current epoch only will be used.
 
 
 | Field | Type | Label | Description |