Compare commits

..

41 commits

Author SHA1 Message Date
1b67ab9608 [#275] client: Return status from all methods
All checks were successful
DCO / DCO (pull_request) Successful in 52s
Tests and linters / Tests (pull_request) Successful in 1m0s
Tests and linters / Lint (pull_request) Successful in 1m55s
Since status is checked first in handleError method, it should be returned from client methods

Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
2024-09-24 18:29:32 +03:00
99d5bf913b [#269] netmap: Add tests for non-ascii attributes in SELECT IN
All checks were successful
DCO / DCO (pull_request) Successful in 30s
Tests and linters / Lint (pull_request) Successful in 2m8s
Tests and linters / Tests (pull_request) Successful in 54s
Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-09-20 11:09:16 +00:00
e50838a33d [#269] netmap: Regenerate policy parser
Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-09-20 11:09:16 +00:00
97cf56ba41 [#269] netmap: Support non-ascii attributes in SELECT IN
Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-09-20 11:09:16 +00:00
07625e3bd1 [#269] Update ANTLR version 4.13.0 -> 4.13.1
Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-09-20 11:09:16 +00:00
da2f0e7532 [#269] .gitignore: Ignore ANTLR jar file
The previous wildcard failed to properly match the ANTLR jar file.

Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-09-20 11:09:16 +00:00
114b4c14b5 [#269] Makefile: Update policy target
The previous policy target generated device-specific comments
(e.g., `/home/john_doe/repos/<...>`), which could result in
unnecessary file changes This behavior would confuse git and
require manual changes to resolve. The update ensures comments
are now device-agnostic, preventing unwanted changes.

Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-09-20 11:09:16 +00:00
e580ee991d [#271] Drop handling of system attributes with NeoFS prefix
All checks were successful
DCO / DCO (pull_request) Successful in 43s
Tests and linters / Tests (pull_request) Successful in 55s
Tests and linters / Lint (pull_request) Successful in 1m25s
Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-09-18 09:59:38 +00:00
6821fe6fb2 [#271] object: Add UserAttributes method
Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-09-18 09:59:38 +00:00
6009d089fc [#270] client: Use RPC call instead of Dial
All checks were successful
DCO / DCO (pull_request) Successful in 44s
Tests and linters / Tests (pull_request) Successful in 55s
Tests and linters / Lint (pull_request) Successful in 1m22s
After api-go upgrade created client doesn't establish connection after created,
so RPC call is required to establish and check connection.
RPC call returns status error, so conversion from status error to context error
is required to satisfy Dial contract and unit tests.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-16 15:31:51 +03:00
3e455777fd [#270] go.mod: Upgrade api-go version
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-16 15:10:02 +03:00
1dc3b77ac7 [#270] pool: Replace deprecated DialContext
`Healthcheck` request performed after client creation, so no extra RPC required.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-16 15:09:59 +03:00
88c6556c37 [#270] go.mod: Upgrade google.golang.org/grpc
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-16 15:09:54 +03:00
d342c0bc16 [#268] client: Make PayloadPatch correctly receive empty patch payload
All checks were successful
DCO / DCO (pull_request) Successful in 40s
Tests and linters / Tests (pull_request) Successful in 53s
Tests and linters / Lint (pull_request) Successful in 1m21s
* Make the method `PatchPayload` send a patch with empty
  payload patch if range's length is non-zero and if it's the
  first call.
* Empty payload patches just cut original object payload. So, these
  patches are also valid.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-09-12 15:12:52 +03:00
f0c599d06d [#268] client: Fix sequential PayloadPatch calls
* The flag 'firstPayloadPatch' keeps its state after first
  `PatchPayload` that make other calls incorrectly set patch
  ranges.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-09-12 15:09:01 +03:00
7d84d104fb [#260] *: Fix linter warnings
All checks were successful
DCO / DCO (pull_request) Successful in 1m2s
Tests and linters / Tests (pull_request) Successful in 1m12s
Tests and linters / Lint (pull_request) Successful in 1m26s
Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-09-09 08:55:32 +00:00
812126a8ff [#260] .golangci.yml: Add protogetter linter
Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-09-09 08:55:32 +00:00
d86223ed56 [#260] Makefile: Add pre-commit targets
Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-09-09 08:55:32 +00:00
76a0cfdadb [#217] netmap: Return node netmap state directly
Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-09-09 08:51:51 +00:00
46ee543899 [#265] go.mod: Use range over int
All checks were successful
DCO / DCO (pull_request) Successful in 47s
Tests and linters / Tests (1.22) (pull_request) Successful in 1m7s
Tests and linters / Tests (1.23) (pull_request) Successful in 1m7s
Tests and linters / Lint (pull_request) Successful in 1m42s
Since Go 1.22 a `for` statement with a `range` clause is able
to iterate through integer values from zero to an upper limit.

gopatch script:
@@
var i, e expression
@@
-for i := 0; i <= e - 1; i++ {
+for i := range e {
    ...
}

@@
var i, e expression
@@
-for i := 0; i <= e; i++ {
+for i := range e + 1 {
    ...
}

@@
var i, e expression
@@
-for i := 0; i < e; i++ {
+for i := range e {
    ...
}

Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2024-09-04 12:37:46 +03:00
8f751d9dd0
[#263] go.mod: Update api-go
All checks were successful
DCO / DCO (pull_request) Successful in 42s
Tests and linters / Tests (1.22) (pull_request) Successful in 1m5s
Tests and linters / Tests (1.23) (pull_request) Successful in 1m4s
Tests and linters / Lint (pull_request) Successful in 1m44s
Remove `client.ContainerEACL` and related references. This change
was initiated by the removal of `ContainerService.GetExtendedACL`
from the API.

Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-09-03 12:36:28 +03:00
3c00f4eeac [#203] Add pool docs
All checks were successful
DCO / DCO (pull_request) Successful in 48s
Tests and linters / Tests (1.22) (pull_request) Successful in 1m3s
Tests and linters / Tests (1.23) (pull_request) Successful in 1m2s
Tests and linters / Lint (pull_request) Successful in 1m41s
Signed-off-by: Nikita Zinkevich <n.zinkevich@yadro.com>
2024-08-30 08:24:31 +03:00
f0b9493ce3 [#261] go.mod: Update api-go
All checks were successful
DCO / DCO (pull_request) Successful in 39s
Tests and linters / Tests (1.23) (pull_request) Successful in 1m1s
Tests and linters / Tests (1.22) (pull_request) Successful in 1m3s
Tests and linters / Lint (pull_request) Successful in 2m3s
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-08-27 11:33:09 +03:00
28f140bf06 [#254] pool: Add parameter gracefulCloseOnSwitchTimeout
Add new param for waiting a little until current in-flight requests will be finished

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-08-22 08:02:51 +00:00
9115d3f281 [#1316] lint: Fix warnings
All checks were successful
DCO / DCO (pull_request) Successful in 39s
Tests and linters / Tests (1.22) (pull_request) Successful in 51s
Tests and linters / Tests (1.23) (pull_request) Successful in 53s
Tests and linters / Lint (pull_request) Successful in 1m18s
Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2024-08-21 17:59:10 +03:00
cf225be0df [#1316] go.mod: Bump go version to 1.22
Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2024-08-21 16:42:58 +03:00
338d1ef254 [#258] pool/tree: Add node address to error
All checks were successful
DCO / DCO (pull_request) Successful in 39s
Tests and linters / Tests (1.22) (pull_request) Successful in 50s
Tests and linters / Tests (1.21) (pull_request) Successful in 54s
Tests and linters / Lint (pull_request) Successful in 1m8s
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-08-20 14:19:58 +03:00
6dd7be11d1 [#256] go.mod: Update api-go
All checks were successful
DCO / DCO (pull_request) Successful in 2m24s
Tests and linters / Tests (1.21) (pull_request) Successful in 2m21s
Tests and linters / Tests (1.22) (pull_request) Successful in 2m22s
Tests and linters / Lint (pull_request) Successful in 3m19s
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-08-20 10:20:28 +03:00
203bba65a0 [#253] pool: Don't count regular FrostFS errors
All checks were successful
DCO / DCO (pull_request) Successful in 54s
Tests and linters / Tests (1.21) (pull_request) Successful in 1m13s
Tests and linters / Tests (1.22) (pull_request) Successful in 1m11s
Tests and linters / Lint (pull_request) Successful in 1m36s
Previously we count all frostfs errors like:
ObjectNotFound, EACLNotFound
because frostfs status is unconditionally resolved into built-in go errors
but handleError method handled built-in errors like internal network ones.
Since after resolving frostfs errors status is also returned we start check this first

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-08-14 16:48:51 +03:00
98aabc45a7 [#252] patcher: Fix applying patch from the same offset
All checks were successful
DCO / DCO (pull_request) Successful in 38s
Tests and linters / Tests (1.22) (pull_request) Successful in 50s
Tests and linters / Tests (1.21) (pull_request) Successful in 56s
Tests and linters / Lint (pull_request) Successful in 1m54s
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-08-13 18:58:21 +03:00
908c96a94d [#251] pool: Fix handlerError panic for objectPatch
All checks were successful
DCO / DCO (pull_request) Successful in 55s
Tests and linters / Tests (1.21) (pull_request) Successful in 1m0s
Tests and linters / Tests (1.22) (pull_request) Successful in 1m0s
Tests and linters / Lint (pull_request) Successful in 1m56s
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-08-13 15:28:39 +03:00
2077b35736 [#231] netmap: Add LIKE operation for filter
All checks were successful
Tests and linters / Tests (1.22) (pull_request) Successful in 53s
Tests and linters / Tests (1.21) (pull_request) Successful in 55s
DCO / DCO (pull_request) Successful in 33s
Tests and linters / Lint (pull_request) Successful in 1m19s
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-08-12 09:49:12 +03:00
92c7596157 [#231] go.mod: Update version for frostfs-api-go/v2
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-08-12 09:49:04 +03:00
a15b1264f5 [#231] policy: Bump jre version to 17
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-08-12 09:44:30 +03:00
5d58519253 [#249] pool: Introduce objectPatch method
All checks were successful
DCO / DCO (pull_request) Successful in 38s
Tests and linters / Tests (1.22) (pull_request) Successful in 53s
Tests and linters / Tests (1.21) (pull_request) Successful in 55s
Tests and linters / Lint (pull_request) Successful in 1m17s
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-08-08 18:26:40 +03:00
93171b3319 [#249] session: Support patch verb
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-08-08 18:26:40 +03:00
3ba7446157 [#249] client: Introduce ObjectPatch
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-08-08 18:26:35 +03:00
335aa18dc6 [#249] go.mod: Bump frostfs-api-go/v2 version
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-08-06 16:57:42 +03:00
361739e860 [#247] object: Introduce patcher package
* Introduce `patcher` package that contains such interfaces to be
  implemented:
  - `PatchApplier` - the main patching engine that merges the stream
    of patches and the stream of original object payload divided by
    ranges. The merged streams result is output to `ChunkedObjectWriter`;
  - `RangeProvider` - provides the original object payload by ranges;
  - `HeaderProvider` - provides the original object header.
* Introduce `patcher` that implements `PatchApplier`;
* Cover all possible cases with unit-tests.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-08-05 12:32:14 +00:00
6dd500def9 [#247] object: Introduce Patch type
* Make ToV2, FromV2 converters for Patch and PayloadPatch;
* Add unit-tests.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-08-05 12:32:14 +00:00
e83d6b7c6a [#244] pool/tree: Collect request duration statistic
All checks were successful
DCO / DCO (pull_request) Successful in 38s
Tests and linters / Tests (1.22) (pull_request) Successful in 50s
Tests and linters / Tests (1.21) (pull_request) Successful in 56s
Tests and linters / Lint (pull_request) Successful in 1m22s
After each request for tree pool statistic accumulated values are reset to zero.

Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
2024-08-02 13:01:14 +03:00
69 changed files with 1673 additions and 718 deletions

View file

@ -13,7 +13,7 @@ jobs:
- name: Setup Go
uses: actions/setup-go@v3
with:
go-version: '1.22'
go-version: '1.23'
- name: Run commit format checker
uses: https://git.frostfs.info/TrueCloudLab/dco-go@v3

View file

@ -11,7 +11,7 @@ jobs:
- name: Set up Go
uses: actions/setup-go@v3
with:
go-version: '1.22'
go-version: '1.23'
cache: true
- name: Install linters
@ -25,7 +25,7 @@ jobs:
runs-on: ubuntu-latest
strategy:
matrix:
go_versions: [ '1.21', '1.22' ]
go_versions: [ '1.22', '1.23' ]
fail-fast: false
steps:
- uses: actions/checkout@v3

2
.gitignore vendored
View file

@ -23,7 +23,7 @@ coverage.txt
coverage.html
# antlr tool jar
antlr-*.jar
antlr*.jar
# tempfiles
.cache

View file

@ -12,7 +12,8 @@ run:
# output configuration options
output:
# colored-line-number|line-number|json|tab|checkstyle|code-climate, default is "colored-line-number"
format: tab
formats:
- format: tab
# all available settings of specific linters
linters-settings:
@ -62,5 +63,6 @@ linters:
- funlen
- gocognit
- contextcheck
- protogetter
disable-all: true
fast: false

View file

@ -1,4 +1,4 @@
FROM golang:1.21
FROM golang:1.22
RUN apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install make openjdk-11-jre -y
RUN apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install make openjdk-17-jre -y
WORKDIR /work

View file

@ -1,9 +1,9 @@
#!/usr/bin/make -f
ANTLR_VERSION="4.13.0"
ANTLR_VERSION=4.13.1
TMP_DIR := .cache
LINT_VERSION ?= 1.56.2
TRUECLOUDLAB_LINT_VERSION ?= 0.0.2
LINT_VERSION ?= 1.60.1
TRUECLOUDLAB_LINT_VERSION ?= 0.0.6
OUTPUT_LINT_DIR ?= $(shell pwd)/bin
LINT_DIR = $(OUTPUT_LINT_DIR)/golangci-lint-$(LINT_VERSION)-v$(TRUECLOUDLAB_LINT_VERSION)
@ -53,7 +53,8 @@ format:
policy:
@wget -q https://www.antlr.org/download/antlr-${ANTLR_VERSION}-complete.jar -O antlr4-tool.jar
@java -Xmx500M -cp "`pwd`/antlr4-tool.jar" "org.antlr.v4.Tool" -o `pwd`/netmap/parser/ -Dlanguage=Go -no-listener -visitor `pwd`/netmap/parser/Query.g4 `pwd`/netmap/parser/QueryLexer.g4
@java -Xmx500M -cp antlr4-tool.jar org.antlr.v4.Tool -Dlanguage=Go \
-no-listener -visitor netmap/parser/Query.g4 netmap/parser/QueryLexer.g4
# Run `make %` in truecloudlab/frostfs-sdk-go container(Golang+Java)
docker/%:
@ -77,3 +78,15 @@ help:
@echo ' Targets:'
@echo ''
@awk '/^#/{ comment = substr($$0,3) } comment && /^[a-zA-Z][a-zA-Z0-9_-]+ ?:/{ print " ", $$1, comment }' $(MAKEFILE_LIST) | column -t -s ':' | grep -v 'IGNORE' | sort -u
# Activate pre-commit hooks
pre-commit:
pre-commit install --hook-type pre-commit
# Deactivate pre-commit hooks
unpre-commit:
pre-commit uninstall --hook-type pre-commit
# Run pre-commit hooks
pre-commit-run:
@pre-commit run --all-files --hook-stage manual

View file

@ -65,7 +65,7 @@ func (c *Client) APEManagerListChains(ctx context.Context, prm PrmAPEManagerList
var res ResAPEManagerListChains
res.st, err = c.processResponse(resp)
if err != nil || !apistatus.IsSuccessful(res.st) {
return nil, err
return &res, err
}
for _, ch := range resp.GetBody().GetChains() {

View file

@ -11,6 +11,8 @@ import (
"git.frostfs.info/TrueCloudLab/frostfs-api-go/v2/rpc"
"git.frostfs.info/TrueCloudLab/frostfs-api-go/v2/rpc/client"
"google.golang.org/grpc"
"google.golang.org/grpc/codes"
"google.golang.org/grpc/status"
)
// Client represents virtual connection to the FrostFS network to communicate
@ -98,14 +100,22 @@ func (c *Client) Dial(ctx context.Context, prm PrmDial) error {
c.setFrostFSAPIServer((*coreServer)(&c.c))
// TODO: (neofs-api-go#382) perform generic dial stage of the client.Client
_, err := rpc.Balance(&c.c, new(v2accounting.BalanceRequest),
client.WithContext(ctx),
)
if err != nil {
// return context errors since they signal about dial problem
if errors.Is(err, context.Canceled) || errors.Is(err, context.DeadlineExceeded) {
return err
}
st, ok := status.FromError(err)
if ok && st.Code() == codes.Canceled {
return context.Canceled
}
if ok && st.Code() == codes.DeadlineExceeded {
return context.DeadlineExceeded
}
}
return nil
}

View file

@ -1,123 +0,0 @@
package client
import (
"context"
"fmt"
v2container "git.frostfs.info/TrueCloudLab/frostfs-api-go/v2/container"
"git.frostfs.info/TrueCloudLab/frostfs-api-go/v2/refs"
rpcapi "git.frostfs.info/TrueCloudLab/frostfs-api-go/v2/rpc"
"git.frostfs.info/TrueCloudLab/frostfs-api-go/v2/rpc/client"
v2session "git.frostfs.info/TrueCloudLab/frostfs-api-go/v2/session"
"git.frostfs.info/TrueCloudLab/frostfs-api-go/v2/signature"
apistatus "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/client/status"
cid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/id"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/eacl"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/session"
)
// PrmContainerEACL groups parameters of ContainerEACL operation.
type PrmContainerEACL struct {
// FrostFS request X-Headers.
XHeaders []string
ContainerID *cid.ID
Session *session.Container
}
// SetContainer sets identifier of the FrostFS container to read the eACL table.
// Required parameter.
//
// Deprecated: Use PrmContainerEACL.ContainerID instead.
func (x *PrmContainerEACL) SetContainer(id cid.ID) {
x.ContainerID = &id
}
func (x *PrmContainerEACL) buildRequest(c *Client) (*v2container.GetExtendedACLRequest, error) {
if x.ContainerID == nil {
return nil, errorMissingContainer
}
if len(x.XHeaders)%2 != 0 {
return nil, errorInvalidXHeaders
}
var cidV2 refs.ContainerID
x.ContainerID.WriteToV2(&cidV2)
reqBody := new(v2container.GetExtendedACLRequestBody)
reqBody.SetContainerID(&cidV2)
var meta v2session.RequestMetaHeader
writeXHeadersToMeta(x.XHeaders, &meta)
if x.Session != nil {
var tokv2 v2session.Token
x.Session.WriteToV2(&tokv2)
meta.SetSessionToken(&tokv2)
}
var req v2container.GetExtendedACLRequest
req.SetBody(reqBody)
c.prepareRequest(&req, &meta)
return &req, nil
}
// ResContainerEACL groups resulting values of ContainerEACL operation.
type ResContainerEACL struct {
statusRes
table eacl.Table
}
// Table returns eACL table of the requested container.
func (x ResContainerEACL) Table() eacl.Table {
return x.table
}
// ContainerEACL reads eACL table of the FrostFS container.
//
// Exactly one return value is non-nil. By default, server status is returned in res structure.
// Any client's internal or transport errors are returned as `error`.
// If PrmInit.DisableFrostFSFailuresResolution has been called, unsuccessful
// FrostFS status codes are included in the returned result structure,
// otherwise, are also returned as `error`.
//
// Returns an error if parameters are set incorrectly (see PrmContainerEACL docs).
// Context is required and must not be nil. It is used for network communication.
//
// Return statuses:
// - global (see Client docs);
// - *apistatus.ContainerNotFound;
// - *apistatus.EACLNotFound.
func (c *Client) ContainerEACL(ctx context.Context, prm PrmContainerEACL) (*ResContainerEACL, error) {
req, err := prm.buildRequest(c)
if err != nil {
return nil, err
}
if err := signature.SignServiceMessage(&c.prm.Key, req); err != nil {
return nil, fmt.Errorf("sign request: %w", err)
}
resp, err := rpcapi.GetEACL(&c.c, req, client.WithContext(ctx))
if err != nil {
return nil, err
}
var res ResContainerEACL
res.st, err = c.processResponse(resp)
if err != nil || !apistatus.IsSuccessful(res.st) {
return &res, err
}
eACL := resp.GetBody().GetEACL()
if eACL == nil {
return &res, newErrMissingResponseField("eACL")
}
res.table = *eacl.NewTableFromV2(eACL)
return &res, nil
}

View file

@ -239,12 +239,8 @@ func (c *Client) NetMapSnapshot(ctx context.Context, _ PrmNetMapSnapshot) (*ResN
var res ResNetMapSnapshot
res.st, err = c.processResponse(resp)
if err != nil {
return nil, err
}
if !apistatus.IsSuccessful(res.st) {
return &res, nil
if err != nil || !apistatus.IsSuccessful(res.st) {
return &res, err
}
const fieldNetMap = "network map"

View file

@ -148,12 +148,8 @@ func (c *Client) ObjectDelete(ctx context.Context, prm PrmObjectDelete) (*ResObj
var res ResObjectDelete
res.st, err = c.processResponse(resp)
if err != nil {
return nil, err
}
if !apistatus.IsSuccessful(res.st) {
return &res, nil
if err != nil || !apistatus.IsSuccessful(res.st) {
return &res, err
}
const fieldTombstone = "tombstone"

View file

@ -492,12 +492,8 @@ func (c *Client) ObjectHead(ctx context.Context, prm PrmObjectHead) (*ResObjectH
var res ResObjectHead
res.st, err = c.processResponse(resp)
if err != nil {
return nil, err
}
if !apistatus.IsSuccessful(res.st) {
return &res, nil
if err != nil || !apistatus.IsSuccessful(res.st) {
return &res, err
}
res.idObj = *prm.ObjectID

View file

@ -189,12 +189,8 @@ func (c *Client) ObjectHash(ctx context.Context, prm PrmObjectHash) (*ResObjectH
var res ResObjectHash
res.st, err = c.processResponse(resp)
if err != nil {
return nil, err
}
if !apistatus.IsSuccessful(res.st) {
return &res, nil
if err != nil || !apistatus.IsSuccessful(res.st) {
return &res, err
}
res.checksums = resp.GetBody().GetHashList()

View file

@ -20,12 +20,54 @@ import (
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/session"
)
// ObjectPatcher is designed to patch an object.
//
// Must be initialized using Client.ObjectPatchInit, any other
// usage is unsafe.
type ObjectPatcher interface {
// PatchAttributes patches attributes. Attributes can be patched no more than once,
// otherwise, the server returns an error.
//
// Result means success. Failure reason can be received via Close.
PatchAttributes(ctx context.Context, newAttrs []object.Attribute, replace bool) bool
// PatchPayload patches the object's payload.
//
// PatchPayload receives `payloadReader` and thus the payload of the patch is read and sent by chunks of
// `MaxChunkLength` length.
//
// Result means success. Failure reason can be received via Close.
PatchPayload(ctx context.Context, rng *object.Range, payloadReader io.Reader) bool
// Close ends patching the object and returns the result of the operation
// along with the final results. Must be called after using the ObjectPatcher.
//
// Exactly one return value is non-nil. By default, server status is returned in res structure.
// Any client's internal or transport errors are returned as Go built-in error.
// If Client is tuned to resolve FrostFS API statuses, then FrostFS failures
// codes are returned as error.
//
// Return statuses:
// - global (see Client docs);
// - *apistatus.ContainerNotFound;
// - *apistatus.ContainerAccessDenied;
// - *apistatus.ObjectAccessDenied;
// - *apistatus.ObjectAlreadyRemoved;
// - *apistatus.ObjectLocked;
// - *apistatus.ObjectOutOfRange;
// - *apistatus.SessionTokenNotFound;
// - *apistatus.SessionTokenExpired.
Close(_ context.Context) (*ResObjectPatch, error)
}
// ResObjectPatch groups resulting values of ObjectPatch operation.
type ResObjectPatch struct {
statusRes
obj oid.ID
}
// ObjectID returns an object ID of the patched object.
func (r ResObjectPatch) ObjectID() oid.ID {
return r.obj
}
@ -34,43 +76,44 @@ func (r ResObjectPatch) ObjectID() oid.ID {
type PrmObjectPatch struct {
XHeaders []string
Address oid.Address
BearerToken *bearer.Token
Session *session.Object
Local bool
Address oid.Address
Key *ecdsa.PrivateKey
Offset uint64
Length uint64
Chunk []byte
MaxChunkLength int
}
func (c *Client) ObjectPatchInit(ctx context.Context, prm PrmObjectPatch) (*objectPatcher, error) {
// ObjectPatchInit initializes object patcher.
func (c *Client) ObjectPatchInit(ctx context.Context, prm PrmObjectPatch) (ObjectPatcher, error) {
if len(prm.XHeaders)%2 != 0 {
return nil, errorInvalidXHeaders
}
var p objectPatcher
stream, err := rpcapi.Patch(&c.c, &p.respV2, client.WithContext(ctx))
var objectPatcher objectPatcher
stream, err := rpcapi.Patch(&c.c, &objectPatcher.respV2, client.WithContext(ctx))
if err != nil {
return nil, fmt.Errorf("open stream: %w", err)
}
p.key = &c.prm.Key
objectPatcher.addr = prm.Address
objectPatcher.key = &c.prm.Key
if prm.Key != nil {
p.key = prm.Key
objectPatcher.key = prm.Key
}
p.client = c
p.stream = stream
p.addr = prm.Address
objectPatcher.client = c
objectPatcher.stream = stream
p.req.SetBody(&v2object.PatchRequestBody{})
if prm.MaxChunkLength > 0 {
objectPatcher.maxChunkLen = prm.MaxChunkLength
} else {
objectPatcher.maxChunkLen = defaultGRPCPayloadChunkLen
}
objectPatcher.req.SetBody(&v2object.PatchRequestBody{})
meta := new(v2session.RequestMetaHeader)
writeXHeadersToMeta(prm.XHeaders, meta)
@ -87,12 +130,9 @@ func (c *Client) ObjectPatchInit(ctx context.Context, prm PrmObjectPatch) (*obje
meta.SetSessionToken(v2SessionToken)
}
if prm.Local {
meta.SetTTL(1)
}
c.prepareRequest(&objectPatcher.req, meta)
c.prepareRequest(&p.req, meta)
return &p, nil
return &objectPatcher, nil
}
type objectPatcher struct {
@ -111,9 +151,75 @@ type objectPatcher struct {
req v2object.PatchRequest
respV2 v2object.PatchResponse
maxChunkLen int
}
func (x *objectPatcher) Patch(_ context.Context, patch *object.Patch) bool {
func (x *objectPatcher) PatchAttributes(_ context.Context, newAttrs []object.Attribute, replace bool) bool {
return x.patch(&object.Patch{
Address: x.addr,
NewAttributes: newAttrs,
ReplaceAttributes: replace,
})
}
func (x *objectPatcher) PatchPayload(_ context.Context, rng *object.Range, payloadReader io.Reader) bool {
offset := rng.GetOffset()
buf := make([]byte, x.maxChunkLen)
for patchIter := 0; ; patchIter++ {
n, err := payloadReader.Read(buf)
if err != nil && err != io.EOF {
x.err = fmt.Errorf("read payload: %w", err)
return false
}
if n == 0 {
if patchIter == 0 {
if rng.GetLength() == 0 {
x.err = errors.New("zero-length empty payload patch can't be applied")
return false
}
if !x.patch(&object.Patch{
Address: x.addr,
PayloadPatch: &object.PayloadPatch{
Range: rng,
Chunk: []byte{},
},
}) {
return false
}
}
break
}
rngPart := object.NewRange()
if patchIter == 0 {
rngPart.SetOffset(offset)
rngPart.SetLength(rng.GetLength())
} else {
rngPart.SetOffset(offset + rng.GetLength())
}
if !x.patch(&object.Patch{
Address: x.addr,
PayloadPatch: &object.PayloadPatch{
Range: rngPart,
Chunk: buf[:n],
},
}) {
return false
}
if err == io.EOF {
break
}
}
return true
}
func (x *objectPatcher) patch(patch *object.Patch) bool {
x.req.SetBody(patch.ToV2())
x.req.SetVerificationHeader(nil)
@ -124,7 +230,6 @@ func (x *objectPatcher) Patch(_ context.Context, patch *object.Patch) bool {
}
x.err = x.stream.Write(&x.req)
return x.err == nil
}
@ -141,12 +246,8 @@ func (x *objectPatcher) Close(_ context.Context) (*ResObjectPatch, error) {
}
x.res.st, x.err = x.client.processResponse(&x.respV2)
if x.err != nil {
return nil, x.err
}
if !apistatus.IsSuccessful(x.res.st) {
return &x.res, nil
if x.err != nil || !apistatus.IsSuccessful(x.res.st) {
return &x.res, x.err
}
const fieldID = "ID"

295
client/object_patch_test.go Normal file
View file

@ -0,0 +1,295 @@
package client
import (
"bytes"
"context"
"crypto/ecdsa"
"crypto/elliptic"
"crypto/rand"
"testing"
v2object "git.frostfs.info/TrueCloudLab/frostfs-api-go/v2/object"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object"
oidtest "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id/test"
"github.com/stretchr/testify/require"
)
type mockPatchStream struct {
streamedPayloadPatches []*object.PayloadPatch
}
func (m *mockPatchStream) Write(r *v2object.PatchRequest) error {
pp := new(object.PayloadPatch)
pp.FromV2(r.GetBody().GetPatch())
if r.GetBody().GetPatch() != nil {
bodyChunk := r.GetBody().GetPatch().Chunk
pp.Chunk = make([]byte, len(bodyChunk))
copy(pp.Chunk, bodyChunk)
}
m.streamedPayloadPatches = append(m.streamedPayloadPatches, pp)
return nil
}
func (m *mockPatchStream) Close() error {
return nil
}
func TestObjectPatcher(t *testing.T) {
type part struct {
offset int
length int
chunk string
}
for _, test := range []struct {
name string
patchPayload string
rng *object.Range
maxChunkLen int
expectParts []part
}{
{
name: "no split payload patch",
patchPayload: "011111",
rng: newRange(0, 6),
maxChunkLen: defaultGRPCPayloadChunkLen,
expectParts: []part{
{
offset: 0,
length: 6,
chunk: "011111",
},
},
},
{
name: "splitted payload patch",
patchPayload: "012345",
rng: newRange(0, 6),
maxChunkLen: 2,
expectParts: []part{
{
offset: 0,
length: 6,
chunk: "01",
},
{
offset: 6,
length: 0,
chunk: "23",
},
{
offset: 6,
length: 0,
chunk: "45",
},
},
},
{
name: "splitted payload patch with zero-length subpatches",
patchPayload: "0123456789!@",
rng: newRange(0, 4),
maxChunkLen: 2,
expectParts: []part{
{
offset: 0,
length: 4,
chunk: "01",
},
{
offset: 4,
length: 0,
chunk: "23",
},
{
offset: 4,
length: 0,
chunk: "45",
},
{
offset: 4,
length: 0,
chunk: "67",
},
{
offset: 4,
length: 0,
chunk: "89",
},
{
offset: 4,
length: 0,
chunk: "!@",
},
},
},
{
name: "splitted payload patch with zero-length subpatches only",
patchPayload: "0123456789!@",
rng: newRange(0, 0),
maxChunkLen: 2,
expectParts: []part{
{
offset: 0,
length: 0,
chunk: "01",
},
{
offset: 0,
length: 0,
chunk: "23",
},
{
offset: 0,
length: 0,
chunk: "45",
},
{
offset: 0,
length: 0,
chunk: "67",
},
{
offset: 0,
length: 0,
chunk: "89",
},
{
offset: 0,
length: 0,
chunk: "!@",
},
},
},
} {
t.Run(test.name, func(t *testing.T) {
m := &mockPatchStream{}
pk, _ := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
patcher := objectPatcher{
client: &Client{},
stream: m,
addr: oidtest.Address(),
key: pk,
maxChunkLen: test.maxChunkLen,
}
success := patcher.PatchAttributes(context.Background(), nil, false)
require.True(t, success)
success = patcher.PatchPayload(context.Background(), test.rng, bytes.NewReader([]byte(test.patchPayload)))
require.True(t, success)
require.Len(t, m.streamedPayloadPatches, len(test.expectParts)+1)
// m.streamedPayloadPatches[0] is attribute patch, so skip it
for i, part := range test.expectParts {
requireRangeChunk(t, m.streamedPayloadPatches[i+1], part.offset, part.length, part.chunk)
}
})
}
}
func TestRepeatPayloadPatch(t *testing.T) {
t.Run("no payload patch partioning", func(t *testing.T) {
m := &mockPatchStream{}
pk, _ := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
const maxChunkLen = 20
patcher := objectPatcher{
client: &Client{},
stream: m,
addr: oidtest.Address(),
key: pk,
maxChunkLen: maxChunkLen,
}
for _, pp := range []struct {
patchPayload string
rng *object.Range
}{
{
patchPayload: "xxxxxxxxxx",
rng: newRange(1, 6),
},
{
patchPayload: "yyyyyyyyyy",
rng: newRange(5, 9),
},
{
patchPayload: "zzzzzzzzzz",
rng: newRange(10, 0),
},
} {
success := patcher.PatchPayload(context.Background(), pp.rng, bytes.NewReader([]byte(pp.patchPayload)))
require.True(t, success)
}
requireRangeChunk(t, m.streamedPayloadPatches[0], 1, 6, "xxxxxxxxxx")
requireRangeChunk(t, m.streamedPayloadPatches[1], 5, 9, "yyyyyyyyyy")
requireRangeChunk(t, m.streamedPayloadPatches[2], 10, 0, "zzzzzzzzzz")
})
t.Run("payload patch partioning", func(t *testing.T) {
m := &mockPatchStream{}
pk, _ := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
const maxChunkLen = 5
patcher := objectPatcher{
client: &Client{},
stream: m,
addr: oidtest.Address(),
key: pk,
maxChunkLen: maxChunkLen,
}
for _, pp := range []struct {
patchPayload string
rng *object.Range
}{
{
patchPayload: "xxxxxxxxxx",
rng: newRange(1, 6),
},
{
patchPayload: "yyyyyyyyyy",
rng: newRange(5, 9),
},
{
patchPayload: "zzzzzzzzzz",
rng: newRange(10, 0),
},
} {
success := patcher.PatchPayload(context.Background(), pp.rng, bytes.NewReader([]byte(pp.patchPayload)))
require.True(t, success)
}
requireRangeChunk(t, m.streamedPayloadPatches[0], 1, 6, "xxxxx")
requireRangeChunk(t, m.streamedPayloadPatches[1], 7, 0, "xxxxx")
requireRangeChunk(t, m.streamedPayloadPatches[2], 5, 9, "yyyyy")
requireRangeChunk(t, m.streamedPayloadPatches[3], 14, 0, "yyyyy")
requireRangeChunk(t, m.streamedPayloadPatches[4], 10, 0, "zzzzz")
requireRangeChunk(t, m.streamedPayloadPatches[5], 10, 0, "zzzzz")
})
}
func requireRangeChunk(t *testing.T, pp *object.PayloadPatch, offset, length int, chunk string) {
require.NotNil(t, pp)
require.Equal(t, uint64(offset), pp.Range.GetOffset())
require.Equal(t, uint64(length), pp.Range.GetLength())
require.Equal(t, []byte(chunk), pp.Chunk)
}
func newRange(offest, length uint64) *object.Range {
rng := &object.Range{}
rng.SetOffset(offest)
rng.SetLength(length)
return rng
}

View file

@ -156,12 +156,8 @@ func (x *objectWriterRaw) Close(_ context.Context) (*ResObjectPut, error) {
}
x.res.st, x.err = x.client.processResponse(&x.respV2)
if x.err != nil {
return nil, x.err
}
if !apistatus.IsSuccessful(x.res.st) {
return &x.res, nil
if x.err != nil || !apistatus.IsSuccessful(x.res.st) {
return &x.res, x.err
}
const fieldID = "ID"

View file

@ -167,7 +167,7 @@ func (c *Client) ObjectPutSingle(ctx context.Context, prm PrmObjectPutSingle) (*
var res ResObjectPutSingle
res.st, err = c.processResponse(resp)
if err != nil {
return nil, err
return &res, err
}
res.epoch = resp.GetMetaHeader().GetEpoch()

View file

@ -308,7 +308,7 @@ func (x *Container) SetAttribute(key, value string) {
attrs := x.v2.GetAttributes()
ln := len(attrs)
for i := 0; i < ln; i++ {
for i := range ln {
if attrs[i].GetKey() == key {
attrs[i].SetValue(value)
return
@ -356,8 +356,7 @@ func (x Container) IterateUserAttributes(f func(key, val string)) {
attrs := x.v2.GetAttributes()
for _, attr := range attrs {
key := attr.GetKey()
if !strings.HasPrefix(key, container.SysAttributePrefix) &&
!strings.HasPrefix(key, container.SysAttributePrefixNeoFS) {
if !strings.HasPrefix(key, container.SysAttributePrefix) {
f(key, attr.GetValue())
}
}
@ -417,8 +416,7 @@ func DisableHomomorphicHashing(cnr *Container) {
//
// Zero Container has enabled hashing.
func IsHomomorphicHashingDisabled(cnr Container) bool {
return cnr.Attribute(container.SysAttributeHomomorphicHashing) == attributeHomoHashEnabled ||
cnr.Attribute(container.SysAttributeHomomorphicHashingNeoFS) == attributeHomoHashEnabled
return cnr.Attribute(container.SysAttributeHomomorphicHashing) == attributeHomoHashEnabled
}
// Domain represents information about container domain registered in the NNS
@ -467,9 +465,6 @@ func ReadDomain(cnr Container) (res Domain) {
if name := cnr.Attribute(container.SysAttributeName); name != "" {
res.SetName(name)
res.SetZone(cnr.Attribute(container.SysAttributeZone))
} else if name = cnr.Attribute(container.SysAttributeNameNeoFS); name != "" {
res.SetName(name)
res.SetZone(cnr.Attribute(container.SysAttributeZoneNeoFS))
}
return

View file

@ -150,7 +150,7 @@ func assertContainsAttribute(t *testing.T, m v2container.Container, key, val str
}
func TestContainer_Attribute(t *testing.T) {
const attrKey1, attrKey2 = v2container.SysAttributePrefix + "key1", v2container.SysAttributePrefixNeoFS + "key2"
const attrKey1, attrKey2 = v2container.SysAttributePrefix + "key1", v2container.SysAttributePrefix + "key2"
const attrVal1, attrVal2 = "val1", "val2"
val := containertest.Container()

BIN
doc/image/pool.png Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 194 KiB

View file

@ -428,6 +428,13 @@ Comparison operators (all binary):
- `LE`: less or equal
- `LT`: less than
Pattern operator:
- `LIKE`: specifies pattern for an attribute. Uses as a wildcard symbol `*`.
- `... ATTR LIKE "VAL"` - the behaviour is equal to `EQ`
- `... ATTR LIKE "VAL*"` - matches all which starts with `VAL`
- `... ATTR LIKE "*VAL"` - matches all which ends with `VAL`
- `... ATTR LIKE "*VAL*"` - matches all which contains `VAL`
Logical operators:
- `NOT`: negation (unary)
- `AND`: conjunction (binary)

27
doc/pool.md Normal file
View file

@ -0,0 +1,27 @@
# Node connection pool
* Distributes requests between fixed number of nodes
* Wraps
The distribution between nodes in connection pool is based on priority and weight parameters from
NodeParam struct. The distribution model is presented below. On this scheme nodes with the same
priority have the same color.
![](./image/pool.png "Pool connections distribution model")
## Priority
Pool component forwards request to the nodes with the highest priority (the lower the value -
the higher the priority). The `node 1` from the image's scenario (I) is healthy
and has the highest priority (1), that's why the pool forwards requests from it. There are no other
nodes with priority 1, so `node 1` receives all requests. In the second scenario (II) `node 1`
becomes unhealthy. In that case pool tries to connect nodes with next in priority nodes e.g.
`Node 4` and `node 2`. If all of them become unhealthy too, the pool sends requests to nodes with
priority 3 in scenario (III) and so on.
## Weights
If there are several nodes with the same priority, then requests are distributed randomly between
these nodes based on their weights. To do that the proportion of weights is calculated.
For example, for `node 2` and `node 4` with weights 2 and 8 the distribution would be 20 and 80 percent
respectively.

View file

@ -286,13 +286,13 @@ func equalRecords(r1, r2 Record) bool {
return false
}
for i := 0; i < len(fs1); i++ {
for i := range len(fs1) {
if !equalFilters(fs1[i], fs2[i]) {
return false
}
}
for i := 0; i < len(ts1); i++ {
for i := range len(ts1) {
if !equalTargets(ts1[i], ts2[i]) {
return false
}

View file

@ -212,7 +212,7 @@ func EqualTables(t1, t2 Table) bool {
return false
}
for i := 0; i < len(rs1); i++ {
for i := range len(rs1) {
if !equalRecords(rs1[i], rs2[i]) {
return false
}

View file

@ -51,7 +51,7 @@ func SetTargetECDSAKeys(t *Target, pubs ...*ecdsa.PublicKey) {
binKeys = make([][]byte, 0, ln)
}
for i := 0; i < ln; i++ {
for i := range ln {
binKeys = append(binKeys, (*keys.PublicKey)(pubs[i]).Bytes())
}
@ -67,7 +67,7 @@ func TargetECDSAKeys(t *Target) []*ecdsa.PublicKey {
pubs := make([]*ecdsa.PublicKey, ln)
for i := 0; i < ln; i++ {
for i := range ln {
p := new(keys.PublicKey)
if p.DecodeBytes(binKeys[i]) == nil {
pubs[i] = (*ecdsa.PublicKey)(p)
@ -169,7 +169,7 @@ func equalTargets(t1, t2 Target) bool {
return false
}
for i := 0; i < len(keys1); i++ {
for i := range len(keys1) {
if !bytes.Equal(keys1[i], keys2[i]) {
return false
}

View file

@ -19,7 +19,7 @@ func baseBenchmarkTableBinaryComparison(b *testing.B, factor int) {
b.StopTimer()
b.ResetTimer()
b.StartTimer()
for i := 0; i < b.N; i++ {
for range b.N {
got, _ := t.Marshal()
if !bytes.Equal(exp, got) {
b.Fail()
@ -38,7 +38,7 @@ func baseBenchmarkTableEqualsComparison(b *testing.B, factor int) {
b.StopTimer()
b.ResetTimer()
b.StartTimer()
for i := 0; i < b.N; i++ {
for range b.N {
if !eacl.EqualTables(*t, *t2) {
b.Fail()
}
@ -76,7 +76,7 @@ func TargetN(n int) *eacl.Target {
x.SetRole(eacl.RoleSystem)
keys := make([][]byte, n)
for i := 0; i < n; i++ {
for i := range n {
keys[i] = make([]byte, 32)
rand.Read(keys[i])
}
@ -94,7 +94,7 @@ func RecordN(n int) *eacl.Record {
x.SetOperation(eacl.OperationRangeHash)
x.SetTargets(*TargetN(n))
for i := 0; i < n; i++ {
for range n {
x.AddFilter(eacl.HeaderFromObject, eacl.MatchStringEqual, "", cidtest.ID().EncodeToString())
}
@ -106,7 +106,7 @@ func TableN(n int) *eacl.Table {
x.SetCID(cidtest.ID())
for i := 0; i < n; i++ {
for range n {
x.AddRecord(RecordN(n))
}

28
go.mod
View file

@ -1,13 +1,13 @@
module git.frostfs.info/TrueCloudLab/frostfs-sdk-go
go 1.21
go 1.22
require (
git.frostfs.info/TrueCloudLab/frostfs-api-go/v2 v2.16.1-0.20240730145254-c27b978770a3
git.frostfs.info/TrueCloudLab/frostfs-api-go/v2 v2.16.1-0.20240916093537-13fa0da3741e
git.frostfs.info/TrueCloudLab/frostfs-contract v0.19.3-0.20240621131249-49e5270f673e
git.frostfs.info/TrueCloudLab/hrw v1.2.1
git.frostfs.info/TrueCloudLab/tzhash v1.8.0
github.com/antlr4-go/antlr/v4 v4.13.0
github.com/antlr4-go/antlr/v4 v4.13.1
github.com/google/uuid v1.6.0
github.com/hashicorp/golang-lru/v2 v2.0.7
github.com/klauspost/reedsolomon v1.12.1
@ -15,20 +15,22 @@ require (
github.com/nspcc-dev/neo-go v0.106.2
github.com/stretchr/testify v1.9.0
go.uber.org/zap v1.27.0
google.golang.org/grpc v1.62.0
google.golang.org/protobuf v1.33.0
google.golang.org/grpc v1.66.2
google.golang.org/protobuf v1.34.1
gopkg.in/yaml.v3 v3.0.1
)
require (
git.frostfs.info/TrueCloudLab/frostfs-crypto v0.6.0 // indirect
git.frostfs.info/TrueCloudLab/rfc6979 v0.4.0 // indirect
github.com/VictoriaMetrics/easyproto v0.1.4 // indirect
github.com/davecgh/go-spew v1.1.1 // indirect
github.com/decred/dcrd/dcrec/secp256k1/v4 v4.2.0 // indirect
github.com/golang/protobuf v1.5.3 // indirect
github.com/golang/snappy v0.0.1 // indirect
github.com/gorilla/websocket v1.5.1 // indirect
github.com/josharian/intern v1.0.0 // indirect
github.com/klauspost/cpuid/v2 v2.2.6 // indirect
github.com/mailru/easyjson v0.7.7 // indirect
github.com/nspcc-dev/go-ordered-json v0.0.0-20240301084351-0246b013f8b2 // indirect
github.com/nspcc-dev/neo-go/pkg/interop v0.0.0-20240521091047-78685785716d // indirect
github.com/nspcc-dev/rfc6979 v0.2.1 // indirect
@ -37,11 +39,11 @@ require (
github.com/twmb/murmur3 v1.1.8 // indirect
go.etcd.io/bbolt v1.3.9 // indirect
go.uber.org/multierr v1.11.0 // indirect
golang.org/x/crypto v0.21.0 // indirect
golang.org/x/exp v0.0.0-20240222234643-814bf88cf225 // indirect
golang.org/x/net v0.23.0 // indirect
golang.org/x/sync v0.6.0 // indirect
golang.org/x/sys v0.18.0 // indirect
golang.org/x/text v0.14.0 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20240221002015-b0ce06bbee7c // indirect
golang.org/x/crypto v0.24.0 // indirect
golang.org/x/exp v0.0.0-20240506185415-9bf2ced13842 // indirect
golang.org/x/net v0.26.0 // indirect
golang.org/x/sync v0.7.0 // indirect
golang.org/x/sys v0.21.0 // indirect
golang.org/x/text v0.16.0 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20240604185151-ef581f913117 // indirect
)

72
go.sum
View file

@ -1,5 +1,5 @@
git.frostfs.info/TrueCloudLab/frostfs-api-go/v2 v2.16.1-0.20240730145254-c27b978770a3 h1:BbtF/98HU0nBl4szdDYAV3XadNE5sJ92uSFmNePQmfA=
git.frostfs.info/TrueCloudLab/frostfs-api-go/v2 v2.16.1-0.20240730145254-c27b978770a3/go.mod h1:OBDSr+DqV1z4VDouoX3YMleNc4DPBVBWTG3WDT2PK1o=
git.frostfs.info/TrueCloudLab/frostfs-api-go/v2 v2.16.1-0.20240916093537-13fa0da3741e h1:740ABnOBYx4o6jxULHdSSnVW2fYIO35ohg+Uz59sxd0=
git.frostfs.info/TrueCloudLab/frostfs-api-go/v2 v2.16.1-0.20240916093537-13fa0da3741e/go.mod h1:F5GS7hRb62PUy5sTYDC4ajVdeffoAfjHSSHTKUJEaYU=
git.frostfs.info/TrueCloudLab/frostfs-contract v0.19.3-0.20240621131249-49e5270f673e h1:kcBqZBiFIUBATUqEuvVigtkJJWQ2Gug/eYXn967o3M4=
git.frostfs.info/TrueCloudLab/frostfs-contract v0.19.3-0.20240621131249-49e5270f673e/go.mod h1:F/fe1OoIDKr5Bz99q4sriuHDuf3aZefZy9ZsCqEtgxc=
git.frostfs.info/TrueCloudLab/frostfs-crypto v0.6.0 h1:FxqFDhQYYgpe41qsIHVOcdzSVCB8JNSfPG7Uk4r2oSk=
@ -10,14 +10,16 @@ git.frostfs.info/TrueCloudLab/rfc6979 v0.4.0 h1:M2KR3iBj7WpY3hP10IevfIB9MURr4O9m
git.frostfs.info/TrueCloudLab/rfc6979 v0.4.0/go.mod h1:okpbKfVYf/BpejtfFTfhZqFP+sZ8rsHrP8Rr/jYPNRc=
git.frostfs.info/TrueCloudLab/tzhash v1.8.0 h1:UFMnUIk0Zh17m8rjGHJMqku2hCgaXDqjqZzS4gsb4UA=
git.frostfs.info/TrueCloudLab/tzhash v1.8.0/go.mod h1:dhY+oy274hV8wGvGL4MwwMpdL3GYvaX1a8GQZQHvlF8=
github.com/antlr4-go/antlr/v4 v4.13.0 h1:lxCg3LAv+EUK6t1i0y1V6/SLeUi0eKEKdhQAlS8TVTI=
github.com/antlr4-go/antlr/v4 v4.13.0/go.mod h1:pfChB/xh/Unjila75QW7+VU4TSnWnnk9UTnmpPaOR2g=
github.com/VictoriaMetrics/easyproto v0.1.4 h1:r8cNvo8o6sR4QShBXQd1bKw/VVLSQma/V2KhTBPf+Sc=
github.com/VictoriaMetrics/easyproto v0.1.4/go.mod h1:QlGlzaJnDfFd8Lk6Ci/fuLxfTo3/GThPs2KH23mv710=
github.com/antlr4-go/antlr/v4 v4.13.1 h1:SqQKkuVZ+zWkMMNkjy5FZe5mr5WURWnlpmOuzYWrPrQ=
github.com/antlr4-go/antlr/v4 v4.13.1/go.mod h1:GKmUxMtwp6ZgGwZSva4eWPC5mS6vUAmOABFgjdkM7Nw=
github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM=
github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw=
github.com/bits-and-blooms/bitset v1.8.0 h1:FD+XqgOZDUxxZ8hzoBFuV9+cGWY9CslN6d5MS5JVb4c=
github.com/bits-and-blooms/bitset v1.8.0/go.mod h1:7hO7Gc7Pp1vODcmWvKMRA9BNmbv6a/7QIWpPxHddWR8=
github.com/cespare/xxhash/v2 v2.2.0 h1:DC2CZ1Ep5Y4k3ZQ899DldepgrayRUGE6BBZ/cd9Cj44=
github.com/cespare/xxhash/v2 v2.2.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs=
github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
github.com/consensys/bavard v0.1.13 h1:oLhMLOFGTLdlda/kma4VOJazblc7IM5y5QPd2A/YjhQ=
github.com/consensys/bavard v0.1.13/go.mod h1:9ItSMtA/dXMAiL7BG6bqW2m3NdSEObYWoH223nGHukI=
github.com/consensys/gnark-crypto v0.12.2-0.20231013160410-1f65e75b6dfb h1:f0BMgIjhZy4lSRHCXFbQst85f5agZAjtDMixQqBWNpc=
@ -39,15 +41,11 @@ github.com/golang/protobuf v1.4.0-rc.2/go.mod h1:LlEzMj4AhA7rCAGe4KMBDvJI+AwstrU
github.com/golang/protobuf v1.4.0-rc.4.0.20200313231945-b860323f09d0/go.mod h1:WU3c8KckQ9AFe+yFwt9sWVRKCVIyN9cPHBJSNnbL67w=
github.com/golang/protobuf v1.4.0/go.mod h1:jodUvKwWbYaEsadDk5Fwe5c77LiNKVO9IDvqG2KuDX0=
github.com/golang/protobuf v1.4.2/go.mod h1:oDoupMAO8OvCJWAcko0GGGIgR6R6ocIYbsSw735rRwI=
github.com/golang/protobuf v1.5.0/go.mod h1:FsONVRAS9T7sI+LIUmWTfcYkHO4aIWwzhcaSAoJOfIk=
github.com/golang/protobuf v1.5.3 h1:KhyjKVUg7Usr/dYsdSqoFveMYd5ko72D+zANwlG1mmg=
github.com/golang/protobuf v1.5.3/go.mod h1:XVQd3VNwM+JqD3oG2Ue2ip4fOMUkwXdXDdiuN0vRsmY=
github.com/golang/snappy v0.0.1 h1:Qgr9rKW7uDUkrbSmQeiDsGa8SjGyCOGtuasMWwvp2P4=
github.com/golang/snappy v0.0.1/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q=
github.com/google/go-cmp v0.3.0/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU=
github.com/google/go-cmp v0.3.1/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU=
github.com/google/go-cmp v0.4.0/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/google/go-cmp v0.5.5/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/google/go-cmp v0.6.0 h1:ofyhxvXcZhMsU5ulbFiLKl/XBFqE1GSq7atu8tAmTRI=
github.com/google/go-cmp v0.6.0/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY=
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
@ -59,6 +57,8 @@ github.com/hashicorp/golang-lru/v2 v2.0.7/go.mod h1:QeFd9opnmA6QUJc5vARoKUSoFhyf
github.com/holiman/uint256 v1.2.4 h1:jUc4Nk8fm9jZabQuqr2JzednajVmBpC+oiTiXZJEApU=
github.com/holiman/uint256 v1.2.4/go.mod h1:EOMSn4q6Nyt9P6efbI3bueV4e1b3dGlUCXeiRV4ng7E=
github.com/hpcloud/tail v1.0.0/go.mod h1:ab1qPbhIpdTxEkNHXyeSf5vhxWSCs/tWer42PpOxQnU=
github.com/josharian/intern v1.0.0 h1:vlS4z54oSdjm0bgjRigI+G1HpF+tI+9rE5LLzOg8HmY=
github.com/josharian/intern v1.0.0/go.mod h1:5DoeVV0s6jJacbCEi61lwdGj/aVlrQvzHFFd8Hwg//Y=
github.com/klauspost/cpuid/v2 v2.2.6 h1:ndNyv040zDGIDh8thGkXYjnFtiN02M1PVVF+JE/48xc=
github.com/klauspost/cpuid/v2 v2.2.6/go.mod h1:Lcz8mBdAVJIBVzewtcLocK12l3Y+JytZYpaMropDUws=
github.com/klauspost/reedsolomon v1.12.1 h1:NhWgum1efX1x58daOBGCFWcxtEhOhXKKl1HAPQUp03Q=
@ -67,6 +67,8 @@ github.com/kr/pretty v0.1.0 h1:L/CwN0zerZDmRFUapSPitk6f+Q3+0za1rQkzVuMiMFI=
github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo=
github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=
github.com/mailru/easyjson v0.7.7 h1:UGYAvKxe3sBsEDzO8ZeWOSlIQfWFlxbzLZe7hwFURr0=
github.com/mailru/easyjson v0.7.7/go.mod h1:xzfreul335JAWq5oZzymOObrkdz5UnU4kGfJJLY9Nlc=
github.com/mmcloughlin/addchain v0.4.0 h1:SobOdjm2xLj1KkXN5/n0xTIWyZA2+s99UCY1iPfkHRY=
github.com/mmcloughlin/addchain v0.4.0/go.mod h1:A86O+tHqZLMNO4w6ZZ4FlVQEadcoqkyU72HC5wJ4RlU=
github.com/mr-tron/base58 v1.2.0 h1:T/HDJBh4ZCPbU39/+c3rRvE0uKBQlU27+QI8LJ4t64o=
@ -120,21 +122,21 @@ go.uber.org/zap v1.27.0 h1:aJMhYGrd5QSmlpLMr2MftRKl7t8J8PTZPA732ud/XR8=
go.uber.org/zap v1.27.0/go.mod h1:GB2qFLM7cTU87MWRP2mPIjqfIDnGu+VIO4V/SdhGo2E=
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
golang.org/x/crypto v0.21.0 h1:X31++rzVUdKhX5sWmSOFZxx8UW/ldWx55cbf08iNAMA=
golang.org/x/crypto v0.21.0/go.mod h1:0BP7YvVV9gBbVKyeTG0Gyn+gZm94bibOW5BjDEYAOMs=
golang.org/x/exp v0.0.0-20240222234643-814bf88cf225 h1:LfspQV/FYTatPTr/3HzIcmiUFH7PGP+OQ6mgDYo3yuQ=
golang.org/x/exp v0.0.0-20240222234643-814bf88cf225/go.mod h1:CxmFvTBINI24O/j8iY7H1xHzx2i4OsyguNBmN/uPtqc=
golang.org/x/mod v0.16.0 h1:QX4fJ0Rr5cPQCF7O9lh9Se4pmwfwskqZfq5moyldzic=
golang.org/x/mod v0.16.0/go.mod h1:hTbmBsO62+eylJbnUtE2MGJUyE7QWk4xUqPFrRgJ+7c=
golang.org/x/crypto v0.24.0 h1:mnl8DM0o513X8fdIkmyFE/5hTYxbwYOjDS/+rK6qpRI=
golang.org/x/crypto v0.24.0/go.mod h1:Z1PMYSOR5nyMcyAVAIQSKCDwalqy85Aqn1x3Ws4L5DM=
golang.org/x/exp v0.0.0-20240506185415-9bf2ced13842 h1:vr/HnozRka3pE4EsMEg1lgkXJkTFJCVUX+S/ZT6wYzM=
golang.org/x/exp v0.0.0-20240506185415-9bf2ced13842/go.mod h1:XtvwrStGgqGPLc4cjQfWqZHG1YFdYs6swckp8vpsjnc=
golang.org/x/mod v0.17.0 h1:zY54UmvipHiNd+pm+m0x9KhZ9hl1/7QNMyxXbc6ICqA=
golang.org/x/mod v0.17.0/go.mod h1:hTbmBsO62+eylJbnUtE2MGJUyE7QWk4xUqPFrRgJ+7c=
golang.org/x/net v0.0.0-20180906233101-161cd47e91fd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
golang.org/x/net v0.0.0-20200520004742-59133d7f0dd7/go.mod h1:qpuaurCH72eLCgpAm/N6yyVIVM9cpaDIP3A8BGJEC5A=
golang.org/x/net v0.0.0-20200813134508-3edf25e44fcc/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA=
golang.org/x/net v0.23.0 h1:7EYJ93RZ9vYSZAIb2x3lnuvqO5zneoD6IvWjuhfxjTs=
golang.org/x/net v0.23.0/go.mod h1:JKghWKKOSdJwpW2GEx0Ja7fmaKnMsbu+MWVZTokSYmg=
golang.org/x/net v0.26.0 h1:soB7SVo0PWrY4vPW/+ay0jKDNScG2X9wFeYlXIvJsOQ=
golang.org/x/net v0.26.0/go.mod h1:5YKkiSynbBIh3p6iOc/vibscux0x38BZDkn8sCUPxHE=
golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.6.0 h1:5BMeUDZ7vkXGfEr1x9B4bRcTH4lpkTkpdh0T/J+qjbQ=
golang.org/x/sync v0.6.0/go.mod h1:Czt+wKu1gCyEFDUtn0jG5QVvpJ6rzVqr5aXyt9drQfk=
golang.org/x/sync v0.7.0 h1:YsImfSBoP9QPYL0xyKJPq0gcaJdG3rInoqxTWbfQu9M=
golang.org/x/sync v0.7.0/go.mod h1:Czt+wKu1gCyEFDUtn0jG5QVvpJ6rzVqr5aXyt9drQfk=
golang.org/x/sys v0.0.0-20180909124046-d0be0721c37e/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
@ -146,35 +148,33 @@ golang.org/x/sys v0.0.0-20200519105757-fe76b779f299/go.mod h1:h1NjWce9XRLGQEsW7w
golang.org/x/sys v0.0.0-20200814200057-3d37ad5750ed/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20220114195835-da31bd327af9/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.5.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.18.0 h1:DBdB3niSjOA/O0blCZBqDefyWNYveAYMNF1Wum0DYQ4=
golang.org/x/sys v0.18.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/term v0.18.0 h1:FcHjZXDMxI8mM3nwhX9HlKop4C0YQvCVCdwYl2wOtE8=
golang.org/x/term v0.18.0/go.mod h1:ILwASektA3OnRv7amZ1xhE/KTR+u50pbXfZ03+6Nx58=
golang.org/x/sys v0.21.0 h1:rF+pYz3DAGSQAxAu1CbC7catZg4ebC4UIeIhKxBZvws=
golang.org/x/sys v0.21.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/term v0.21.0 h1:WVXCp+/EBEHOj53Rvu+7KiT/iElMrO8ACK16SMZ3jaA=
golang.org/x/term v0.21.0/go.mod h1:ooXLefLobQVslOqselCNF4SxFAaoS6KujMbsGzSDmX0=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.2/go.mod h1:bEr9sfX3Q8Zfm5fL9x+3itogRgK3+ptLWKqgva+5dAk=
golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
golang.org/x/text v0.14.0 h1:ScX5w1eTa3QqT8oi6+ziP7dTV1S2+ALU0bI+0zXKWiQ=
golang.org/x/text v0.14.0/go.mod h1:18ZOQIKpY8NJVqYksKHtTdi31H5itFRjB5/qKTNYzSU=
golang.org/x/text v0.16.0 h1:a94ExnEXNtEwYLGJSIUxnWoxoRz/ZcCsV63ROupILh4=
golang.org/x/text v0.16.0/go.mod h1:GhwF1Be+LQoKShO3cGOHzqOgRrGaYc9AvblQOmPVHnI=
golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.19.0 h1:tfGCXNR1OsFG+sVdLAitlpjAvD/I6dHDKnYrpEZUHkw=
golang.org/x/tools v0.19.0/go.mod h1:qoJWxmGSIBmAeriMx19ogtrEPrGtDbPK634QFIcLAhc=
golang.org/x/tools v0.21.1-0.20240508182429-e35e4ccd0d2d h1:vU5i/LfpvrRCpgM/VPfJLg5KjxD3E+hfT1SH+d9zLwg=
golang.org/x/tools v0.21.1-0.20240508182429-e35e4ccd0d2d/go.mod h1:aiJjzUbINMkxbQROHiO6hDPo2LHcIPhhQsa9DLh0yGk=
golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1 h1:go1bK/D/BFZV2I8cIQd1NKEZ+0owSTG1fDTci4IqFcE=
golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
google.golang.org/genproto/googleapis/rpc v0.0.0-20240221002015-b0ce06bbee7c h1:NUsgEN92SQQqzfA+YtqYNqYmB3DMMYLlIwUZAQFVFbo=
google.golang.org/genproto/googleapis/rpc v0.0.0-20240221002015-b0ce06bbee7c/go.mod h1:H4O17MA/PE9BsGx3w+a+W2VOLLD1Qf7oJneAoU6WktY=
google.golang.org/grpc v1.62.0 h1:HQKZ/fa1bXkX1oFOvSjmZEUL8wLSaZTjCcLAlmZRtdk=
google.golang.org/grpc v1.62.0/go.mod h1:IWTG0VlJLCh1SkC58F7np9ka9mx/WNkjl4PGJaiq+QE=
google.golang.org/genproto/googleapis/rpc v0.0.0-20240604185151-ef581f913117 h1:1GBuWVLM/KMVUv1t1En5Gs+gFZCNd360GGb4sSxtrhU=
google.golang.org/genproto/googleapis/rpc v0.0.0-20240604185151-ef581f913117/go.mod h1:EfXuqaE1J41VCDicxHzUDm+8rk+7ZdXzHV0IhO/I6s0=
google.golang.org/grpc v1.66.2 h1:3QdXkuq3Bkh7w+ywLdLvM56cmGvQHUMZpiCzt6Rqaoo=
google.golang.org/grpc v1.66.2/go.mod h1:s3/l6xSSCURdVfAnL+TqCNMyTDAGN6+lZeVxnZR128Y=
google.golang.org/protobuf v0.0.0-20200109180630-ec00e32a8dfd/go.mod h1:DFci5gLYBciE7Vtevhsrf46CRTquxDuWsQurQQe4oz8=
google.golang.org/protobuf v0.0.0-20200221191635-4d8936d0db64/go.mod h1:kwYJMbMJ01Woi6D6+Kah6886xMZcty6N08ah7+eCXa0=
google.golang.org/protobuf v0.0.0-20200228230310-ab0ca4ff8a60/go.mod h1:cfTl7dwQJ+fmap5saPgwCLgHXTUD7jkjRqWcaiX5VyM=
google.golang.org/protobuf v1.20.1-0.20200309200217-e05f789c0967/go.mod h1:A+miEFZTKqfCUM6K7xSMQL9OKL/b6hQv+e19PK+JZNE=
google.golang.org/protobuf v1.21.0/go.mod h1:47Nbq4nVaFHyn7ilMalzfO3qCViNmqZ2kzikPIcrTAo=
google.golang.org/protobuf v1.23.0/go.mod h1:EGpADcykh3NcUnDUJcl1+ZksZNG86OlYog2l/sGQquU=
google.golang.org/protobuf v1.26.0-rc.1/go.mod h1:jlhhOSvTdKEhbULTjvd4ARK9grFBp09yW+WbY/TyQbw=
google.golang.org/protobuf v1.26.0/go.mod h1:9q0QmTI4eRPtz6boOQmLYwt+qCgq0jsYwAQnmE0givc=
google.golang.org/protobuf v1.33.0 h1:uNO2rsAINq/JlFpSdYEKIZ0uKD/R9cpdv0T+yoGwGmI=
google.golang.org/protobuf v1.33.0/go.mod h1:c6P6GXX6sHbq/GpV6MGZEdwhWPcYBgnhAHhKbcUYpos=
google.golang.org/protobuf v1.34.1 h1:9ddQBjfCyZPOHPUiPxpYESBLc+T8P3E+Vo4IbKZgFWg=
google.golang.org/protobuf v1.34.1/go.mod h1:c6P6GXX6sHbq/GpV6MGZEdwhWPcYBgnhAHhKbcUYpos=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15 h1:YR8cESwS4TdDjEe65xsg0ogRM/Nc3DYOhEAlW+xobZo=
gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=

View file

@ -73,8 +73,8 @@ func newMinAgg() aggregator {
// newReverseMinNorm returns a normalizer which
// normalize values in range of 0.0 to 1.0 to a minimum value.
func newReverseMinNorm(min float64) normalizer {
return &reverseMinNorm{min: min}
func newReverseMinNorm(minV float64) normalizer {
return &reverseMinNorm{min: minV}
}
// newSigmoidNorm returns a normalizer which
@ -125,22 +125,22 @@ func (a *meanIQRAgg) Compute() float64 {
slices.Sort(a.arr)
var min, max float64
var minV, maxV float64
const minLn = 4
if l < minLn {
min, max = a.arr[0], a.arr[l-1]
minV, maxV = a.arr[0], a.arr[l-1]
} else {
start, end := l/minLn, l*3/minLn-1
min, max = a.arr[start], a.arr[end]
minV, maxV = a.arr[start], a.arr[end]
}
count := 0
sum := float64(0)
for _, e := range a.arr {
if e >= min && e <= max {
if e >= minV && e <= maxV {
sum += e
count++
}

View file

@ -47,7 +47,7 @@ func BenchmarkNetmap_ContainerNodes(b *testing.B) {
b.ResetTimer()
b.ReportAllocs()
for i := 0; i < b.N; i++ {
for range b.N {
_, err := nm.ContainerNodes(p, pivot)
if err != nil {
b.Fatal(err)

View file

@ -94,14 +94,14 @@ func (c *context) addUsedNodes(ns ...NodeInfo) {
func defaultWeightFunc(ns nodes) weightFunc {
mean := newMeanAgg()
min := newMinAgg()
minV := newMinAgg()
for i := range ns {
mean.Add(float64(ns[i].capacity()))
min.Add(float64(ns[i].Price()))
minV.Add(float64(ns[i].Price()))
}
return newWeightFunc(
newSigmoidNorm(mean.Compute()),
newReverseMinNorm(min.Compute()))
newReverseMinNorm(minV.Compute()))
}

View file

@ -3,6 +3,7 @@ package netmap
import (
"fmt"
"strconv"
"strings"
"git.frostfs.info/TrueCloudLab/frostfs-api-go/v2/netmap"
)
@ -11,6 +12,8 @@ import (
// which points to the whole netmap.
const mainFilterName = "*"
const likeWildcard = "*"
// processFilters processes filters and returns error is any of them is invalid.
func (c *context) processFilters(p PlacementPolicy) error {
for i := range p.filters {
@ -53,7 +56,7 @@ func (c *context) processFilter(f netmap.Filter, top bool) error {
}
switch op {
case netmap.EQ, netmap.NE:
case netmap.EQ, netmap.NE, netmap.LIKE:
case netmap.GT, netmap.GE, netmap.LT, netmap.LE:
val := f.GetValue()
n, err := strconv.ParseUint(val, 10, 64)
@ -110,6 +113,19 @@ func (c *context) matchKeyValue(f *netmap.Filter, b NodeInfo) bool {
switch op := f.GetOp(); op {
case netmap.EQ:
return b.Attribute(f.GetKey()) == f.GetValue()
case netmap.LIKE:
str, prefix := strings.CutPrefix(f.GetValue(), likeWildcard)
str, suffix := strings.CutSuffix(str, likeWildcard)
if prefix && suffix {
return strings.Contains(b.Attribute(f.GetKey()), str)
}
if prefix && !suffix {
return strings.HasSuffix(b.Attribute(f.GetKey()), str)
}
if !prefix && suffix {
return strings.HasPrefix(b.Attribute(f.GetKey()), str)
}
return b.Attribute(f.GetKey()) == f.GetValue()
case netmap.NE:
return b.Attribute(f.GetKey()) != f.GetValue()
default:

View file

@ -460,6 +460,10 @@ func (x *NodeInfo) SortAttributes() {
// SetOffline sets the state of the node to "offline". When a node updates
// information about itself in the network map, this action is interpreted as
// an intention to leave the network.
//
// See also IsOffline.
//
// Deprecated: use SetStatus instead.
func (x *NodeInfo) SetOffline() {
x.m.SetState(netmap.Offline)
}
@ -470,6 +474,8 @@ func (x *NodeInfo) SetOffline() {
// mean online).
//
// See also SetOffline.
//
// Deprecated: use Status instead.
func (x NodeInfo) IsOffline() bool {
return x.m.GetState() == netmap.Offline
}
@ -479,6 +485,8 @@ func (x NodeInfo) IsOffline() bool {
// action is interpreted as an intention to enter the network.
//
// See also IsOnline.
//
// Deprecated: use SetStatus instead.
func (x *NodeInfo) SetOnline() {
x.m.SetState(netmap.Online)
}
@ -489,6 +497,8 @@ func (x *NodeInfo) SetOnline() {
// mean offline).
//
// See also SetOnline.
//
// Deprecated: use Status instead.
func (x NodeInfo) IsOnline() bool {
return x.m.GetState() == netmap.Online
}
@ -498,6 +508,8 @@ func (x NodeInfo) IsOnline() bool {
// state declares temporal unavailability for a node.
//
// See also IsMaintenance.
//
// Deprecated: use SetStatus instead.
func (x *NodeInfo) SetMaintenance() {
x.m.SetState(netmap.Maintenance)
}
@ -507,6 +519,63 @@ func (x *NodeInfo) SetMaintenance() {
// Zero NodeInfo has undefined state.
//
// See also SetMaintenance.
//
// Deprecated: use Status instead.
func (x NodeInfo) IsMaintenance() bool {
return x.m.GetState() == netmap.Maintenance
}
type NodeState netmap.NodeState
const (
UnspecifiedState = NodeState(netmap.UnspecifiedState)
Online = NodeState(netmap.Online)
Offline = NodeState(netmap.Offline)
Maintenance = NodeState(netmap.Maintenance)
)
// ToV2 converts NodeState to v2.
func (ns NodeState) ToV2() netmap.NodeState {
return netmap.NodeState(ns)
}
// FromV2 reads NodeState to v2.
func (ns *NodeState) FromV2(state netmap.NodeState) {
*ns = NodeState(state)
}
// Status returns the current state of the node in the network map.
//
// Zero NodeInfo has an undefined state, neither online nor offline.
func (x NodeInfo) Status() NodeState {
return NodeState(x.m.GetState())
}
// SetState updates the state of the node in the network map.
//
// The state determines the node's current status within the network:
// - "online": Indicates the node intends to enter the network.
// - "offline": Indicates the node intends to leave the network.
// - "maintenance": Indicates the node is temporarily unavailable.
//
// See also Status.
func (x *NodeInfo) SetStatus(state NodeState) {
x.m.SetState(netmap.NodeState(state))
}
// String implements fmt.Stringer.
//
// String is designed to be human-readable, and its format MAY differ between
// SDK versions.
func (ns NodeState) String() string {
return netmap.NodeState(ns).String()
}
// IsOnline checks if the current state is "online".
func (ns NodeState) IsOnline() bool { return ns == Online }
// IsOffline checks if the current state is "offline".
func (ns NodeState) IsOffline() bool { return ns == Offline }
// IsMaintenance checks if the current state is "maintenance".
func (ns NodeState) IsMaintenance() bool { return ns == Maintenance }

View file

@ -3,6 +3,7 @@ package netmap
import (
"testing"
"git.frostfs.info/TrueCloudLab/frostfs-api-go/v2/netmap"
"github.com/stretchr/testify/require"
)
@ -23,7 +24,31 @@ func TestNodeInfo_SetAttribute(t *testing.T) {
require.Equal(t, val, n.Attribute(key))
}
func TestNodeState(t *testing.T) {
m := map[NodeState]netmap.NodeState{
UnspecifiedState: netmap.UnspecifiedState,
Online: netmap.Online,
Offline: netmap.Offline,
Maintenance: netmap.Maintenance,
}
t.Run("from sdk to v2", func(t *testing.T) {
for stateSDK, stateV2 := range m {
require.Equal(t, stateV2, stateSDK.ToV2())
}
})
t.Run("from v2 to sdk", func(t *testing.T) {
for stateSDK, stateV2 := range m {
var state NodeState
state.FromV2(stateV2)
require.Equal(t, stateSDK, state)
}
})
}
func TestNodeInfo_Status(t *testing.T) {
t.Run("deprecated getters/setters", func(t *testing.T) {
var n NodeInfo
require.False(t, n.IsOnline())
@ -44,6 +69,30 @@ func TestNodeInfo_Status(t *testing.T) {
require.True(t, n.IsMaintenance())
require.False(t, n.IsOnline())
require.False(t, n.IsOffline())
})
t.Run("brand new getters/setters", func(t *testing.T) {
var n NodeInfo
require.False(t, n.Status().IsOnline())
require.False(t, n.Status().IsOffline())
require.False(t, n.Status().IsMaintenance())
n.SetStatus(Online)
require.True(t, n.Status().IsOnline())
require.False(t, n.Status().IsOffline())
require.False(t, n.Status().IsMaintenance())
n.SetStatus(Offline)
require.False(t, n.Status().IsOnline())
require.True(t, n.Status().IsOffline())
require.False(t, n.Status().IsMaintenance())
n.SetStatus(Maintenance)
require.False(t, n.Status().IsOnline())
require.False(t, n.Status().IsOffline())
require.True(t, n.Status().IsMaintenance())
})
}
func TestNodeInfo_ExternalAddr(t *testing.T) {

View file

@ -20,7 +20,7 @@ cbfStmt: CBF BackupFactor = NUMBER1; // container backup factor
selectStmt:
SELECT Count = NUMBER1 // number of nodes to select without container backup factor *)
(IN clause? Bucket = ident)? // bucket name
(IN clause? Bucket = filterKey)? // bucket name
FROM Filter = identWC // filter reference or whole netmap
(AS Name = ident)? // optional selector name
;

Binary file not shown.

View file

@ -3,7 +3,7 @@ lexer grammar QueryLexer;
NOT_OP : 'NOT';
AND_OP : 'AND';
OR_OP : 'OR';
SIMPLE_OP : 'EQ' | 'NE' | 'GE' | 'GT' | 'LT' | 'LE';
SIMPLE_OP : 'EQ' | 'NE' | 'GE' | 'GT' | 'LT' | 'LE' | 'LIKE' ;
UNIQUE : 'UNIQUE';
REP : 'REP';

Binary file not shown.

View file

@ -1,4 +1,5 @@
package parser
// ANTLR can be downloaded from https://www.antlr.org/download/antlr-4.13.0-complete.jar
//go:generate java -Xmx500M -cp "./antlr-4.13.0-complete.jar:$CLASSPATH" org.antlr.v4.Tool -Dlanguage=Go -no-listener -visitor QueryLexer.g4 Query.g4
// You can download ANTLR from https://www.antlr.org/download/antlr-4.13.1-complete.jar,
// then run generate or simply run the dedicated Makefile target like this `make policy`.
//go:generate java -Xmx500M -cp "./antlr-4.13.1-complete.jar:$CLASSPATH" org.antlr.v4.Tool -Dlanguage=Go -no-listener -visitor QueryLexer.g4 Query.g4

View file

@ -1,4 +1,4 @@
// Code generated from /repo/frostfs/sdk-go/netmap/parser/Query.g4 by ANTLR 4.13.0. DO NOT EDIT.
// Code generated from netmap/parser/Query.g4 by ANTLR 4.13.1. DO NOT EDIT.
package parser // Query

View file

@ -1,4 +1,4 @@
// Code generated from /repo/frostfs/sdk-go/netmap/parser/QueryLexer.g4 by ANTLR 4.13.0. DO NOT EDIT.
// Code generated from netmap/parser/QueryLexer.g4 by ANTLR 4.13.1. DO NOT EDIT.
package parser
@ -62,7 +62,7 @@ func querylexerLexerInit() {
}
staticData.PredictionContextCache = antlr.NewPredictionContextCache()
staticData.serializedATN = []int32{
4, 0, 25, 222, 6, -1, 2, 0, 7, 0, 2, 1, 7, 1, 2, 2, 7, 2, 2, 3, 7, 3, 2,
4, 0, 25, 226, 6, -1, 2, 0, 7, 0, 2, 1, 7, 1, 2, 2, 7, 2, 2, 3, 7, 3, 2,
4, 7, 4, 2, 5, 7, 5, 2, 6, 7, 6, 2, 7, 7, 7, 2, 8, 7, 8, 2, 9, 7, 9, 2,
10, 7, 10, 2, 11, 7, 11, 2, 12, 7, 12, 2, 13, 7, 13, 2, 14, 7, 14, 2, 15,
7, 15, 2, 16, 7, 16, 2, 17, 7, 17, 2, 18, 7, 18, 2, 19, 7, 19, 2, 20, 7,
@ -70,96 +70,98 @@ func querylexerLexerInit() {
2, 26, 7, 26, 2, 27, 7, 27, 2, 28, 7, 28, 2, 29, 7, 29, 2, 30, 7, 30, 2,
31, 7, 31, 1, 0, 1, 0, 1, 0, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 1, 2,
1, 2, 1, 3, 1, 3, 1, 3, 1, 3, 1, 3, 1, 3, 1, 3, 1, 3, 1, 3, 1, 3, 1, 3,
1, 3, 3, 3, 89, 8, 3, 1, 4, 1, 4, 1, 4, 1, 4, 1, 4, 1, 4, 1, 4, 1, 5, 1,
5, 1, 5, 1, 5, 1, 6, 1, 6, 1, 6, 1, 7, 1, 7, 1, 7, 1, 8, 1, 8, 1, 8, 1,
9, 1, 9, 1, 9, 1, 9, 1, 10, 1, 10, 1, 10, 1, 10, 1, 10, 1, 10, 1, 10, 1,
11, 1, 11, 1, 11, 1, 11, 1, 11, 1, 12, 1, 12, 1, 12, 1, 12, 1, 12, 1, 12,
1, 12, 1, 13, 1, 13, 1, 14, 1, 14, 1, 15, 1, 15, 1, 15, 1, 15, 1, 15, 1,
16, 1, 16, 1, 16, 1, 16, 1, 16, 1, 16, 1, 16, 1, 16, 1, 16, 1, 17, 1, 17,
1, 18, 1, 18, 1, 19, 1, 19, 1, 20, 1, 20, 1, 20, 5, 20, 161, 8, 20, 10,
20, 12, 20, 164, 9, 20, 1, 21, 1, 21, 1, 22, 1, 22, 1, 23, 1, 23, 5, 23,
172, 8, 23, 10, 23, 12, 23, 175, 9, 23, 1, 24, 1, 24, 1, 25, 1, 25, 1,
25, 5, 25, 182, 8, 25, 10, 25, 12, 25, 185, 9, 25, 1, 25, 1, 25, 1, 25,
1, 25, 5, 25, 191, 8, 25, 10, 25, 12, 25, 194, 9, 25, 1, 25, 3, 25, 197,
8, 25, 1, 26, 1, 26, 1, 26, 3, 26, 202, 8, 26, 1, 27, 1, 27, 1, 27, 1,
27, 1, 27, 1, 27, 1, 28, 1, 28, 1, 29, 1, 29, 1, 30, 1, 30, 1, 31, 4, 31,
217, 8, 31, 11, 31, 12, 31, 218, 1, 31, 1, 31, 0, 0, 32, 1, 1, 3, 2, 5,
3, 7, 4, 9, 5, 11, 6, 13, 7, 15, 8, 17, 9, 19, 10, 21, 11, 23, 12, 25,
13, 27, 14, 29, 15, 31, 16, 33, 17, 35, 18, 37, 19, 39, 20, 41, 21, 43,
0, 45, 0, 47, 22, 49, 23, 51, 24, 53, 0, 55, 0, 57, 0, 59, 0, 61, 0, 63,
25, 1, 0, 8, 1, 0, 48, 57, 3, 0, 65, 90, 95, 95, 97, 122, 1, 0, 49, 57,
9, 0, 34, 34, 39, 39, 47, 47, 92, 92, 98, 98, 102, 102, 110, 110, 114,
114, 116, 116, 3, 0, 48, 57, 65, 70, 97, 102, 3, 0, 0, 31, 39, 39, 92,
92, 3, 0, 0, 31, 34, 34, 92, 92, 3, 0, 9, 10, 13, 13, 32, 32, 229, 0, 1,
1, 0, 0, 0, 0, 3, 1, 0, 0, 0, 0, 5, 1, 0, 0, 0, 0, 7, 1, 0, 0, 0, 0, 9,
1, 0, 0, 0, 0, 11, 1, 0, 0, 0, 0, 13, 1, 0, 0, 0, 0, 15, 1, 0, 0, 0, 0,
17, 1, 0, 0, 0, 0, 19, 1, 0, 0, 0, 0, 21, 1, 0, 0, 0, 0, 23, 1, 0, 0, 0,
0, 25, 1, 0, 0, 0, 0, 27, 1, 0, 0, 0, 0, 29, 1, 0, 0, 0, 0, 31, 1, 0, 0,
0, 0, 33, 1, 0, 0, 0, 0, 35, 1, 0, 0, 0, 0, 37, 1, 0, 0, 0, 0, 39, 1, 0,
0, 0, 0, 41, 1, 0, 0, 0, 0, 47, 1, 0, 0, 0, 0, 49, 1, 0, 0, 0, 0, 51, 1,
0, 0, 0, 0, 63, 1, 0, 0, 0, 1, 65, 1, 0, 0, 0, 3, 69, 1, 0, 0, 0, 5, 73,
1, 0, 0, 0, 7, 88, 1, 0, 0, 0, 9, 90, 1, 0, 0, 0, 11, 97, 1, 0, 0, 0, 13,
101, 1, 0, 0, 0, 15, 104, 1, 0, 0, 0, 17, 107, 1, 0, 0, 0, 19, 110, 1,
0, 0, 0, 21, 114, 1, 0, 0, 0, 23, 121, 1, 0, 0, 0, 25, 126, 1, 0, 0, 0,
27, 133, 1, 0, 0, 0, 29, 135, 1, 0, 0, 0, 31, 137, 1, 0, 0, 0, 33, 142,
1, 0, 0, 0, 35, 151, 1, 0, 0, 0, 37, 153, 1, 0, 0, 0, 39, 155, 1, 0, 0,
0, 41, 157, 1, 0, 0, 0, 43, 165, 1, 0, 0, 0, 45, 167, 1, 0, 0, 0, 47, 169,
1, 0, 0, 0, 49, 176, 1, 0, 0, 0, 51, 196, 1, 0, 0, 0, 53, 198, 1, 0, 0,
0, 55, 203, 1, 0, 0, 0, 57, 209, 1, 0, 0, 0, 59, 211, 1, 0, 0, 0, 61, 213,
1, 0, 0, 0, 63, 216, 1, 0, 0, 0, 65, 66, 5, 78, 0, 0, 66, 67, 5, 79, 0,
0, 67, 68, 5, 84, 0, 0, 68, 2, 1, 0, 0, 0, 69, 70, 5, 65, 0, 0, 70, 71,
5, 78, 0, 0, 71, 72, 5, 68, 0, 0, 72, 4, 1, 0, 0, 0, 73, 74, 5, 79, 0,
0, 74, 75, 5, 82, 0, 0, 75, 6, 1, 0, 0, 0, 76, 77, 5, 69, 0, 0, 77, 89,
5, 81, 0, 0, 78, 79, 5, 78, 0, 0, 79, 89, 5, 69, 0, 0, 80, 81, 5, 71, 0,
0, 81, 89, 5, 69, 0, 0, 82, 83, 5, 71, 0, 0, 83, 89, 5, 84, 0, 0, 84, 85,
5, 76, 0, 0, 85, 89, 5, 84, 0, 0, 86, 87, 5, 76, 0, 0, 87, 89, 5, 69, 0,
0, 88, 76, 1, 0, 0, 0, 88, 78, 1, 0, 0, 0, 88, 80, 1, 0, 0, 0, 88, 82,
1, 0, 0, 0, 88, 84, 1, 0, 0, 0, 88, 86, 1, 0, 0, 0, 89, 8, 1, 0, 0, 0,
90, 91, 5, 85, 0, 0, 91, 92, 5, 78, 0, 0, 92, 93, 5, 73, 0, 0, 93, 94,
5, 81, 0, 0, 94, 95, 5, 85, 0, 0, 95, 96, 5, 69, 0, 0, 96, 10, 1, 0, 0,
0, 97, 98, 5, 82, 0, 0, 98, 99, 5, 69, 0, 0, 99, 100, 5, 80, 0, 0, 100,
12, 1, 0, 0, 0, 101, 102, 5, 69, 0, 0, 102, 103, 5, 67, 0, 0, 103, 14,
1, 0, 0, 0, 104, 105, 5, 73, 0, 0, 105, 106, 5, 78, 0, 0, 106, 16, 1, 0,
0, 0, 107, 108, 5, 65, 0, 0, 108, 109, 5, 83, 0, 0, 109, 18, 1, 0, 0, 0,
110, 111, 5, 67, 0, 0, 111, 112, 5, 66, 0, 0, 112, 113, 5, 70, 0, 0, 113,
20, 1, 0, 0, 0, 114, 115, 5, 83, 0, 0, 115, 116, 5, 69, 0, 0, 116, 117,
5, 76, 0, 0, 117, 118, 5, 69, 0, 0, 118, 119, 5, 67, 0, 0, 119, 120, 5,
84, 0, 0, 120, 22, 1, 0, 0, 0, 121, 122, 5, 70, 0, 0, 122, 123, 5, 82,
0, 0, 123, 124, 5, 79, 0, 0, 124, 125, 5, 77, 0, 0, 125, 24, 1, 0, 0, 0,
126, 127, 5, 70, 0, 0, 127, 128, 5, 73, 0, 0, 128, 129, 5, 76, 0, 0, 129,
130, 5, 84, 0, 0, 130, 131, 5, 69, 0, 0, 131, 132, 5, 82, 0, 0, 132, 26,
1, 0, 0, 0, 133, 134, 5, 42, 0, 0, 134, 28, 1, 0, 0, 0, 135, 136, 5, 46,
0, 0, 136, 30, 1, 0, 0, 0, 137, 138, 5, 83, 0, 0, 138, 139, 5, 65, 0, 0,
139, 140, 5, 77, 0, 0, 140, 141, 5, 69, 0, 0, 141, 32, 1, 0, 0, 0, 142,
143, 5, 68, 0, 0, 143, 144, 5, 73, 0, 0, 144, 145, 5, 83, 0, 0, 145, 146,
5, 84, 0, 0, 146, 147, 5, 73, 0, 0, 147, 148, 5, 78, 0, 0, 148, 149, 5,
67, 0, 0, 149, 150, 5, 84, 0, 0, 150, 34, 1, 0, 0, 0, 151, 152, 5, 40,
0, 0, 152, 36, 1, 0, 0, 0, 153, 154, 5, 41, 0, 0, 154, 38, 1, 0, 0, 0,
155, 156, 5, 64, 0, 0, 156, 40, 1, 0, 0, 0, 157, 162, 3, 45, 22, 0, 158,
161, 3, 43, 21, 0, 159, 161, 3, 45, 22, 0, 160, 158, 1, 0, 0, 0, 160, 159,
1, 0, 0, 0, 161, 164, 1, 0, 0, 0, 162, 160, 1, 0, 0, 0, 162, 163, 1, 0,
0, 0, 163, 42, 1, 0, 0, 0, 164, 162, 1, 0, 0, 0, 165, 166, 7, 0, 0, 0,
166, 44, 1, 0, 0, 0, 167, 168, 7, 1, 0, 0, 168, 46, 1, 0, 0, 0, 169, 173,
7, 2, 0, 0, 170, 172, 3, 43, 21, 0, 171, 170, 1, 0, 0, 0, 172, 175, 1,
0, 0, 0, 173, 171, 1, 0, 0, 0, 173, 174, 1, 0, 0, 0, 174, 48, 1, 0, 0,
0, 175, 173, 1, 0, 0, 0, 176, 177, 5, 48, 0, 0, 177, 50, 1, 0, 0, 0, 178,
183, 5, 34, 0, 0, 179, 182, 3, 53, 26, 0, 180, 182, 3, 61, 30, 0, 181,
179, 1, 0, 0, 0, 181, 180, 1, 0, 0, 0, 182, 185, 1, 0, 0, 0, 183, 181,
1, 0, 0, 0, 183, 184, 1, 0, 0, 0, 184, 186, 1, 0, 0, 0, 185, 183, 1, 0,
0, 0, 186, 197, 5, 34, 0, 0, 187, 192, 5, 39, 0, 0, 188, 191, 3, 53, 26,
0, 189, 191, 3, 59, 29, 0, 190, 188, 1, 0, 0, 0, 190, 189, 1, 0, 0, 0,
191, 194, 1, 0, 0, 0, 192, 190, 1, 0, 0, 0, 192, 193, 1, 0, 0, 0, 193,
195, 1, 0, 0, 0, 194, 192, 1, 0, 0, 0, 195, 197, 5, 39, 0, 0, 196, 178,
1, 0, 0, 0, 196, 187, 1, 0, 0, 0, 197, 52, 1, 0, 0, 0, 198, 201, 5, 92,
0, 0, 199, 202, 7, 3, 0, 0, 200, 202, 3, 55, 27, 0, 201, 199, 1, 0, 0,
0, 201, 200, 1, 0, 0, 0, 202, 54, 1, 0, 0, 0, 203, 204, 5, 117, 0, 0, 204,
205, 3, 57, 28, 0, 205, 206, 3, 57, 28, 0, 206, 207, 3, 57, 28, 0, 207,
208, 3, 57, 28, 0, 208, 56, 1, 0, 0, 0, 209, 210, 7, 4, 0, 0, 210, 58,
1, 0, 0, 0, 211, 212, 8, 5, 0, 0, 212, 60, 1, 0, 0, 0, 213, 214, 8, 6,
0, 0, 214, 62, 1, 0, 0, 0, 215, 217, 7, 7, 0, 0, 216, 215, 1, 0, 0, 0,
217, 218, 1, 0, 0, 0, 218, 216, 1, 0, 0, 0, 218, 219, 1, 0, 0, 0, 219,
220, 1, 0, 0, 0, 220, 221, 6, 31, 0, 0, 221, 64, 1, 0, 0, 0, 12, 0, 88,
160, 162, 173, 181, 183, 190, 192, 196, 201, 218, 1, 6, 0, 0,
1, 3, 1, 3, 1, 3, 1, 3, 1, 3, 3, 3, 93, 8, 3, 1, 4, 1, 4, 1, 4, 1, 4, 1,
4, 1, 4, 1, 4, 1, 5, 1, 5, 1, 5, 1, 5, 1, 6, 1, 6, 1, 6, 1, 7, 1, 7, 1,
7, 1, 8, 1, 8, 1, 8, 1, 9, 1, 9, 1, 9, 1, 9, 1, 10, 1, 10, 1, 10, 1, 10,
1, 10, 1, 10, 1, 10, 1, 11, 1, 11, 1, 11, 1, 11, 1, 11, 1, 12, 1, 12, 1,
12, 1, 12, 1, 12, 1, 12, 1, 12, 1, 13, 1, 13, 1, 14, 1, 14, 1, 15, 1, 15,
1, 15, 1, 15, 1, 15, 1, 16, 1, 16, 1, 16, 1, 16, 1, 16, 1, 16, 1, 16, 1,
16, 1, 16, 1, 17, 1, 17, 1, 18, 1, 18, 1, 19, 1, 19, 1, 20, 1, 20, 1, 20,
5, 20, 165, 8, 20, 10, 20, 12, 20, 168, 9, 20, 1, 21, 1, 21, 1, 22, 1,
22, 1, 23, 1, 23, 5, 23, 176, 8, 23, 10, 23, 12, 23, 179, 9, 23, 1, 24,
1, 24, 1, 25, 1, 25, 1, 25, 5, 25, 186, 8, 25, 10, 25, 12, 25, 189, 9,
25, 1, 25, 1, 25, 1, 25, 1, 25, 5, 25, 195, 8, 25, 10, 25, 12, 25, 198,
9, 25, 1, 25, 3, 25, 201, 8, 25, 1, 26, 1, 26, 1, 26, 3, 26, 206, 8, 26,
1, 27, 1, 27, 1, 27, 1, 27, 1, 27, 1, 27, 1, 28, 1, 28, 1, 29, 1, 29, 1,
30, 1, 30, 1, 31, 4, 31, 221, 8, 31, 11, 31, 12, 31, 222, 1, 31, 1, 31,
0, 0, 32, 1, 1, 3, 2, 5, 3, 7, 4, 9, 5, 11, 6, 13, 7, 15, 8, 17, 9, 19,
10, 21, 11, 23, 12, 25, 13, 27, 14, 29, 15, 31, 16, 33, 17, 35, 18, 37,
19, 39, 20, 41, 21, 43, 0, 45, 0, 47, 22, 49, 23, 51, 24, 53, 0, 55, 0,
57, 0, 59, 0, 61, 0, 63, 25, 1, 0, 8, 1, 0, 48, 57, 3, 0, 65, 90, 95, 95,
97, 122, 1, 0, 49, 57, 9, 0, 34, 34, 39, 39, 47, 47, 92, 92, 98, 98, 102,
102, 110, 110, 114, 114, 116, 116, 3, 0, 48, 57, 65, 70, 97, 102, 3, 0,
0, 31, 39, 39, 92, 92, 3, 0, 0, 31, 34, 34, 92, 92, 3, 0, 9, 10, 13, 13,
32, 32, 234, 0, 1, 1, 0, 0, 0, 0, 3, 1, 0, 0, 0, 0, 5, 1, 0, 0, 0, 0, 7,
1, 0, 0, 0, 0, 9, 1, 0, 0, 0, 0, 11, 1, 0, 0, 0, 0, 13, 1, 0, 0, 0, 0,
15, 1, 0, 0, 0, 0, 17, 1, 0, 0, 0, 0, 19, 1, 0, 0, 0, 0, 21, 1, 0, 0, 0,
0, 23, 1, 0, 0, 0, 0, 25, 1, 0, 0, 0, 0, 27, 1, 0, 0, 0, 0, 29, 1, 0, 0,
0, 0, 31, 1, 0, 0, 0, 0, 33, 1, 0, 0, 0, 0, 35, 1, 0, 0, 0, 0, 37, 1, 0,
0, 0, 0, 39, 1, 0, 0, 0, 0, 41, 1, 0, 0, 0, 0, 47, 1, 0, 0, 0, 0, 49, 1,
0, 0, 0, 0, 51, 1, 0, 0, 0, 0, 63, 1, 0, 0, 0, 1, 65, 1, 0, 0, 0, 3, 69,
1, 0, 0, 0, 5, 73, 1, 0, 0, 0, 7, 92, 1, 0, 0, 0, 9, 94, 1, 0, 0, 0, 11,
101, 1, 0, 0, 0, 13, 105, 1, 0, 0, 0, 15, 108, 1, 0, 0, 0, 17, 111, 1,
0, 0, 0, 19, 114, 1, 0, 0, 0, 21, 118, 1, 0, 0, 0, 23, 125, 1, 0, 0, 0,
25, 130, 1, 0, 0, 0, 27, 137, 1, 0, 0, 0, 29, 139, 1, 0, 0, 0, 31, 141,
1, 0, 0, 0, 33, 146, 1, 0, 0, 0, 35, 155, 1, 0, 0, 0, 37, 157, 1, 0, 0,
0, 39, 159, 1, 0, 0, 0, 41, 161, 1, 0, 0, 0, 43, 169, 1, 0, 0, 0, 45, 171,
1, 0, 0, 0, 47, 173, 1, 0, 0, 0, 49, 180, 1, 0, 0, 0, 51, 200, 1, 0, 0,
0, 53, 202, 1, 0, 0, 0, 55, 207, 1, 0, 0, 0, 57, 213, 1, 0, 0, 0, 59, 215,
1, 0, 0, 0, 61, 217, 1, 0, 0, 0, 63, 220, 1, 0, 0, 0, 65, 66, 5, 78, 0,
0, 66, 67, 5, 79, 0, 0, 67, 68, 5, 84, 0, 0, 68, 2, 1, 0, 0, 0, 69, 70,
5, 65, 0, 0, 70, 71, 5, 78, 0, 0, 71, 72, 5, 68, 0, 0, 72, 4, 1, 0, 0,
0, 73, 74, 5, 79, 0, 0, 74, 75, 5, 82, 0, 0, 75, 6, 1, 0, 0, 0, 76, 77,
5, 69, 0, 0, 77, 93, 5, 81, 0, 0, 78, 79, 5, 78, 0, 0, 79, 93, 5, 69, 0,
0, 80, 81, 5, 71, 0, 0, 81, 93, 5, 69, 0, 0, 82, 83, 5, 71, 0, 0, 83, 93,
5, 84, 0, 0, 84, 85, 5, 76, 0, 0, 85, 93, 5, 84, 0, 0, 86, 87, 5, 76, 0,
0, 87, 93, 5, 69, 0, 0, 88, 89, 5, 76, 0, 0, 89, 90, 5, 73, 0, 0, 90, 91,
5, 75, 0, 0, 91, 93, 5, 69, 0, 0, 92, 76, 1, 0, 0, 0, 92, 78, 1, 0, 0,
0, 92, 80, 1, 0, 0, 0, 92, 82, 1, 0, 0, 0, 92, 84, 1, 0, 0, 0, 92, 86,
1, 0, 0, 0, 92, 88, 1, 0, 0, 0, 93, 8, 1, 0, 0, 0, 94, 95, 5, 85, 0, 0,
95, 96, 5, 78, 0, 0, 96, 97, 5, 73, 0, 0, 97, 98, 5, 81, 0, 0, 98, 99,
5, 85, 0, 0, 99, 100, 5, 69, 0, 0, 100, 10, 1, 0, 0, 0, 101, 102, 5, 82,
0, 0, 102, 103, 5, 69, 0, 0, 103, 104, 5, 80, 0, 0, 104, 12, 1, 0, 0, 0,
105, 106, 5, 69, 0, 0, 106, 107, 5, 67, 0, 0, 107, 14, 1, 0, 0, 0, 108,
109, 5, 73, 0, 0, 109, 110, 5, 78, 0, 0, 110, 16, 1, 0, 0, 0, 111, 112,
5, 65, 0, 0, 112, 113, 5, 83, 0, 0, 113, 18, 1, 0, 0, 0, 114, 115, 5, 67,
0, 0, 115, 116, 5, 66, 0, 0, 116, 117, 5, 70, 0, 0, 117, 20, 1, 0, 0, 0,
118, 119, 5, 83, 0, 0, 119, 120, 5, 69, 0, 0, 120, 121, 5, 76, 0, 0, 121,
122, 5, 69, 0, 0, 122, 123, 5, 67, 0, 0, 123, 124, 5, 84, 0, 0, 124, 22,
1, 0, 0, 0, 125, 126, 5, 70, 0, 0, 126, 127, 5, 82, 0, 0, 127, 128, 5,
79, 0, 0, 128, 129, 5, 77, 0, 0, 129, 24, 1, 0, 0, 0, 130, 131, 5, 70,
0, 0, 131, 132, 5, 73, 0, 0, 132, 133, 5, 76, 0, 0, 133, 134, 5, 84, 0,
0, 134, 135, 5, 69, 0, 0, 135, 136, 5, 82, 0, 0, 136, 26, 1, 0, 0, 0, 137,
138, 5, 42, 0, 0, 138, 28, 1, 0, 0, 0, 139, 140, 5, 46, 0, 0, 140, 30,
1, 0, 0, 0, 141, 142, 5, 83, 0, 0, 142, 143, 5, 65, 0, 0, 143, 144, 5,
77, 0, 0, 144, 145, 5, 69, 0, 0, 145, 32, 1, 0, 0, 0, 146, 147, 5, 68,
0, 0, 147, 148, 5, 73, 0, 0, 148, 149, 5, 83, 0, 0, 149, 150, 5, 84, 0,
0, 150, 151, 5, 73, 0, 0, 151, 152, 5, 78, 0, 0, 152, 153, 5, 67, 0, 0,
153, 154, 5, 84, 0, 0, 154, 34, 1, 0, 0, 0, 155, 156, 5, 40, 0, 0, 156,
36, 1, 0, 0, 0, 157, 158, 5, 41, 0, 0, 158, 38, 1, 0, 0, 0, 159, 160, 5,
64, 0, 0, 160, 40, 1, 0, 0, 0, 161, 166, 3, 45, 22, 0, 162, 165, 3, 43,
21, 0, 163, 165, 3, 45, 22, 0, 164, 162, 1, 0, 0, 0, 164, 163, 1, 0, 0,
0, 165, 168, 1, 0, 0, 0, 166, 164, 1, 0, 0, 0, 166, 167, 1, 0, 0, 0, 167,
42, 1, 0, 0, 0, 168, 166, 1, 0, 0, 0, 169, 170, 7, 0, 0, 0, 170, 44, 1,
0, 0, 0, 171, 172, 7, 1, 0, 0, 172, 46, 1, 0, 0, 0, 173, 177, 7, 2, 0,
0, 174, 176, 3, 43, 21, 0, 175, 174, 1, 0, 0, 0, 176, 179, 1, 0, 0, 0,
177, 175, 1, 0, 0, 0, 177, 178, 1, 0, 0, 0, 178, 48, 1, 0, 0, 0, 179, 177,
1, 0, 0, 0, 180, 181, 5, 48, 0, 0, 181, 50, 1, 0, 0, 0, 182, 187, 5, 34,
0, 0, 183, 186, 3, 53, 26, 0, 184, 186, 3, 61, 30, 0, 185, 183, 1, 0, 0,
0, 185, 184, 1, 0, 0, 0, 186, 189, 1, 0, 0, 0, 187, 185, 1, 0, 0, 0, 187,
188, 1, 0, 0, 0, 188, 190, 1, 0, 0, 0, 189, 187, 1, 0, 0, 0, 190, 201,
5, 34, 0, 0, 191, 196, 5, 39, 0, 0, 192, 195, 3, 53, 26, 0, 193, 195, 3,
59, 29, 0, 194, 192, 1, 0, 0, 0, 194, 193, 1, 0, 0, 0, 195, 198, 1, 0,
0, 0, 196, 194, 1, 0, 0, 0, 196, 197, 1, 0, 0, 0, 197, 199, 1, 0, 0, 0,
198, 196, 1, 0, 0, 0, 199, 201, 5, 39, 0, 0, 200, 182, 1, 0, 0, 0, 200,
191, 1, 0, 0, 0, 201, 52, 1, 0, 0, 0, 202, 205, 5, 92, 0, 0, 203, 206,
7, 3, 0, 0, 204, 206, 3, 55, 27, 0, 205, 203, 1, 0, 0, 0, 205, 204, 1,
0, 0, 0, 206, 54, 1, 0, 0, 0, 207, 208, 5, 117, 0, 0, 208, 209, 3, 57,
28, 0, 209, 210, 3, 57, 28, 0, 210, 211, 3, 57, 28, 0, 211, 212, 3, 57,
28, 0, 212, 56, 1, 0, 0, 0, 213, 214, 7, 4, 0, 0, 214, 58, 1, 0, 0, 0,
215, 216, 8, 5, 0, 0, 216, 60, 1, 0, 0, 0, 217, 218, 8, 6, 0, 0, 218, 62,
1, 0, 0, 0, 219, 221, 7, 7, 0, 0, 220, 219, 1, 0, 0, 0, 221, 222, 1, 0,
0, 0, 222, 220, 1, 0, 0, 0, 222, 223, 1, 0, 0, 0, 223, 224, 1, 0, 0, 0,
224, 225, 6, 31, 0, 0, 225, 64, 1, 0, 0, 0, 12, 0, 92, 164, 166, 177, 185,
187, 194, 196, 200, 205, 222, 1, 6, 0, 0,
}
deserializer := antlr.NewATNDeserializer(nil)
staticData.atn = deserializer.Deserialize(staticData.serializedATN)

View file

@ -1,4 +1,4 @@
// Code generated from /repo/frostfs/sdk-go/netmap/parser/Query.g4 by ANTLR 4.13.0. DO NOT EDIT.
// Code generated from netmap/parser/Query.g4 by ANTLR 4.13.1. DO NOT EDIT.
package parser // Query
@ -93,7 +93,7 @@ func queryParserInit() {
85, 1, 0, 0, 0, 85, 7, 1, 0, 0, 0, 86, 87, 5, 10, 0, 0, 87, 88, 5, 22,
0, 0, 88, 9, 1, 0, 0, 0, 89, 90, 5, 11, 0, 0, 90, 96, 5, 22, 0, 0, 91,
93, 5, 8, 0, 0, 92, 94, 3, 12, 6, 0, 93, 92, 1, 0, 0, 0, 93, 94, 1, 0,
0, 0, 94, 95, 1, 0, 0, 0, 95, 97, 3, 28, 14, 0, 96, 91, 1, 0, 0, 0, 96,
0, 0, 94, 95, 1, 0, 0, 0, 95, 97, 3, 20, 10, 0, 96, 91, 1, 0, 0, 0, 96,
97, 1, 0, 0, 0, 97, 98, 1, 0, 0, 0, 98, 99, 5, 12, 0, 0, 99, 102, 3, 30,
15, 0, 100, 101, 5, 9, 0, 0, 101, 103, 3, 28, 14, 0, 102, 100, 1, 0, 0,
0, 102, 103, 1, 0, 0, 0, 103, 11, 1, 0, 0, 0, 104, 105, 7, 0, 0, 0, 105,
@ -1364,7 +1364,7 @@ type ISelectStmtContext interface {
SetCount(antlr.Token)
// GetBucket returns the Bucket rule contexts.
GetBucket() IIdentContext
GetBucket() IFilterKeyContext
// GetFilter returns the Filter rule contexts.
GetFilter() IIdentWCContext
@ -1373,7 +1373,7 @@ type ISelectStmtContext interface {
GetName() IIdentContext
// SetBucket sets the Bucket rule contexts.
SetBucket(IIdentContext)
SetBucket(IFilterKeyContext)
// SetFilter sets the Filter rule contexts.
SetFilter(IIdentWCContext)
@ -1388,8 +1388,8 @@ type ISelectStmtContext interface {
IdentWC() IIdentWCContext
IN() antlr.TerminalNode
AS() antlr.TerminalNode
AllIdent() []IIdentContext
Ident(i int) IIdentContext
FilterKey() IFilterKeyContext
Ident() IIdentContext
Clause() IClauseContext
// IsSelectStmtContext differentiates from other interfaces.
@ -1400,7 +1400,7 @@ type SelectStmtContext struct {
antlr.BaseParserRuleContext
parser antlr.Parser
Count antlr.Token
Bucket IIdentContext
Bucket IFilterKeyContext
Filter IIdentWCContext
Name IIdentContext
}
@ -1436,13 +1436,13 @@ func (s *SelectStmtContext) GetCount() antlr.Token { return s.Count }
func (s *SelectStmtContext) SetCount(v antlr.Token) { s.Count = v }
func (s *SelectStmtContext) GetBucket() IIdentContext { return s.Bucket }
func (s *SelectStmtContext) GetBucket() IFilterKeyContext { return s.Bucket }
func (s *SelectStmtContext) GetFilter() IIdentWCContext { return s.Filter }
func (s *SelectStmtContext) GetName() IIdentContext { return s.Name }
func (s *SelectStmtContext) SetBucket(v IIdentContext) { s.Bucket = v }
func (s *SelectStmtContext) SetBucket(v IFilterKeyContext) { s.Bucket = v }
func (s *SelectStmtContext) SetFilter(v IIdentWCContext) { s.Filter = v }
@ -1484,37 +1484,28 @@ func (s *SelectStmtContext) AS() antlr.TerminalNode {
return s.GetToken(QueryAS, 0)
}
func (s *SelectStmtContext) AllIdent() []IIdentContext {
children := s.GetChildren()
len := 0
for _, ctx := range children {
if _, ok := ctx.(IIdentContext); ok {
len++
}
}
tst := make([]IIdentContext, len)
i := 0
for _, ctx := range children {
if t, ok := ctx.(IIdentContext); ok {
tst[i] = t.(IIdentContext)
i++
}
}
return tst
}
func (s *SelectStmtContext) Ident(i int) IIdentContext {
func (s *SelectStmtContext) FilterKey() IFilterKeyContext {
var t antlr.RuleContext
j := 0
for _, ctx := range s.GetChildren() {
if _, ok := ctx.(IIdentContext); ok {
if j == i {
if _, ok := ctx.(IFilterKeyContext); ok {
t = ctx.(antlr.RuleContext)
break
}
j++
}
if t == nil {
return nil
}
return t.(IFilterKeyContext)
}
func (s *SelectStmtContext) Ident() IIdentContext {
var t antlr.RuleContext
for _, ctx := range s.GetChildren() {
if _, ok := ctx.(IIdentContext); ok {
t = ctx.(antlr.RuleContext)
break
}
}
@ -1617,7 +1608,7 @@ func (p *Query) SelectStmt() (localctx ISelectStmtContext) {
{
p.SetState(95)
var _x = p.Ident()
var _x = p.FilterKey()
localctx.(*SelectStmtContext).Bucket = _x
}

View file

@ -1,4 +1,4 @@
// Code generated from /repo/frostfs/sdk-go/netmap/parser/Query.g4 by ANTLR 4.13.0. DO NOT EDIT.
// Code generated from netmap/parser/Query.g4 by ANTLR 4.13.1. DO NOT EDIT.
package parser // Query

View file

@ -324,6 +324,13 @@ func (x *Filter) setAttribute(key string, op netmap.Operation, val string) {
x.m.SetValue(val)
}
// Like applies the rule to accept only nodes with attribute like value.
//
// Method SHOULD NOT be called along with other similar methods.
func (x *Filter) Like(key, value string) {
x.setAttribute(key, netmap.LIKE, value)
}
// Equal applies the rule to accept only nodes with the same attribute value.
//
// Method SHOULD NOT be called along with other similar methods.

View file

@ -82,6 +82,13 @@ CBF 1
SELECT 1 FROM Color
FILTER (Color EQ Red OR Color EQ Blue OR Color EQ Yellow) AND Color NE Green AS Color`,
},
{
name: "non-ascii attributes in SELECT IN",
input: `REP 1
CBF 1
SELECT 1 IN SAME 'Цвет' FROM Colorful
FILTER 'Цвет' EQ 'Красный' OR 'Цвет' EQ 'Синий' AS Colorful`,
},
}
for _, tc := range testCases {
@ -122,6 +129,16 @@ func TestDecodeSelectFilterExpr(t *testing.T) {
FILTER Color EQ 'Red' AS RedNode
FILTER @RedNode AND Shape EQ 'Cirle' AS RedCircleNode
`,
`
CBF 1
SELECT 1 FROM R
FILTER Color LIKE 'R' AS R
`,
`
CBF 1
SELECT 1 IN SAME 'Цвет' FROM Colorful
FILTER 'Цвет' EQ 'Красный' OR 'Цвет' EQ 'Синий' AS Colorful
`,
} {
_, err := DecodeSelectFilterString(s)
require.NoError(t, err)

View file

@ -6,12 +6,15 @@ import (
"encoding/binary"
"fmt"
mrand "math/rand"
"reflect"
"slices"
"strconv"
"strings"
"testing"
"git.frostfs.info/TrueCloudLab/frostfs-api-go/v2/netmap"
"git.frostfs.info/TrueCloudLab/hrw"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
@ -37,7 +40,7 @@ func BenchmarkHRWSort(b *testing.B) {
b.Run("sort by index, no weight", func(b *testing.B) {
realNodes := make([]nodes, netmapSize)
b.ResetTimer()
for i := 0; i < b.N; i++ {
for range b.N {
b.StopTimer()
copy(realNodes, vectors)
b.StartTimer()
@ -48,7 +51,7 @@ func BenchmarkHRWSort(b *testing.B) {
b.Run("sort by value, no weight", func(b *testing.B) {
realNodes := make([]nodes, netmapSize)
b.ResetTimer()
for i := 0; i < b.N; i++ {
for range b.N {
b.StopTimer()
copy(realNodes, vectors)
b.StartTimer()
@ -59,7 +62,7 @@ func BenchmarkHRWSort(b *testing.B) {
b.Run("only sort by index", func(b *testing.B) {
realNodes := make([]nodes, netmapSize)
b.ResetTimer()
for i := 0; i < b.N; i++ {
for range b.N {
b.StopTimer()
copy(realNodes, vectors)
b.StartTimer()
@ -70,7 +73,7 @@ func BenchmarkHRWSort(b *testing.B) {
b.Run("sort by value", func(b *testing.B) {
realNodes := make([]nodes, netmapSize)
b.ResetTimer()
for i := 0; i < b.N; i++ {
for range b.N {
b.StopTimer()
copy(realNodes, vectors)
b.StartTimer()
@ -81,7 +84,7 @@ func BenchmarkHRWSort(b *testing.B) {
b.Run("sort by ID, then by index (deterministic)", func(b *testing.B) {
realNodes := make([]nodes, netmapSize)
b.ResetTimer()
for i := 0; i < b.N; i++ {
for range b.N {
b.StopTimer()
copy(realNodes, vectors)
b.StartTimer()
@ -133,7 +136,7 @@ func BenchmarkPolicyHRWType(b *testing.B) {
nm.SetNodes(nodes)
b.ResetTimer()
for i := 0; i < b.N; i++ {
for range b.N {
_, err := nm.ContainerNodes(p, []byte{1})
if err != nil {
b.Fatal()
@ -194,7 +197,7 @@ func TestPlacementPolicy_DeterministicOrder(t *testing.T) {
}
a, b := getIndices(t)
for i := 0; i < 10; i++ {
for range 10 {
x, y := getIndices(t)
require.Equal(t, a, x)
require.Equal(t, b, y)
@ -251,6 +254,92 @@ func TestPlacementPolicy_ProcessSelectors(t *testing.T) {
}
}
func TestPlacementPolicy_Like(t *testing.T) {
nodes := []NodeInfo{
nodeInfoFromAttributes("Country", "Russia"),
nodeInfoFromAttributes("Country", "Germany"),
nodeInfoFromAttributes("Country", "Belarus"),
}
var nm NetMap
nm.SetNodes(nodes)
t.Run("LIKE all", func(t *testing.T) {
ssNamed := []Selector{newSelector("X", "Country", 4, "FromRU", (*Selector).SelectDistinct)}
fsNamed := []Filter{newFilter("FromRU", "Country", "*", netmap.LIKE)}
rsNamed := []ReplicaDescriptor{newReplica(4, "X")}
pNamed := newPlacementPolicy(0, rsNamed, ssNamed, fsNamed)
n, err := nm.ContainerNodes(pNamed, []byte{1})
require.NoError(t, err)
require.Equal(t, 3, len(n[0]))
for _, n := range n[0] {
require.True(t, strings.Contains("GermanyRussiaBelarus", n.Attribute("Country")))
}
})
t.Run("LIKE no wildcard", func(t *testing.T) {
ssNamed := []Selector{newSelector("X", "Country", 1, "FromRU", (*Selector).SelectDistinct)}
fsNamed := []Filter{newFilter("FromRU", "Country", "Russia", netmap.LIKE)}
rsNamed := []ReplicaDescriptor{newReplica(1, "X")}
pNamed := newPlacementPolicy(0, rsNamed, ssNamed, fsNamed)
n, err := nm.ContainerNodes(pNamed, []byte{1})
require.NoError(t, err)
require.Equal(t, 1, len(n[0]))
for _, n := range n[0] {
require.True(t, n.Attribute("Country") == "Russia")
}
})
t.Run("LIKE prefix", func(t *testing.T) {
ssNamed := []Selector{newSelector("X", "Country", 1, "FromRU", (*Selector).SelectDistinct)}
fsNamed := []Filter{newFilter("FromRU", "Country", "Ge*", netmap.LIKE)}
rsNamed := []ReplicaDescriptor{newReplica(1, "X")}
pNamed := newPlacementPolicy(0, rsNamed, ssNamed, fsNamed)
n, err := nm.ContainerNodes(pNamed, []byte{1})
require.NoError(t, err)
require.Equal(t, 1, len(n[0]))
for _, n := range n[0] {
require.True(t, n.Attribute("Country") == "Germany")
}
})
t.Run("LIKE suffix", func(t *testing.T) {
ssNamed := []Selector{newSelector("X", "Country", 1, "FromRU", (*Selector).SelectDistinct)}
fsNamed := []Filter{newFilter("FromRU", "Country", "*sia", netmap.LIKE)}
rsNamed := []ReplicaDescriptor{newReplica(1, "X")}
pNamed := newPlacementPolicy(0, rsNamed, ssNamed, fsNamed)
n, err := nm.ContainerNodes(pNamed, []byte{1})
require.NoError(t, err)
require.Equal(t, 1, len(n[0]))
for _, n := range n[0] {
require.True(t, n.Attribute("Country") == "Russia")
}
})
t.Run("LIKE middle", func(t *testing.T) {
ssNamed := []Selector{newSelector("X", "Country", 2, "FromRU", (*Selector).SelectDistinct)}
fsNamed := []Filter{newFilter("FromRU", "Country", "*us*", netmap.LIKE)}
rsNamed := []ReplicaDescriptor{newReplica(2, "X")}
pNamed := newPlacementPolicy(0, rsNamed, ssNamed, fsNamed)
n, err := nm.ContainerNodes(pNamed, []byte{1})
require.NoError(t, err)
require.Equal(t, 2, len(n[0]))
for _, n := range n[0] {
require.True(t, strings.Contains("RussiaBelarus", n.Attribute("Country")))
}
})
}
func TestPlacementPolicy_Unique(t *testing.T) {
p := newPlacementPolicy(2,
[]ReplicaDescriptor{
@ -265,7 +354,7 @@ func TestPlacementPolicy_Unique(t *testing.T) {
var nodes []NodeInfo
for i, city := range []string{"Moscow", "Berlin", "Shenzhen"} {
for j := 0; j < 3; j++ {
for j := range 3 {
node := nodeInfoFromAttributes("City", city)
node.SetPublicKey(binary.BigEndian.AppendUint16(nil, uint16(i*4+j)))
nodes = append(nodes, node)
@ -279,7 +368,7 @@ func TestPlacementPolicy_Unique(t *testing.T) {
require.NoError(t, err)
for i, vi := range v {
for _, ni := range vi {
for j := 0; j < i; j++ {
for j := range i {
for _, nj := range v[j] {
require.NotEqual(t, ni.hash, nj.hash)
}
@ -368,7 +457,7 @@ func TestPlacementPolicy_MultiREP(t *testing.T) {
for _, additional := range []int{0, 1, 2} {
t.Run(fmt.Sprintf("unique=%t, additional=%d", unique, additional), func(t *testing.T) {
rs := []ReplicaDescriptor{newReplica(1, "SameRU")}
for i := 0; i < additional; i++ {
for range additional {
rs = append(rs, newReplica(1, ""))
}
@ -455,6 +544,66 @@ func TestPlacementPolicy_ProcessSelectorsExceptForNodes(t *testing.T) {
}
}
func TestPlacementPolicy_NonAsciiAttributes(t *testing.T) {
p := newPlacementPolicy(
1,
[]ReplicaDescriptor{
newReplica(2, "Nodes"),
newReplica(2, "Nodes"),
},
[]Selector{
newSelector("Nodes", "Цвет", 2, "Colorful", (*Selector).SelectSame),
},
[]Filter{
newFilter("Colorful", "", "", netmap.OR,
newFilter("", "Цвет", "Красный", netmap.EQ),
newFilter("", "Цвет", "Синий", netmap.EQ),
),
},
)
p.SetUnique(true)
nodes := []NodeInfo{
nodeInfoFromAttributes("Цвет", "Красный", "Форма", "Треугольник"),
nodeInfoFromAttributes("Цвет", "Красный", "Форма", "Круг"),
nodeInfoFromAttributes("Цвет", "Синий", "Форма", "Треугольник"),
nodeInfoFromAttributes("Цвет", "Синий", "Форма", "Круг"),
nodeInfoFromAttributes("Свойство", "Мягкий", "Форма", "Треугольник"),
nodeInfoFromAttributes("Свойство", "Теплый", "Форма", "Круг"),
}
for i := range nodes {
nodes[i].SetPublicKey([]byte{byte(i)})
}
redNodes := nodes[:2]
blueNodes := nodes[2:4]
var nm NetMap
nm.SetNodes(nodes)
pivot := make([]byte, 42)
_, _ = rand.Read(pivot)
nodesPerReplica, err := nm.ContainerNodes(p, pivot)
require.NoError(t, err)
require.Len(t, nodesPerReplica, 2)
for i := range nodesPerReplica {
slices.SortFunc(nodesPerReplica[i], func(n1, n2 NodeInfo) int {
pk1, pk2 := string(n1.PublicKey()), string(n2.PublicKey())
return cmp.Compare(pk1, pk2)
})
}
redMatchFirst := reflect.DeepEqual(redNodes, nodesPerReplica[0])
blueMatchFirst := reflect.DeepEqual(blueNodes, nodesPerReplica[0])
redMatchSecond := reflect.DeepEqual(redNodes, nodesPerReplica[1])
blueMatchSecond := reflect.DeepEqual(blueNodes, nodesPerReplica[1])
assert.True(t, redMatchFirst && blueMatchSecond || blueMatchFirst && redMatchSecond)
}
func TestSelector_SetName(t *testing.T) {
const name = "some name"
var s Selector

View file

@ -130,7 +130,7 @@ func BenchmarkPlacementPolicyInteropability(b *testing.B) {
b.Run(name, func(b *testing.B) {
b.ReportAllocs()
b.ResetTimer()
for i := 0; i < b.N; i++ {
for range b.N {
b.StartTimer()
v, err := nm.ContainerNodes(tt.Policy, tt.Pivot)
b.StopTimer()
@ -173,7 +173,7 @@ func BenchmarkManySelects(b *testing.B) {
b.ResetTimer()
b.ReportAllocs()
for i := 0; i < b.N; i++ {
for range b.N {
_, err = nm.ContainerNodes(tt.Policy, tt.Pivot)
if err != nil {
b.FailNow()

View file

@ -53,6 +53,7 @@ var stringToOperationMap = map[string]netmap.Operation{
"OR": netmap.OR,
"AND": netmap.AND,
"NOT": netmap.NOT,
"LIKE": netmap.LIKE,
}
func convertStringToOperation(opStr string) netmap.Operation {

View file

@ -53,7 +53,7 @@ func TestErasureCodeReconstruct(t *testing.T) {
})
t.Run("from parity", func(t *testing.T) {
parts := cloneSlice(parts)
for i := 0; i < parityCount; i++ {
for i := range parityCount {
parts[i] = nil
}
reconstructed, err := c.ReconstructHeader(parts)
@ -138,7 +138,7 @@ func TestErasureCodeReconstruct(t *testing.T) {
})
t.Run("from parity", func(t *testing.T) {
parts := cloneSlice(parts)
for i := 0; i < parityCount; i++ {
for i := range parityCount {
parts[i] = nil
}
reconstructed, err := c.Reconstruct(parts)
@ -180,7 +180,7 @@ func TestErasureCodeReconstruct(t *testing.T) {
t.Run("from parity", func(t *testing.T) {
oldParts := parts
parts := cloneSlice(parts)
for i := 0; i < parityCount; i++ {
for i := range parityCount {
parts[i] = nil
}

View file

@ -61,7 +61,7 @@ func TestID_Equal(t *testing.T) {
func TestID_Parse(t *testing.T) {
t.Run("should parse successful", func(t *testing.T) {
for i := 0; i < 10; i++ {
for i := range 10 {
t.Run(strconv.Itoa(i), func(t *testing.T) {
cs := randSHA256Checksum(t)
str := base58.Encode(cs[:])
@ -78,7 +78,7 @@ func TestID_Parse(t *testing.T) {
})
t.Run("should failure on parse", func(t *testing.T) {
for i := 0; i < 10; i++ {
for i := range 10 {
j := i
t.Run(strconv.Itoa(j), func(t *testing.T) {
cs := []byte{1, 2, 3, 4, 5, byte(j)}
@ -98,7 +98,7 @@ func TestID_String(t *testing.T) {
})
t.Run("should be equal", func(t *testing.T) {
for i := 0; i < 10; i++ {
for i := range 10 {
t.Run(strconv.Itoa(i), func(t *testing.T) {
cs := randSHA256Checksum(t)
str := base58.Encode(cs[:])

View file

@ -3,7 +3,10 @@ package object
import (
"errors"
"fmt"
"slices"
"strings"
"git.frostfs.info/TrueCloudLab/frostfs-api-go/v2/container"
"git.frostfs.info/TrueCloudLab/frostfs-api-go/v2/object"
"git.frostfs.info/TrueCloudLab/frostfs-api-go/v2/refs"
v2session "git.frostfs.info/TrueCloudLab/frostfs-api-go/v2/session"
@ -312,6 +315,23 @@ func (o *Object) Attributes() []Attribute {
return res
}
// UserAttributes returns object user attributes.
func (o *Object) UserAttributes() []Attribute {
attrs := (*object.Object)(o).
GetHeader().
GetAttributes()
res := make([]Attribute, 0, len(attrs))
for _, attr := range attrs {
if !strings.HasPrefix(attr.GetKey(), container.SysAttributePrefix) {
res = append(res, *NewAttributeFromV2(&attr))
}
}
return slices.Clip(res)
}
// SetAttributes sets object attributes.
func (o *Object) SetAttributes(v ...Attribute) {
attrs := make([]object.Attribute, len(v))

View file

@ -3,8 +3,10 @@ package object_test
import (
"testing"
v2container "git.frostfs.info/TrueCloudLab/frostfs-api-go/v2/container"
cidtest "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/id/test"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object"
objecttest "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/test"
usertest "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/user/test"
"github.com/stretchr/testify/require"
)
@ -24,3 +26,26 @@ func TestInitCreation(t *testing.T) {
require.Equal(t, cnr, cID)
require.Equal(t, own, o.OwnerID())
}
func Test_Attributes(t *testing.T) {
obj := objecttest.Object()
t.Run("get user attributes", func(t *testing.T) {
// See how we create a test object. It's created with two attributes.
require.Len(t, obj.UserAttributes(), 2)
})
userAttrs := obj.UserAttributes()
sysAttr := *object.NewAttribute()
sysAttr.SetKey(v2container.SysAttributePrefix + "key")
sysAttr.SetValue("value")
attr := append(userAttrs, sysAttr)
obj.SetAttributes(attr...)
t.Run("get attributes", func(t *testing.T) {
require.ElementsMatch(t, obj.UserAttributes(), userAttrs)
require.ElementsMatch(t, obj.Attributes(), attr)
})
}

View file

@ -49,8 +49,8 @@ func TestPatch(t *testing.T) {
var attr1, attr2 Attribute
attr1.SetKey("key1")
attr1.SetValue("val1")
attr1.SetKey("key2")
attr1.SetValue("val2")
attr2.SetKey("key2")
attr2.SetValue("val2")
p.Address = oidtest.Address()
p.NewAttributes = []Attribute{attr1, attr2}

View file

@ -25,12 +25,10 @@ type PatchRes struct {
// PatchApplier is the interface that provides method to apply header and payload patches.
type PatchApplier interface {
// ApplyAttributesPatch applies the patch only for the object's attributes.
// This method must be ALWAYS invoked first as attribute patch must income ONLY with the first patch
// request message.
//
// ApplyAttributesPatch can't be invoked few times, otherwise it returns `ErrAttrPatchAlreadyApplied` error.
//
// ApplyAttributesPatch is idempotent for the original header if it's invoked with empty `newAttrs` and
// The call is idempotent for the original header if it's invoked with empty `newAttrs` and
// `replaceAttrs = false`.
ApplyAttributesPatch(ctx context.Context, newAttrs []objectSDK.Attribute, replaceAttrs bool) error
@ -46,10 +44,10 @@ type PatchApplier interface {
// RangeProvider is the interface that provides a method to get original object payload
// by a given range.
type RangeProvider interface {
// ReadRange reads an original object payload by the given range.
// GetRange reads an original object payload by the given range.
// The method returns io.Reader over the data range only. This means if the data is read out,
// then ReadRange has to be invoked to provide reader over the next range.
ReadRange(ctx context.Context, rng *objectSDK.Range) io.Reader
// then GetRange has to be invoked to provide reader over the next range.
GetRange(ctx context.Context, rng *objectSDK.Range) io.Reader
}
type patcher struct {
@ -124,7 +122,7 @@ func (p *patcher) ApplyAttributesPatch(ctx context.Context, newAttrs []objectSDK
}
if err := p.objectWriter.WriteHeader(ctx, p.hdr); err != nil {
return fmt.Errorf("writer header error: %w", err)
return fmt.Errorf("writer header: %w", err)
}
return nil
}
@ -144,7 +142,7 @@ func (p *patcher) ApplyPayloadPatch(ctx context.Context, payloadPatch *objectSDK
var err error
if p.currOffset, err = p.applyPatch(ctx, payloadPatch, p.currOffset); err != nil {
return fmt.Errorf("apply patch error: %w", err)
return fmt.Errorf("apply patch: %w", err)
}
return nil
@ -157,12 +155,12 @@ func (p *patcher) Close(ctx context.Context) (PatchRes, error) {
// copy remaining originial payload
if err := p.copyRange(ctx, rng); err != nil {
return PatchRes{}, fmt.Errorf("copy payload error: %w", err)
return PatchRes{}, fmt.Errorf("copy payload: %w", err)
}
aid, err := p.objectWriter.Close(ctx)
if err != nil {
return PatchRes{}, fmt.Errorf("close object writer error: %w", err)
return PatchRes{}, fmt.Errorf("close object writer: %w", err)
}
return PatchRes{
@ -171,18 +169,18 @@ func (p *patcher) Close(ctx context.Context) (PatchRes, error) {
}
func (p *patcher) copyRange(ctx context.Context, rng *objectSDK.Range) error {
rdr := p.rangeProvider.ReadRange(ctx, rng)
rdr := p.rangeProvider.GetRange(ctx, rng)
for {
buffOrigPayload := make([]byte, p.readerBuffSize)
n, readErr := rdr.Read(buffOrigPayload)
if readErr != nil {
if readErr != io.EOF {
return fmt.Errorf("read error: %w", readErr)
return fmt.Errorf("read: %w", readErr)
}
}
_, wrErr := p.objectWriter.Write(ctx, buffOrigPayload[:n])
if wrErr != nil {
return fmt.Errorf("write error: %w", wrErr)
return fmt.Errorf("write: %w", wrErr)
}
if readErr == io.EOF {
break
@ -192,6 +190,8 @@ func (p *patcher) copyRange(ctx context.Context, rng *objectSDK.Range) error {
}
func (p *patcher) applyPatch(ctx context.Context, payloadPatch *objectSDK.PayloadPatch, offset uint64) (newOffset uint64, err error) {
newOffset = offset
// write the original payload chunk before the start of the patch
if payloadPatch.Range.GetOffset() > offset {
rng := new(objectSDK.Range)
@ -199,7 +199,7 @@ func (p *patcher) applyPatch(ctx context.Context, payloadPatch *objectSDK.Payloa
rng.SetLength(payloadPatch.Range.GetOffset() - offset)
if err = p.copyRange(ctx, rng); err != nil {
err = fmt.Errorf("copy payload error: %w", err)
err = fmt.Errorf("copy payload: %w", err)
return
}
@ -219,19 +219,19 @@ func (p *patcher) applyPatch(ctx context.Context, payloadPatch *objectSDK.Payloa
}
func mergeAttributes(newAttrs, oldAttrs []objectSDK.Attribute) []objectSDK.Attribute {
attrMap := make(map[string]string)
attrMap := make(map[string]string, len(newAttrs))
for _, attr := range newAttrs {
attrMap[attr.Key()] = attr.Value()
}
for _, oldAttr := range oldAttrs {
newVal, ok := attrMap[oldAttr.Key()]
for i := range oldAttrs {
newVal, ok := attrMap[oldAttrs[i].Key()]
if !ok {
continue
}
oldAttr.SetValue(newVal)
delete(attrMap, oldAttr.Key())
oldAttrs[i].SetValue(newVal)
delete(attrMap, oldAttrs[i].Key())
}
for _, newAttr := range newAttrs {

View file

@ -41,7 +41,7 @@ type mockRangeProvider struct {
var _ RangeProvider = (*mockRangeProvider)(nil)
func (m *mockRangeProvider) ReadRange(_ context.Context, rng *objectSDK.Range) io.Reader {
func (m *mockRangeProvider) GetRange(_ context.Context, rng *objectSDK.Range) io.Reader {
offset := rng.GetOffset()
length := rng.GetLength()
@ -243,7 +243,9 @@ func TestPatch(t *testing.T) {
name string
patches []objectSDK.Patch
originalObjectPayload []byte
patched []byte
patchedPayload []byte
originalHeaderAttributes []objectSDK.Attribute
patchedHeaderAttributes []objectSDK.Attribute
expectedPayloadPatchErr error
}{
{
@ -289,7 +291,11 @@ func TestPatch(t *testing.T) {
},
},
originalObjectPayload: []byte("0123456789qwertyuiopasdfghjklzxcvbnm"),
patched: []byte("0123456789qwertyuiopasdfghjklzxcvbnm"),
patchedPayload: []byte("0123456789qwertyuiopasdfghjklzxcvbnm"),
patchedHeaderAttributes: []objectSDK.Attribute{
newTestAttribute("key1", "val2"),
newTestAttribute("key2", "val2"),
},
},
{
name: "header and payload",
@ -306,7 +312,37 @@ func TestPatch(t *testing.T) {
},
},
originalObjectPayload: []byte("0123456789qwertyuiopasdfghjklzxcvbnm"),
patched: []byte("inserted at the beginning0123456789qwertyuiopasdfghjklzxcvbnm"),
patchedPayload: []byte("inserted at the beginning0123456789qwertyuiopasdfghjklzxcvbnm"),
patchedHeaderAttributes: []objectSDK.Attribute{
newTestAttribute("key1", "val2"),
newTestAttribute("key2", "val2"),
},
},
{
name: "header only merge attributes",
patches: []objectSDK.Patch{
{
NewAttributes: []objectSDK.Attribute{
newTestAttribute("key1", "val2"),
newTestAttribute("key2", "val2-incoming"),
},
PayloadPatch: &objectSDK.PayloadPatch{
Range: rangeWithOffestWithLength(0, 0),
Chunk: []byte("inserted at the beginning"),
},
},
},
originalHeaderAttributes: []objectSDK.Attribute{
newTestAttribute("key2", "to be popped out"),
newTestAttribute("key3", "val3"),
},
originalObjectPayload: []byte("0123456789qwertyuiopasdfghjklzxcvbnm"),
patchedPayload: []byte("inserted at the beginning0123456789qwertyuiopasdfghjklzxcvbnm"),
patchedHeaderAttributes: []objectSDK.Attribute{
newTestAttribute("key2", "val2-incoming"),
newTestAttribute("key3", "val3"),
newTestAttribute("key1", "val2"),
},
},
{
name: "header only then payload",
@ -325,8 +361,11 @@ func TestPatch(t *testing.T) {
},
},
originalObjectPayload: []byte("0123456789qwertyuiopasdfghjklzxcvbnm"),
patched: []byte("inserted at the beginning0123456789qwertyuiopasdfghjklzxcvbnm"),
expectedPayloadPatchErr: ErrPayloadPatchIsNil,
patchedPayload: []byte("inserted at the beginning0123456789qwertyuiopasdfghjklzxcvbnm"),
patchedHeaderAttributes: []objectSDK.Attribute{
newTestAttribute("key1", "val2"),
newTestAttribute("key2", "val2"),
},
},
{
name: "no effect",
@ -351,7 +390,7 @@ func TestPatch(t *testing.T) {
},
},
originalObjectPayload: []byte("0123456789qwertyuiopasdfghjklzxcvbnm"),
patched: []byte("0123456789qwertyuiopasdfghjklzxcvbnm"),
patchedPayload: []byte("0123456789qwertyuiopasdfghjklzxcvbnm"),
},
{
name: "insert prefix",
@ -364,7 +403,7 @@ func TestPatch(t *testing.T) {
},
},
originalObjectPayload: []byte("0123456789qwertyuiopasdfghjklzxcvbnm"),
patched: []byte("inserted at the beginning0123456789qwertyuiopasdfghjklzxcvbnm"),
patchedPayload: []byte("inserted at the beginning0123456789qwertyuiopasdfghjklzxcvbnm"),
},
{
name: "insert in the middle",
@ -377,7 +416,7 @@ func TestPatch(t *testing.T) {
},
},
originalObjectPayload: []byte("0123456789qwertyuiopasdfghjklzxcvbnm"),
patched: []byte("01234inserted somewhere in the middle56789qwertyuiopasdfghjklzxcvbnm"),
patchedPayload: []byte("01234inserted somewhere in the middle56789qwertyuiopasdfghjklzxcvbnm"),
},
{
name: "insert at the end",
@ -390,7 +429,7 @@ func TestPatch(t *testing.T) {
},
},
originalObjectPayload: []byte("0123456789qwertyuiopasdfghjklzxcvbnm"),
patched: []byte("0123456789qwertyuiopasdfghjklzxcvbnminserted somewhere at the end"),
patchedPayload: []byte("0123456789qwertyuiopasdfghjklzxcvbnminserted somewhere at the end"),
},
{
name: "replace by range",
@ -403,7 +442,7 @@ func TestPatch(t *testing.T) {
},
},
originalObjectPayload: []byte("0123456789qwertyuiopasdfghjklzxcvbnm"),
patched: []byte("just replaceertyuiopasdfghjklzxcvbnm"),
patchedPayload: []byte("just replaceertyuiopasdfghjklzxcvbnm"),
},
{
name: "replace and insert some bytes",
@ -416,7 +455,7 @@ func TestPatch(t *testing.T) {
},
},
originalObjectPayload: []byte("0123456789qwertyuiopasdfghjklzxcvbnm"),
patched: []byte("replace and append in the middlewertyuiopasdfghjklzxcvbnm"),
patchedPayload: []byte("replace and append in the middlewertyuiopasdfghjklzxcvbnm"),
},
{
name: "replace and insert some bytes in the middle",
@ -429,7 +468,7 @@ func TestPatch(t *testing.T) {
},
},
originalObjectPayload: []byte("0123456789qwertyuiopasdfghjklzxcvbnm"),
patched: []byte("01234@@@@@89qwertyuiopasdfghjklzxcvbnm"),
patchedPayload: []byte("01234@@@@@89qwertyuiopasdfghjklzxcvbnm"),
},
{
name: "a few patches: prefix, suffix",
@ -448,7 +487,7 @@ func TestPatch(t *testing.T) {
},
},
originalObjectPayload: []byte("0123456789qwertyuiopasdfghjklzxcvbnm"),
patched: []byte("this_will_be_prefix0123456789qwertyuiopasdfghjklzxcvbnmthis_will_be_suffix"),
patchedPayload: []byte("this_will_be_prefix0123456789qwertyuiopasdfghjklzxcvbnmthis_will_be_suffix"),
},
{
name: "a few patches: replace and insert some bytes",
@ -467,7 +506,32 @@ func TestPatch(t *testing.T) {
},
},
originalObjectPayload: []byte("0123456789ABCDEF"),
patched: []byte("0123456789aaaaaDEFbbbbb"),
patchedPayload: []byte("0123456789aaaaaDEFbbbbb"),
},
{
name: "starting from the same offset",
patches: []objectSDK.Patch{
{
PayloadPatch: &objectSDK.PayloadPatch{
Range: rangeWithOffestWithLength(8, 3),
Chunk: []byte("1"),
},
},
{
PayloadPatch: &objectSDK.PayloadPatch{
Range: rangeWithOffestWithLength(11, 0),
Chunk: []byte("2"),
},
},
{
PayloadPatch: &objectSDK.PayloadPatch{
Range: rangeWithOffestWithLength(11, 0),
Chunk: []byte("3"),
},
},
},
originalObjectPayload: []byte("abcdefghijklmnop"),
patchedPayload: []byte("abcdefgh123lmnop"),
},
{
name: "a few patches: various modifiactions",
@ -504,7 +568,7 @@ func TestPatch(t *testing.T) {
},
},
originalObjectPayload: []byte("The ******** mention of Babylon as [insert] appears on a clay ****** from the reign of "),
patched: []byte("The earliest known mention of Babylon as a small town appears on a clay tablet from the reign of Shar-Kali-Sharri"),
patchedPayload: []byte("The earliest known mention of Babylon as a small town appears on a clay tablet from the reign of Shar-Kali-Sharri"),
},
} {
t.Run(test.name, func(t *testing.T) {
@ -515,6 +579,7 @@ func TestPatch(t *testing.T) {
originalObject, _ := newTestObject()
originalObject.SetPayload(test.originalObjectPayload)
originalObject.SetPayloadSize(uint64(len(test.originalObjectPayload)))
originalObject.SetAttributes(test.originalHeaderAttributes...)
patchedObject, _ := newTestObject()
@ -542,22 +607,19 @@ func TestPatch(t *testing.T) {
}
err := patcher.ApplyPayloadPatch(context.Background(), patch.PayloadPatch)
if err != nil {
if test.expectedPayloadPatchErr != nil {
if err != nil && test.expectedPayloadPatchErr != nil {
require.ErrorIs(t, err, test.expectedPayloadPatchErr)
return
} else {
}
require.NoError(t, err)
}
}
}
_, err := patcher.Close(context.Background())
require.NoError(t, err)
require.Equal(t, test.patched, patchedObject.Payload())
require.Equal(t, test.patchedPayload, patchedObject.Payload())
newAttrs := append([]objectSDK.Attribute{}, test.patches[0].NewAttributes...)
require.Equal(t, newAttrs, patchedObject.Attributes())
patchedAttrs := append([]objectSDK.Attribute{}, test.patchedHeaderAttributes...)
require.Equal(t, patchedAttrs, patchedObject.Attributes())
})
}

View file

@ -14,7 +14,7 @@ func generateIDList(sz int) []oid.ID {
res := make([]oid.ID, sz)
cs := [sha256.Size]byte{}
for i := 0; i < sz; i++ {
for i := range sz {
var oID oid.ID
res[i] = oID

View file

@ -72,7 +72,7 @@ func TestTransformer(t *testing.T) {
require.Equal(t, ids.ParentID, &parID)
children := tt.objects[i].Children()
for j := 0; j < i; j++ {
for j := range i {
id, ok := tt.objects[j].ID()
require.True(t, ok)
require.Equal(t, id, children[j])
@ -152,7 +152,7 @@ func benchmarkTransformer(b *testing.B, header *objectSDK.Object, payloadSize, s
b.ReportAllocs()
b.ResetTimer()
for i := 0; i < b.N; i++ {
for range b.N {
f, _ := newPayloadSizeLimiter(maxSize, uint64(sizeHint), func() ObjectWriter { return benchTarget{} })
if err := f.WriteHeader(ctx, header); err != nil {
b.Fatalf("write header: %v", err)

View file

@ -7,8 +7,7 @@ a weighted random selection of the underlying client to make requests.
Create pool instance with 3 nodes connection.
This InitParameters will make pool use 192.168.130.71 node while it is healthy. Otherwise, it will make the pool use
192.168.130.72 for 90% of requests and 192.168.130.73 for remaining 10%.
:
192.168.130.72 for 90% of requests and 192.168.130.73 for remaining 10%:
var prm pool.InitParameters
prm.SetKey(key)

View file

@ -14,7 +14,6 @@ import (
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container"
cid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/id"
frostfsecdsa "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/crypto/ecdsa"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/eacl"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/netmap"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/session"
@ -95,14 +94,6 @@ func (m *mockClient) containerDelete(context.Context, PrmContainerDelete) error
return nil
}
func (m *mockClient) containerEACL(context.Context, PrmContainerEACL) (eacl.Table, error) {
return eacl.Table{}, nil
}
func (m *mockClient) containerSetEACL(context.Context, PrmContainerSetEACL) error {
return nil
}
func (c *mockClient) apeManagerAddChain(ctx context.Context, prm PrmAddAPEChain) error {
return nil
}
@ -145,6 +136,10 @@ func (m *mockClient) objectPut(context.Context, PrmObjectPut) (ResPutObject, err
return ResPutObject{}, nil
}
func (m *mockClient) objectPatch(context.Context, PrmObjectPatch) (ResPatchObject, error) {
return ResPatchObject{}, nil
}
func (m *mockClient) objectDelete(context.Context, PrmObjectDelete) error {
return nil
}

View file

@ -22,7 +22,6 @@ import (
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container"
cid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/id"
frostfsecdsa "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/crypto/ecdsa"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/eacl"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/netmap"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object"
oid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id"
@ -50,8 +49,6 @@ type client interface {
containerList(context.Context, PrmContainerList) ([]cid.ID, error)
// see clientWrapper.containerDelete.
containerDelete(context.Context, PrmContainerDelete) error
// see clientWrapper.containerEACL.
containerEACL(context.Context, PrmContainerEACL) (eacl.Table, error)
// see clientWrapper.apeManagerAddChain.
apeManagerAddChain(context.Context, PrmAddAPEChain) error
// see clientWrapper.apeManagerRemoveChain.
@ -66,6 +63,8 @@ type client interface {
netMapSnapshot(context.Context, prmNetMapSnapshot) (netmap.NetMap, error)
// see clientWrapper.objectPut.
objectPut(context.Context, PrmObjectPut) (ResPutObject, error)
// see clientWrapper.objectPatch.
objectPatch(context.Context, PrmObjectPatch) (ResPatchObject, error)
// see clientWrapper.objectDelete.
objectDelete(context.Context, PrmObjectDelete) error
// see clientWrapper.objectGet.
@ -106,7 +105,7 @@ type clientStatus interface {
// overallErrorRate returns the number of all happened errors.
overallErrorRate() uint64
// methodsStatus returns statistic for all used methods.
methodsStatus() []statusSnapshot
methodsStatus() []StatusSnapshot
}
// errPoolClientUnhealthy is an error to indicate that client in pool is unhealthy.
@ -122,7 +121,7 @@ type clientStatusMonitor struct {
mu sync.RWMutex // protect counters
currentErrorCount uint32
overallErrorCount uint64
methods []*methodStatus
methods []*MethodStatus
}
// values for healthy status of clientStatusMonitor.
@ -141,19 +140,6 @@ const (
statusHealthy
)
// methodStatus provide statistic for specific method.
type methodStatus struct {
name string
mu sync.RWMutex // protect counters
statusSnapshot
}
// statusSnapshot is statistic for specific method.
type statusSnapshot struct {
allTime uint64
allRequests uint64
}
// MethodIndex index of method in list of statuses in clientStatusMonitor.
type MethodIndex int
@ -163,8 +149,6 @@ const (
methodContainerGet
methodContainerList
methodContainerDelete
methodContainerEACL
methodContainerSetEACL
methodEndpointInfo
methodNetworkInfo
methodNetMapSnapshot
@ -173,6 +157,7 @@ const (
methodObjectGet
methodObjectHead
methodObjectRange
methodObjectPatch
methodSessionCreate
methodAPEManagerAddChain
methodAPEManagerRemoveChain
@ -193,10 +178,6 @@ func (m MethodIndex) String() string {
return "containerList"
case methodContainerDelete:
return "containerDelete"
case methodContainerEACL:
return "containerEACL"
case methodContainerSetEACL:
return "containerSetEACL"
case methodEndpointInfo:
return "endpointInfo"
case methodNetworkInfo:
@ -205,6 +186,8 @@ func (m MethodIndex) String() string {
return "netMapSnapshot"
case methodObjectPut:
return "objectPut"
case methodObjectPatch:
return "objectPatch"
case methodObjectDelete:
return "objectDelete"
case methodObjectGet:
@ -229,9 +212,9 @@ func (m MethodIndex) String() string {
}
func newClientStatusMonitor(logger *zap.Logger, addr string, errorThreshold uint32) clientStatusMonitor {
methods := make([]*methodStatus, methodLast)
methods := make([]*MethodStatus, methodLast)
for i := methodBalanceGet; i < methodLast; i++ {
methods[i] = &methodStatus{name: i.String()}
methods[i] = &MethodStatus{name: i.String()}
}
healthy := new(atomic.Uint32)
@ -246,19 +229,6 @@ func newClientStatusMonitor(logger *zap.Logger, addr string, errorThreshold uint
}
}
func (m *methodStatus) snapshot() statusSnapshot {
m.mu.RLock()
defer m.mu.RUnlock()
return m.statusSnapshot
}
func (m *methodStatus) incRequests(elapsed time.Duration) {
m.mu.Lock()
defer m.mu.Unlock()
m.allTime += uint64(elapsed)
m.allRequests++
}
// clientWrapper is used by default, alternative implementations are intended for testing purposes only.
type clientWrapper struct {
clientMutex sync.RWMutex
@ -279,6 +249,8 @@ type wrapperPrm struct {
responseInfoCallback func(sdkClient.ResponseMetaInfo) error
poolRequestInfoCallback func(RequestInfo)
dialOptions []grpc.DialOption
gracefulCloseOnSwitchTimeout time.Duration
}
// setAddress sets endpoint to connect in FrostFS network.
@ -312,6 +284,14 @@ func (x *wrapperPrm) setErrorThreshold(threshold uint32) {
x.errorThreshold = threshold
}
// setGracefulCloseOnSwitchTimeout specifies the timeout after which unhealthy client be closed during rebalancing
// if it will become healthy back.
//
// See also setErrorThreshold.
func (x *wrapperPrm) setGracefulCloseOnSwitchTimeout(timeout time.Duration) {
x.gracefulCloseOnSwitchTimeout = timeout
}
// setPoolRequestCallback sets callback that will be invoked after every pool response.
func (x *wrapperPrm) setPoolRequestCallback(f func(RequestInfo)) {
x.poolRequestInfoCallback = f
@ -383,7 +363,7 @@ func (c *clientWrapper) restartIfUnhealthy(ctx context.Context) (changed bool, e
// if connection is dialed before, to avoid routine / connection leak,
// pool has to close it and then initialize once again.
if c.isDialed() {
_ = c.close()
c.scheduleGracefulClose()
}
var cl sdkClient.Client
@ -428,6 +408,12 @@ func (c *clientWrapper) getClient() (*sdkClient.Client, error) {
return nil, errPoolClientUnhealthy
}
func (c *clientWrapper) getClientRaw() *sdkClient.Client {
c.clientMutex.RLock()
defer c.clientMutex.RUnlock()
return c.client
}
// balanceGet invokes sdkClient.BalanceGet parse response status to error and return result as is.
func (c *clientWrapper) balanceGet(ctx context.Context, prm PrmBalanceGet) (accounting.Decimal, error) {
cl, err := c.getClient()
@ -584,32 +570,6 @@ func (c *clientWrapper) containerDelete(ctx context.Context, prm PrmContainerDel
return waitForContainerRemoved(ctx, c, getPrm, prm.WaitParams)
}
// containerEACL invokes sdkClient.ContainerEACL parse response status to error and return result as is.
func (c *clientWrapper) containerEACL(ctx context.Context, prm PrmContainerEACL) (eacl.Table, error) {
cl, err := c.getClient()
if err != nil {
return eacl.Table{}, err
}
cliPrm := sdkClient.PrmContainerEACL{
ContainerID: &prm.ContainerID,
Session: prm.Session,
}
start := time.Now()
res, err := cl.ContainerEACL(ctx, cliPrm)
c.incRequests(time.Since(start), methodContainerEACL)
var st apistatus.Status
if res != nil {
st = res.Status()
}
if err = c.handleError(ctx, st, err); err != nil {
return eacl.Table{}, fmt.Errorf("get eacl on client: %w", err)
}
return res.Table(), nil
}
// apeManagerAddChain invokes sdkClient.APEManagerAddChain and parse response status to error.
func (c *clientWrapper) apeManagerAddChain(ctx context.Context, prm PrmAddAPEChain) error {
cl, err := c.getClient()
@ -750,6 +710,48 @@ func (c *clientWrapper) netMapSnapshot(ctx context.Context, _ prmNetMapSnapshot)
return res.NetMap(), nil
}
// objectPatch patches object in FrostFS.
func (c *clientWrapper) objectPatch(ctx context.Context, prm PrmObjectPatch) (ResPatchObject, error) {
cl, err := c.getClient()
if err != nil {
return ResPatchObject{}, err
}
start := time.Now()
pObj, err := cl.ObjectPatchInit(ctx, sdkClient.PrmObjectPatch{
Address: prm.addr,
Session: prm.stoken,
Key: prm.key,
BearerToken: prm.btoken,
MaxChunkLength: prm.maxPayloadPatchChunkLength,
})
if err = c.handleError(ctx, nil, err); err != nil {
return ResPatchObject{}, fmt.Errorf("init patching on API client: %w", err)
}
c.incRequests(time.Since(start), methodObjectPatch)
start = time.Now()
attrPatchSuccess := pObj.PatchAttributes(ctx, prm.newAttrs, prm.replaceAttrs)
c.incRequests(time.Since(start), methodObjectPatch)
if attrPatchSuccess {
start = time.Now()
_ = pObj.PatchPayload(ctx, prm.rng, prm.payload)
c.incRequests(time.Since(start), methodObjectPatch)
}
res, err := pObj.Close(ctx)
var st apistatus.Status
if res != nil {
st = res.Status()
}
if err = c.handleError(ctx, st, err); err != nil {
return ResPatchObject{}, fmt.Errorf("client failure: %w", err)
}
return ResPatchObject{ObjectID: res.ObjectID()}, nil
}
// objectPut writes object to FrostFS.
func (c *clientWrapper) objectPut(ctx context.Context, prm PrmObjectPut) (ResPutObject, error) {
if prm.bufferMaxSize == 0 {
@ -1177,10 +1179,10 @@ func (c *clientStatusMonitor) overallErrorRate() uint64 {
return c.overallErrorCount
}
func (c *clientStatusMonitor) methodsStatus() []statusSnapshot {
result := make([]statusSnapshot, len(c.methods))
func (c *clientStatusMonitor) methodsStatus() []StatusSnapshot {
result := make([]StatusSnapshot, len(c.methods))
for i, val := range c.methods {
result[i] = val.snapshot()
result[i] = val.Snapshot()
}
return result
@ -1188,7 +1190,7 @@ func (c *clientStatusMonitor) methodsStatus() []statusSnapshot {
func (c *clientWrapper) incRequests(elapsed time.Duration, method MethodIndex) {
methodStat := c.methods[method]
methodStat.incRequests(elapsed)
methodStat.IncRequests(elapsed)
if c.prm.poolRequestInfoCallback != nil {
c.prm.poolRequestInfoCallback(RequestInfo{
Address: c.prm.address,
@ -1199,13 +1201,42 @@ func (c *clientWrapper) incRequests(elapsed time.Duration, method MethodIndex) {
}
func (c *clientWrapper) close() error {
if c.client != nil {
return c.client.Close()
if cl := c.getClientRaw(); cl != nil {
return cl.Close()
}
return nil
}
func (c *clientWrapper) scheduleGracefulClose() {
cl := c.getClientRaw()
if cl == nil {
return
}
time.AfterFunc(c.prm.gracefulCloseOnSwitchTimeout, func() {
if err := cl.Close(); err != nil {
c.log(zap.DebugLevel, "close unhealthy client during rebalance", zap.String("address", c.address()), zap.Error(err))
}
})
}
func (c *clientStatusMonitor) handleError(ctx context.Context, st apistatus.Status, err error) error {
if stErr := apistatus.ErrFromStatus(st); stErr != nil {
switch stErr.(type) {
case *apistatus.ServerInternal,
*apistatus.WrongMagicNumber,
*apistatus.SignatureVerification,
*apistatus.NodeUnderMaintenance:
c.incErrorRate()
}
if err == nil {
err = stErr
}
return err
}
if err != nil {
if needCountError(ctx, err) {
c.incErrorRate()
@ -1214,16 +1245,7 @@ func (c *clientStatusMonitor) handleError(ctx context.Context, st apistatus.Stat
return err
}
err = apistatus.ErrFromStatus(st)
switch err.(type) {
case *apistatus.ServerInternal,
*apistatus.WrongMagicNumber,
*apistatus.SignatureVerification,
*apistatus.NodeUnderMaintenance:
c.incErrorRate()
}
return err
return nil
}
func needCountError(ctx context.Context, err error) bool {
@ -1272,6 +1294,8 @@ type InitParameters struct {
dialOptions []grpc.DialOption
clientBuilder clientBuilder
gracefulCloseOnSwitchTimeout time.Duration
}
// SetKey specifies default key to be used for the protocol communication by default.
@ -1308,6 +1332,15 @@ func (x *InitParameters) SetClientRebalanceInterval(interval time.Duration) {
x.clientRebalanceInterval = interval
}
// SetGracefulCloseOnSwitchTimeout specifies the timeout after which unhealthy client be closed during rebalancing
// if it will become healthy back.
// Generally this param should be less than client rebalance interval (see SetClientRebalanceInterval).
//
// See also SetErrorThreshold.
func (x *InitParameters) SetGracefulCloseOnSwitchTimeout(timeout time.Duration) {
x.gracefulCloseOnSwitchTimeout = timeout
}
// SetSessionExpirationDuration specifies the session token lifetime in epochs.
func (x *InitParameters) SetSessionExpirationDuration(expirationDuration uint64) {
x.sessionExpirationDuration = expirationDuration
@ -1382,6 +1415,9 @@ func (x *NodeParam) SetPriority(priority int) {
}
// Priority returns priority of the node.
// Requests will be served by nodes subset with the highest priority (the smaller value - the higher priority).
// If there are no healthy nodes in subsets with current or higher priority, requests will be served
// by nodes subset with lower priority.
func (x *NodeParam) Priority() int {
return x.priority
}
@ -1397,6 +1433,7 @@ func (x *NodeParam) Address() string {
}
// SetWeight specifies weight of the node.
// Weights used to adjust requests' distribution between nodes with the same priority.
func (x *NodeParam) SetWeight(weight float64) {
x.weight = weight
}
@ -1440,7 +1477,7 @@ func (x *WaitParams) checkForPositive() {
}
}
// CheckForValid checks if all wait params are non-negative.
// CheckValidity checks if all wait params are non-negative.
func (x *WaitParams) CheckValidity() error {
if x.Timeout <= 0 {
return errors.New("timeout cannot be negative")
@ -1571,6 +1608,53 @@ func (x *PrmObjectPut) setNetworkInfo(ni netmap.NetworkInfo) {
x.networkInfo = ni
}
// PrmObjectPatch groups parameters of PatchObject operation.
type PrmObjectPatch struct {
prmCommon
addr oid.Address
rng *object.Range
payload io.Reader
newAttrs []object.Attribute
replaceAttrs bool
maxPayloadPatchChunkLength int
}
// SetAddress sets the address of the object that is patched.
func (x *PrmObjectPatch) SetAddress(addr oid.Address) {
x.addr = addr
}
// SetRange sets the patch's range.
func (x *PrmObjectPatch) SetRange(rng *object.Range) {
x.rng = rng
}
// SetPayloadReader sets a payload reader.
func (x *PrmObjectPatch) SetPayloadReader(payload io.Reader) {
x.payload = payload
}
// SetRange sets the new attributes to the patch.
func (x *PrmObjectPatch) SetNewAttributes(newAttrs []object.Attribute) {
x.newAttrs = newAttrs
}
// SetRange sets the replace attributes flag to the patch.
func (x *PrmObjectPatch) SetReplaceAttributes(replaceAttrs bool) {
x.replaceAttrs = replaceAttrs
}
// SetMaxPayloadPatchChunkSize sets a max buf size to read the patch's payload.
func (x *PrmObjectPatch) SetMaxPayloadPatchChunkSize(maxPayloadPatchChunkSize int) {
x.maxPayloadPatchChunkLength = maxPayloadPatchChunkSize
}
// PrmObjectDelete groups parameters of DeleteObject operation.
type PrmObjectDelete struct {
prmCommon
@ -1752,29 +1836,6 @@ func (x *PrmContainerDelete) SetWaitParams(waitParams WaitParams) {
x.WaitParams = &waitParams
}
// PrmContainerEACL groups parameters of GetEACL operation.
type PrmContainerEACL struct {
ContainerID cid.ID
Session *session.Container
}
// SetContainerID specifies identifier of the FrostFS container to read the eACL table.
//
// Deprecated: Use PrmContainerEACL.ContainerID instead.
func (x *PrmContainerEACL) SetContainerID(cnrID cid.ID) {
x.ContainerID = cnrID
}
// PrmContainerSetEACL groups parameters of SetEACL operation.
type PrmContainerSetEACL struct {
Table eacl.Table
Session *session.Container
WaitParams *WaitParams
}
type PrmAddAPEChain struct {
Target ape.ChainTarget
@ -1791,36 +1852,6 @@ type PrmListAPEChains struct {
Target ape.ChainTarget
}
// SetTable sets structure of container's extended ACL to be used as a
// parameter of the base client's operation.
//
// See git.frostfs.info/TrueCloudLab/frostfs-sdk-go/client.PrmContainerSetEACL.SetTable.
//
// Deprecated: Use PrmContainerSetEACL.Table instead.
func (x *PrmContainerSetEACL) SetTable(table eacl.Table) {
x.Table = table
}
// WithinSession specifies session to be used as a parameter of the base
// client's operation.
//
// See git.frostfs.info/TrueCloudLab/frostfs-sdk-go/client.PrmContainerSetEACL.WithinSession.
//
// Deprecated: Use PrmContainerSetEACL.Session instead.
func (x *PrmContainerSetEACL) WithinSession(s session.Container) {
x.Session = &s
}
// SetWaitParams specifies timeout params to complete operation.
// If not provided the default one will be used.
// Panics if any of the wait params isn't positive.
//
// Deprecated: Use PrmContainerSetEACL.WaitParams instead.
func (x *PrmContainerSetEACL) SetWaitParams(waitParams WaitParams) {
waitParams.checkForPositive()
x.WaitParams = &waitParams
}
// PrmBalanceGet groups parameters of Balance operation.
type PrmBalanceGet struct {
account user.ID
@ -1909,6 +1940,7 @@ const (
defaultSessionTokenExpirationDuration = 100 // in epochs
defaultErrorThreshold = 100
defaultGracefulCloseOnSwitchTimeout = 10 * time.Second
defaultRebalanceInterval = 15 * time.Second
defaultHealthcheckTimeout = 4 * time.Second
defaultDialTimeout = 5 * time.Second
@ -2034,6 +2066,10 @@ func fillDefaultInitParams(params *InitParameters, cache *sessionCache) {
params.clientRebalanceInterval = defaultRebalanceInterval
}
if params.gracefulCloseOnSwitchTimeout <= 0 {
params.gracefulCloseOnSwitchTimeout = defaultGracefulCloseOnSwitchTimeout
}
if params.healthcheckTimeout <= 0 {
params.healthcheckTimeout = defaultHealthcheckTimeout
}
@ -2059,6 +2095,7 @@ func fillDefaultInitParams(params *InitParameters, cache *sessionCache) {
prm.setDialTimeout(params.nodeDialTimeout)
prm.setStreamTimeout(params.nodeStreamTimeout)
prm.setErrorThreshold(params.errorThreshold)
prm.setGracefulCloseOnSwitchTimeout(params.gracefulCloseOnSwitchTimeout)
prm.setPoolRequestCallback(params.requestCallback)
prm.setGRPCDialOptions(params.dialOptions)
prm.setResponseInfoCallback(func(info sdkClient.ResponseMetaInfo) error {
@ -2219,7 +2256,7 @@ func (p *innerPool) connection() (client, error) {
return nil, errors.New("no healthy client")
}
attempts := 3 * len(p.clients)
for k := 0; k < attempts; k++ {
for range attempts {
i := p.sampler.Next()
if cp := p.clients[i]; cp.isHealthy() {
return cp, nil
@ -2415,6 +2452,44 @@ type ResPutObject struct {
Epoch uint64
}
// ResPatchObject is designed to provide identifier for the saved patched object.
type ResPatchObject struct {
ObjectID oid.ID
}
// PatchObject patches an object through a remote server using FrostFS API protocol.
//
// Main return value MUST NOT be processed on an erroneous return.
func (p *Pool) PatchObject(ctx context.Context, prm PrmObjectPatch) (ResPatchObject, error) {
var prmCtx prmContext
prmCtx.useDefaultSession()
prmCtx.useVerb(session.VerbObjectPatch)
prmCtx.useContainer(prm.addr.Container())
p.fillAppropriateKey(&prm.prmCommon)
var ctxCall callContext
if err := p.initCallContext(&ctxCall, prm.prmCommon, prmCtx); err != nil {
return ResPatchObject{}, fmt.Errorf("init call context: %w", err)
}
if ctxCall.sessionDefault {
ctxCall.sessionTarget = prm.UseSession
if err := p.openDefaultSession(ctx, &ctxCall); err != nil {
return ResPatchObject{}, fmt.Errorf("open default session: %w", err)
}
}
res, err := ctxCall.client.objectPatch(ctx, prm)
if err != nil {
// removes session token from cache in case of token error
p.checkSessionTokenErr(err, ctxCall.endpoint)
return ResPatchObject{}, fmt.Errorf("init patching on API client %s: %w", ctxCall.endpoint, err)
}
return res, nil
}
// PutObject writes an object through a remote server using FrostFS API protocol.
//
// Main return value MUST NOT be processed on an erroneous return.
@ -2771,23 +2846,6 @@ func (p *Pool) DeleteContainer(ctx context.Context, prm PrmContainerDelete) erro
return nil
}
// GetEACL reads eACL table of the FrostFS container.
//
// Main return value MUST NOT be processed on an erroneous return.
func (p *Pool) GetEACL(ctx context.Context, prm PrmContainerEACL) (eacl.Table, error) {
cp, err := p.connection()
if err != nil {
return eacl.Table{}, err
}
eaclResult, err := cp.containerEACL(ctx, prm)
if err != nil {
return eacl.Table{}, fmt.Errorf("get EACL via client '%s': %w", cp.address(), err)
}
return eaclResult, nil
}
// AddAPEChain sends a request to set APE chain rules for a target (basically, for a container).
func (p *Pool) AddAPEChain(ctx context.Context, prm PrmAddAPEChain) error {
cp, err := p.connection()

View file

@ -222,7 +222,7 @@ func TestOneOfTwoFailed(t *testing.T) {
time.Sleep(2 * time.Second)
for i := 0; i < 5; i++ {
for range 5 {
cp, err := pool.connection()
require.NoError(t, err)
st, _ := pool.cache.Get(formCacheKey(cp.address(), pool.key, false))
@ -514,7 +514,7 @@ func TestStatusMonitor(t *testing.T) {
monitor.errorThreshold = 3
count := 10
for i := 0; i < count; i++ {
for range count {
monitor.incErrorRate()
}
@ -609,6 +609,23 @@ func TestHandleError(t *testing.T) {
expectedError: true,
countError: false,
},
{
ctx: ctx,
status: new(apistatus.ObjectNotFound),
err: &apistatus.ObjectNotFound{},
expectedError: true,
countError: false,
},
{
ctx: ctx,
status: nil,
err: &apistatus.EACLNotFound{},
expectedError: true,
// we expect error be counted because status is nil
// currently we assume that DisableFrostFSErrorResolution be always false for pool
// and status be checked first in handleError
countError: true,
},
{
ctx: ctx,
status: new(apistatus.ServerInternal),
@ -707,7 +724,7 @@ func TestSwitchAfterErrorThreshold(t *testing.T) {
require.NoError(t, err)
t.Cleanup(pool.Close)
for i := 0; i < errorThreshold; i++ {
for range errorThreshold {
conn, err := pool.connection()
require.NoError(t, err)
require.Equal(t, nodes[0].address, conn.address())

View file

@ -30,7 +30,7 @@ func newSampler(probabilities []float64, source rand.Source) *sampler {
sampler.alias = make([]int, n)
// Compute scaled probabilities.
p := make([]float64, n)
for i := 0; i < n; i++ {
for i := range n {
p[i] = probabilities[i] * float64(n)
}
for i, pi := range p {

View file

@ -32,7 +32,7 @@ func TestSamplerStability(t *testing.T) {
for _, tc := range cases {
sampler := newSampler(tc.probabilities, rand.NewSource(0))
res := make([]int, len(tc.probabilities))
for i := 0; i < COUNT; i++ {
for range COUNT {
res[sampler.Next()]++
}

View file

@ -2,6 +2,7 @@ package pool
import (
"errors"
"sync"
"time"
)
@ -46,7 +47,7 @@ func (s Statistic) Node(address string) (*NodeStatistic, error) {
// NodeStatistic is metrics of certain connections.
type NodeStatistic struct {
address string
methods []statusSnapshot
methods []StatusSnapshot
overallErrors uint64
currentErrors uint32
}
@ -101,16 +102,6 @@ func (n NodeStatistic) AverageDeleteContainer() time.Duration {
return n.averageTime(methodContainerDelete)
}
// AverageGetContainerEACL returns average time to perform ContainerEACL request.
func (n NodeStatistic) AverageGetContainerEACL() time.Duration {
return n.averageTime(methodContainerEACL)
}
// AverageSetContainerEACL returns average time to perform ContainerSetEACL request.
func (n NodeStatistic) AverageSetContainerEACL() time.Duration {
return n.averageTime(methodContainerSetEACL)
}
// AverageEndpointInfo returns average time to perform EndpointInfo request.
func (n NodeStatistic) AverageEndpointInfo() time.Duration {
return n.averageTime(methodEndpointInfo)
@ -158,3 +149,48 @@ func (n NodeStatistic) averageTime(method MethodIndex) time.Duration {
}
return time.Duration(stat.allTime / stat.allRequests)
}
// MethodStatus provide statistic for specific method.
type MethodStatus struct {
name string
mu sync.RWMutex // protect counters
snapshot StatusSnapshot
}
func NewMethodStatus(name string) *MethodStatus {
return &MethodStatus{name: name}
}
func (m *MethodStatus) Snapshot() StatusSnapshot {
m.mu.RLock()
defer m.mu.RUnlock()
return m.snapshot
}
func (m *MethodStatus) IncRequests(elapsed time.Duration) {
m.mu.Lock()
defer m.mu.Unlock()
m.snapshot.allTime += uint64(elapsed)
m.snapshot.allRequests++
}
func (m *MethodStatus) Reset() {
m.mu.Lock()
defer m.mu.Unlock()
m.snapshot.allTime = 0
m.snapshot.allRequests = 0
}
// StatusSnapshot is statistic for specific method.
type StatusSnapshot struct {
allTime uint64
allRequests uint64
}
func (s StatusSnapshot) AllRequests() uint64 {
return s.allRequests
}
func (s StatusSnapshot) AllTime() uint64 {
return s.allTime
}

View file

@ -43,7 +43,7 @@ func (c *treeClient) dial(ctx context.Context) error {
}
var err error
if c.conn, c.service, err = dialClient(ctx, c.address, c.opts...); err != nil {
if c.conn, c.service, err = createClient(c.address, c.opts...); err != nil {
return err
}
@ -61,7 +61,7 @@ func (c *treeClient) redialIfNecessary(ctx context.Context) (healthHasChanged bo
defer c.mu.Unlock()
if c.conn == nil {
if c.conn, c.service, err = dialClient(ctx, c.address, c.opts...); err != nil {
if c.conn, c.service, err = createClient(c.address, c.opts...); err != nil {
return false, err
}
}
@ -77,7 +77,7 @@ func (c *treeClient) redialIfNecessary(ctx context.Context) (healthHasChanged bo
return !wasHealthy, nil
}
func dialClient(ctx context.Context, addr string, clientOptions ...grpc.DialOption) (*grpc.ClientConn, grpcService.TreeServiceClient, error) {
func createClient(addr string, clientOptions ...grpc.DialOption) (*grpc.ClientConn, grpcService.TreeServiceClient, error) {
host, tlsEnable, err := apiClient.ParseURI(addr)
if err != nil {
return nil, nil, fmt.Errorf("parse address: %w", err)
@ -93,9 +93,9 @@ func dialClient(ctx context.Context, addr string, clientOptions ...grpc.DialOpti
// the order is matter, we want client to be able to overwrite options.
opts := append(options, clientOptions...)
conn, err := grpc.DialContext(ctx, host, opts...)
conn, err := grpc.NewClient(host, opts...)
if err != nil {
return nil, nil, fmt.Errorf("grpc dial node tree service: %w", err)
return nil, nil, fmt.Errorf("grpc create node tree service: %w", err)
}
return conn, grpcService.NewTreeServiceClient(conn), nil

View file

@ -88,6 +88,7 @@ type Pool struct {
rebalanceParams rebalanceParameters
dialOptions []grpc.DialOption
logger *zap.Logger
methods []*pool.MethodStatus
maxRequestAttempts int
@ -169,6 +170,39 @@ type RemoveNodeParams struct {
BearerToken []byte
}
// MethodIndex index of method in list of statuses in Pool.
type MethodIndex int
const (
methodGetNodes MethodIndex = iota
methodGetSubTree
methodAddNode
methodAddNodeByPath
methodMoveNode
methodRemoveNode
methodLast
)
// String implements fmt.Stringer.
func (m MethodIndex) String() string {
switch m {
case methodGetNodes:
return "getNodes"
case methodAddNode:
return "addNode"
case methodGetSubTree:
return "getSubTree"
case methodAddNodeByPath:
return "addNodeByPath"
case methodMoveNode:
return "moveNode"
case methodRemoveNode:
return "removeNode"
default:
return "unknown"
}
}
// NewPool creates connection pool using parameters.
func NewPool(options InitParameters) (*Pool, error) {
if options.key == nil {
@ -182,6 +216,11 @@ func NewPool(options InitParameters) (*Pool, error) {
fillDefaultInitParams(&options)
methods := make([]*pool.MethodStatus, methodLast)
for i := methodGetNodes; i < methodLast; i++ {
methods[i] = pool.NewMethodStatus(i.String())
}
p := &Pool{
key: options.key,
logger: options.logger,
@ -192,6 +231,7 @@ func NewPool(options InitParameters) (*Pool, error) {
clientRebalanceInterval: options.clientRebalanceInterval,
},
maxRequestAttempts: options.maxRequestAttempts,
methods: methods,
}
return p, nil
@ -308,22 +348,25 @@ func (p *Pool) GetNodes(ctx context.Context, prm GetNodesParams) ([]*grpcService
},
}
start := time.Now()
if err := p.signRequest(request); err != nil {
return nil, err
}
var resp *grpcService.GetNodeByPathResponse
if err := p.requestWithRetry(ctx, func(client grpcService.TreeServiceClient) (inErr error) {
err := p.requestWithRetry(ctx, func(client grpcService.TreeServiceClient) (inErr error) {
resp, inErr = client.GetNodeByPath(ctx, request)
// Pool wants to do retry 'GetNodeByPath' request if result is empty.
// Empty result is expected due to delayed tree service sync.
// Return an error there to trigger retry and ignore it after,
// to keep compatibility with 'GetNodeByPath' implementation.
if inErr == nil && len(resp.Body.Nodes) == 0 {
if inErr == nil && len(resp.GetBody().GetNodes()) == 0 {
return errNodeEmptyResult
}
return handleError("failed to get node by path", inErr)
}); err != nil && !errors.Is(err, errNodeEmptyResult) {
})
p.methods[methodGetNodes].IncRequests(time.Since(start))
if err != nil && !errors.Is(err, errNodeEmptyResult) {
return nil, err
}
@ -339,14 +382,14 @@ type SubTreeReader struct {
// Read reads another list of the subtree nodes.
func (x *SubTreeReader) Read(buf []*grpcService.GetSubTreeResponse_Body) (int, error) {
for i := 0; i < len(buf); i++ {
for i := range len(buf) {
resp, err := x.cli.Recv()
if err == io.EOF {
return i, io.EOF
} else if err != nil {
return i, handleError("failed to get sub tree", err)
}
buf[i] = resp.Body
buf[i] = resp.GetBody()
}
return len(buf), nil
@ -362,7 +405,7 @@ func (x *SubTreeReader) ReadAll() ([]*grpcService.GetSubTreeResponse_Body, error
} else if err != nil {
return nil, handleError("failed to get sub tree", err)
}
res = append(res, resp.Body)
res = append(res, resp.GetBody())
}
return res, nil
@ -378,7 +421,7 @@ func (x *SubTreeReader) Next() (*grpcService.GetSubTreeResponse_Body, error) {
return nil, handleError("failed to get sub tree", err)
}
return resp.Body, nil
return resp.GetBody(), nil
}
// GetSubTree invokes eponymous method from TreeServiceClient.
@ -405,15 +448,18 @@ func (p *Pool) GetSubTree(ctx context.Context, prm GetSubTreeParams) (*SubTreeRe
request.Body.OrderBy.Direction = grpcService.GetSubTreeRequest_Body_Order_None
}
start := time.Now()
if err := p.signRequest(request); err != nil {
return nil, err
}
var cli grpcService.TreeService_GetSubTreeClient
if err := p.requestWithRetry(ctx, func(client grpcService.TreeServiceClient) (inErr error) {
err := p.requestWithRetry(ctx, func(client grpcService.TreeServiceClient) (inErr error) {
cli, inErr = client.GetSubTree(ctx, request)
return handleError("failed to get sub tree client", inErr)
}); err != nil {
})
p.methods[methodGetSubTree].IncRequests(time.Since(start))
if err != nil {
return nil, err
}
@ -435,15 +481,19 @@ func (p *Pool) AddNode(ctx context.Context, prm AddNodeParams) (uint64, error) {
BearerToken: prm.BearerToken,
},
}
start := time.Now()
if err := p.signRequest(request); err != nil {
return 0, err
}
var resp *grpcService.AddResponse
if err := p.requestWithRetry(ctx, func(client grpcService.TreeServiceClient) (inErr error) {
err := p.requestWithRetry(ctx, func(client grpcService.TreeServiceClient) (inErr error) {
resp, inErr = client.Add(ctx, request)
return handleError("failed to add node", inErr)
}); err != nil {
})
p.methods[methodAddNode].IncRequests(time.Since(start))
if err != nil {
return 0, err
}
@ -467,27 +517,30 @@ func (p *Pool) AddNodeByPath(ctx context.Context, prm AddNodeByPathParams) (uint
},
}
start := time.Now()
if err := p.signRequest(request); err != nil {
return 0, err
}
var resp *grpcService.AddByPathResponse
if err := p.requestWithRetry(ctx, func(client grpcService.TreeServiceClient) (inErr error) {
err := p.requestWithRetry(ctx, func(client grpcService.TreeServiceClient) (inErr error) {
resp, inErr = client.AddByPath(ctx, request)
return handleError("failed to add node by path", inErr)
}); err != nil {
})
p.methods[methodAddNodeByPath].IncRequests(time.Since(start))
if err != nil {
return 0, err
}
body := resp.GetBody()
if body == nil {
return 0, errors.New("nil body in tree service response")
} else if len(body.Nodes) == 0 {
} else if len(body.GetNodes()) == 0 {
return 0, errors.New("empty list of added nodes in tree service response")
}
// The first node is the leaf that we add, according to tree service docs.
return body.Nodes[0], nil
return body.GetNodes()[0], nil
}
// MoveNode invokes eponymous method from TreeServiceClient.
@ -507,16 +560,20 @@ func (p *Pool) MoveNode(ctx context.Context, prm MoveNodeParams) error {
},
}
start := time.Now()
if err := p.signRequest(request); err != nil {
return err
}
return p.requestWithRetry(ctx, func(client grpcService.TreeServiceClient) error {
err := p.requestWithRetry(ctx, func(client grpcService.TreeServiceClient) error {
if _, err := client.Move(ctx, request); err != nil {
return handleError("failed to move node", err)
}
return nil
})
p.methods[methodMoveNode].IncRequests(time.Since(start))
return err
}
// RemoveNode invokes eponymous method from TreeServiceClient.
@ -533,16 +590,21 @@ func (p *Pool) RemoveNode(ctx context.Context, prm RemoveNodeParams) error {
BearerToken: prm.BearerToken,
},
}
start := time.Now()
if err := p.signRequest(request); err != nil {
return err
}
return p.requestWithRetry(ctx, func(client grpcService.TreeServiceClient) error {
err := p.requestWithRetry(ctx, func(client grpcService.TreeServiceClient) error {
if _, err := client.Remove(ctx, request); err != nil {
return handleError("failed to remove node", err)
}
return nil
})
p.methods[methodRemoveNode].IncRequests(time.Since(start))
return err
}
// Close closes the Pool and releases all the associated resources.
@ -563,6 +625,18 @@ func (p *Pool) Close() error {
return err
}
// Statistic returns tree pool statistics.
func (p *Pool) Statistic() Statistic {
stat := Statistic{make([]pool.StatusSnapshot, len(p.methods))}
for i, method := range p.methods {
stat.methods[i] = method.Snapshot()
method.Reset()
}
return stat
}
func handleError(msg string, err error) error {
if err == nil {
return nil
@ -762,6 +836,11 @@ LOOP:
if startI != indexI || startJ != indexJ {
p.setStartIndices(indexI, indexJ)
}
if err != nil {
err = fmt.Errorf("address %s: %w", p.innerPools[indexI].clients[indexJ].endpoint(), err)
}
return err
}

51
pool/tree/statistic.go Normal file
View file

@ -0,0 +1,51 @@
package tree
import (
"time"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/pool"
)
// Statistic is metrics of the tree pool.
type Statistic struct {
methods []pool.StatusSnapshot
}
func (s *Statistic) Requests() (requests uint64) {
for _, val := range s.methods {
requests += val.AllRequests()
}
return requests
}
func (s *Statistic) AverageGetNodes() time.Duration {
return s.averageTime(methodGetNodes)
}
func (s *Statistic) AverageGetSubTree() time.Duration {
return s.averageTime(methodGetSubTree)
}
func (s *Statistic) AverageAddNode() time.Duration {
return s.averageTime(methodAddNode)
}
func (s *Statistic) AverageAddNodeByPath() time.Duration {
return s.averageTime(methodAddNodeByPath)
}
func (s *Statistic) AverageMoveNode() time.Duration {
return s.averageTime(methodMoveNode)
}
func (s *Statistic) AverageRemoveNode() time.Duration {
return s.averageTime(methodRemoveNode)
}
func (s *Statistic) averageTime(method MethodIndex) time.Duration {
stat := s.methods[method]
if stat.AllRequests() == 0 {
return 0
}
return time.Duration(stat.AllTime() / stat.AllRequests())
}

View file

@ -237,6 +237,7 @@ const (
VerbObjectDelete // Delete rpc
VerbObjectRange // GetRange rpc
VerbObjectRangeHash // GetRangeHash rpc
VerbObjectPatch // Patch rpc
)
// ForVerb specifies the object operation of the session scope. Each

View file

@ -599,6 +599,7 @@ func TestObject_ForVerb(t *testing.T) {
session.VerbObjectRangeHash: v2session.ObjectVerbRangeHash,
session.VerbObjectRange: v2session.ObjectVerbRange,
session.VerbObjectDelete: v2session.ObjectVerbDelete,
session.VerbObjectPatch: v2session.ObjectVerbPatch,
} {
val.ForVerb(from)