Release v0.27.0

Signed-off-by: Alex Vanin <a.vanin@yadro.com>

CHANGELOG.md

@@ -4,433 +4,48 @@ This document outlines major changes between releases.
 
 ## [Unreleased]
 
+## [0.27.0] - Karpinsky - 2023-07-12
+
+This is a first FrostFS S3 Gateway release named after
+[Karpinsky glacier](https://en.wikipedia.org/wiki/Karpinsky_Glacier).
+
 ### Fixed
-- Clean up List and Name caches when object is missing in Tree service (#57)
+- Using multiple servers require only one healthy (#12)
-- Get empty bucket CORS from frostfs (TrueCloudLab#36)
+- Renew token before it expires (#20)
+- Add generated deb builder files to .gitignore, and fix typo (#28)
+- Get empty bucket CORS from frostfs (#36)
 - Don't count pool error on client abort (#35)
+- Handle request cancelling (#69)
+- Clean up List and Name caches when object is missing in Tree service (#57)
+- Don't create unnecessary delete-markers (#83)
+- `Too many pings` error (#145)
 
 ### Added
+- Billing metrics (#5, #26, #29)
+- Return container name in `head-bucket` response (#18)
+- Multiple configs support (#21)
+- Bucket name resolving policy (#25)
+- Support string `Action` and `Resource` fields in `bucketPolicy.Statement` (#32)
+- Add new `kludge.use_default_xmlns_for_complete_multipart` config param (#40)
 - Return `X-Owner-Id` in `head-bucket` response (#79)
-- Return container name in `head-bucket` response (TrueCloudLab#18)
+- Support multiple tree service endpoints (#74, #110, #114)
-- Billing metrics (TrueCloudLab#5)
-- Multiple configs support (TrueCloudLab#21)
-- Bucket name resolving policy (TrueCloudLab#25)
-- Support string `Action` and `Resource` fields in `bucketPolicy.Statement` (TrueCloudLab#32)
-- Add new `kludge.use_default_xmlns_for_complete_multipart` config param (TrueCloudLab#40)
 
 ### Changed
+- Repository rebranding (#1)
 - Update neo-go to v0.101.0 (#14)
 - Update viper to v1.15.0 (#14)
-- Using multiple servers require only one healthy (TrueCloudLab#12)
+- Update go version to go1.18 (#16)
-- Update go version to go1.18 (TrueCloudLab#16)
+- Return error on invalid LocationConstraint (#23)
-- Return error on invalid LocationConstraint (TrueCloudLab#23)
+- Limit number of objects to delete at one time (#37)
-- Place billing metrics to separate url path (TrueCloudLab#26)
-- Add generated deb builder files to .gitignore, and fix typo (TrueCloudLab#28)
-- Limit number of objects to delete at one time (TrueCloudLab#37)
 - CompleteMultipartUpload handler now sends whitespace characters to keep alive client's connection (#60)
 - Support new system attributes (#64)
-
+- Abstract network communication in TreeClient (#59, #75)
-## [0.26.0] - 2022-12-28
+- Changed values for `frostfs_s3_gw_state_health` metric (#91)
-
-### Added
-- Use client time as `now` in some requests (#726)
-- Reload policies on SIGHUP (#747)
-- Authmate flags for pool timeouts (#760)
-- Multiple server listeners (#742)
-
-### Changed
-- Placement policy configuration (#568)
-- Improved debug logging of CID and OID values (#754)
-
-### Removed
-- Deprecated linters (#755)
-
-### Updating from v0.25.1
-New config parameters were added. And old one `defaul_policy` were changed.
-```yaml
-placement_policy:
-  default: "REP 3"
-  region_mapping: /path/to/container/policies.json
-```
-
-Make sure you update the config accordingly:
-If you configure application using environment variables change:
-* `S3_GW_DEFAULT_POLICY` -> `S3_GW_PLACEMENT_POLICY_DEFAULT_POLICY`
-* `S3_GW_LISTEN_ADDRESS` -> `S3_GW_SERVER_0_ADDRESS`
-* `S3_GW_TLS_CERT_FILE` -> `S3_GW_SERVER_0_TLS_CERT_FILE` (and set `S3_GW_SERVER_0_TLS_ENABLED=true`)
-* `S3_GW_TLS_KEY_FILE` -> `S3_GW_SERVER_0_TLS_KEY_FILE` (and set `S3_GW_SERVER_0_TLS_ENABLED=true`)
-
-If you configure application using `.yaml` file change:
-* `defaul_policy` -> `placement_policy.default`
-* `listen_address` -> `server.0.address`
-* `tls.cert_file` -> `server.0.tls.cert_file` (and set `server.0.tls.enabled: true`)
-* `tls.key_file` -> `server.0.tls.key_file` (and set `server.0.tls.enabled: true`)
-
-## [0.25.1] - 2022-10-30
-
-### Fixed
-- Empty bucket policy (#740)
-- Big object removal (#749)
-- Checksum panic (#741)
-
-### Added
-- Debian packaging (#737)
-- Timeout for individual operations in streaming RPC (#750)
-
-## [0.25.0] - 2022-10-31
-
-### Fixed
-- Legal hold object lock enabling (#709)
-- Errors at object locking (#719)
-- Unrestricted access to not owned objects via cache (#713)
-- Check tree service health (#699)
-- Bucket names in listing (#733)
-
-### Added
-- Config reloading on SIGHUP (#702, #715, #716)
-- Stop pool dial on SIGINT (#712)
-
-### Changed
-- GitHub actions update (#710)
-- Makefile help (#725)
-- Optimized object tags setting (#669)
-- Improved logging (#728)
-- Unified unit test names (#617)
-- Improved docs (#732)
-
-### Removed
-- Unused cache methods (#650)
-
-### Updating from v0.24.0
-New config parameters were added. Make sure the default parameters are appropriate for you.
-
-```yaml
-cache:
-  accesscontrol:
-    lifetime: 1m
-    size: 100000
-```
-
-## [0.24.0] - 2022-09-14
-
-### Added
-- Exposure of pool metrics (#615, #680)
-- Configuration of `set_copies_number` (#634, #637)
-- Configuration of list of allowed `AccessKeyID` prefixes (#674)
-- Tagging directive for `CopyObject` (#666, #683)
-- Customer encryption (#595)
-- `CopiesNumber` configuration (#634, #637)
-
-### Changed
-- Improved wallet configuration via `.yaml` config and environment variables (#607)
-- Update go version for build to 1.19 (#694, #705)
-- Update version calculation (#653, #697)
-- Optimized lock creation (#692)
-- Update way to configure `listen_domains` (#667)
-- Use `FilePath` instead of `FileName` for object keys (#657)
-- Optimize listing (#625, #616)
-
-### Removed
-- Drop any object search logic (#545)
-
-### Fixed
-- Responses to `GetObject` and `HeadObject`: removed redundant `VersionID` (#577, #682)
-- Replacement of object tagging in case of overwriting of an object (#645)
-- Using tags cache with empty `versionId` (#643)
-- Fix panic on go1.19 (#678)
-- Fix panic on invalid versioning status (#660)
-- Fix panic on missing decrypt reader (#704)
-- Using multipart uploads with `/` in name (#671)
-- Don't update settings cache when request fails (#661)
-- Fix handling `X-Amz-Copy-Source` header (#672)
-- ACL related problems (#676, #606)
-- Using `ContinuationToken` for "directories" (#684)
-- Fix `connection was closed` error (#656)
-- Fix listing for nested objects (#624)
-- Fix anon requests to tree service (#504, #505)
-
-### Updating from v0.23.0
-Make sure your configuration is valid:
-
-If you configure application using environment variables change:
-* `S3_GW_WALLET` -> `S3_GW_WALLET_PATH`
-* `S3_GW_ADDRESS` -> `S3_GW_WALLET_ADDRESS`
-* `S3_GW_LISTEN_DOMAINS_N` -> `S3_GW_LISTEN_DOMAINS` (use it as array variable)
-
-If you configure application using `.yaml` file change:
-* `wallet` -> `wallet.path`
-* `address` -> `wallet.address`
-* `listen_domains.n` -> `listen_domains` (use it as array param)
-
-
-## [0.23.0] - 2022-08-01
-
-### Fixed
-- System metadata are filtered now (#619)
-- List objects in corner cases (#612, #627)
-- Correct removal of a deleted object (#610)
-- Bucket creation could lead to "no healthy client" error (#636)
-
-### Added
-- New param to configure pool error threshold (#633)
-
-### Changed
-- Pprof and prometheus metrics configuration (#591)
-- Don't set sticky bit in authmate container (#540)
-- Updated compatibility table (#638)
-- Rely on string sanitizing from zap (#498)
-
-### Updating from v0.22.0
-1. To enable pprof use `pprof.enabled` instead of `pprof` in config.
-To enable prometheus metrics use `prometheus.enabled` instead of `metrics` in config.
-If you are using the command line flags you can skip this step.
-
-## [0.22.0] - 2022-07-25
-
-Tree service support
-
-### Fixed
-- Error logging (#450)
-- Default bucket location constraint (#463)
-- Suspended versioning status (#462)
-- CodeQL warnings (#489, #522, #539)
-- Bearer token behaviour with non-owned buckets (#459)
-- ACL issues (#495, #553, #571, #573, #574, #580)
-- Authmate policy parsing (#558)
-
-### Added
-- Public key output in authmate issue-secret command (#482)
-- Support of conditional headers (#484)
-- Cache type cast error logging (#465)
-- `docker/*` target in Makefile (#471)
-- Pre signed requests (#529)
-- Tagging and ACL notifications (#361)
-- AWSv4 signer package to improve compatibility with S3 clients (#528)
-- Extension mimetype detector (#289)
-- Default params documentation (#592)
-- Health metric (#600)
-- Parallel object listing (#525)
-- Tree service (see commit links from #609)
-
-### Changed
-- Reduce number of network requests (#439, #441)
-- Renamed authmate to s3-authmate (#518)
-- Version output (#578)
-- Improved error messages (#539)
-
-### Removed
-- `layer/neofs` package (#438)
-
-## [0.21.1] - 2022-05-16
-
-### Changed
-- Update go version to go1.17 (#427)
-- Set homomorphic hashing disable attribute in container if required (#435)
-
-## [0.21.0] - 2022-05-13
-
-### Added
-- Support of get-object-attributes (#430)
-
-### Fixed
-- Reduced time of bucket creation (#426)
-- Bucket removal (#428)
-- Obtainment of ETag value (#431)
-
-### Changed
-- Authmate doesn't parse session context anymore, now it accepts application defined
-  flexible structure with container ID in human-readable format (#428)
-
-## [0.20.0] - 2022-04-29
-
-### Added
-- Support of object locking (#195)  
-- Support of basic notifications (#357, #358, #359)
-
-### Changed
-- Logger behavior: now it writes to stderr instead of stdout, app name and
-  version are always presented and fixed, all user options except of `level` are
-  dropped (#380)
-- Improved docs, added config examples (#396, #398)
-- Updated NeoFS SDK (#365, #409)
-
-### Fixed
-- Added check of `SetEACL` tokens before processing of requests (#347)
-- Authmate: returned lost session tokens when a parameter `--session-token` is
-  omitted (#387)
-- Error when a bucket hasn't a settings file (#389)
-- Response to a request to delete not existing object (#392)
-- Replaced gate key in ACL Grantee by key of bearer token issuer (#395)
-- Missing attach of bearer token to requests to put system object (#399)
-- Deletion of system object while CompleteMultipartUpload (#400)
-- Improved English in docs and comments (#405)
-- Authmate: reconsidered default bearer token rules (#406)
-
-## [0.19.0] - 2022-03-16
-
-### Added
-- Authmate: support placement policy overriding (#343, #364)
-- Managing bucket notification configuration (#340)
-- Unit tests in go1.17 (#265)
-- NATS settings in application config (#341)
-- Support `Expires` and `Cache-Control` headers (#312)
-- Support `%` as delimiter (#313)
-- Support `null` version deletion (#319)
-- Bucket name resolving order (#285)
-- Authmate: added `timeout` flag (#290)
-- MinIO results in s3 compatibility tables (#304)
-- Support overriding response headers (#310)
-
-### Changed
-- Authmate: check parameters before container creation (#372)
-- Unify cache invalidation on deletion (#368)
-- Updated NeoFS SDK to v1.0.0-rc.3 (#297, #333, #346, #376)
-- Authmate: changed session token rules handling (#329, #336, #338, #352)
-- Changed status code for some failed requests (#308)
-- GetBucketLocation returns policy name used at bucket creation (#301)
-
-### Fixed
-- Waiting for bucket to be deleted (#366)
-- Authmate: changed error message for session context building (#348)
-- Authmate: fixed access key parsing in `obtain-secret` command (#295)
-- Distinguishing `BucketAlreadyExists` errors (#354)
-- Incorrect panic if handler not found (#305)
-- Authmate: use container friendly name as system name (#299, #324)
-- Use UTC `Last-Modified` timestamps (#331)
-- Don't return object system metadata (#307)
-- Handling empty post policy (#306)
-- Use `X-Amz-Verion-Id` in `CompleteMulipartUpload` (#318)
-
-### Removed
-- Drop MinIO related errors (#316)
-
-## [0.18.0] - 2021-12-16
-
-### Added
-- Support for MultipartUpload (#186, #187)
-- CORS support (#217)
-- Authmate supports setting of tokens lifetime in a more convenient format (duration) (#258)
-- Generation of a random key for `--no-sign-request` (#276)
-
-### Changed
-- Bucket name resolving mechanism from listing owner's containers to using DNS (#219)
-
-### Removed
-- Deprecated golint, replaced by revive (#272)
-
-## 0.17.0 (24 Sep 2021)
-With this release we introduce [ceph-based](https://github.com/ceph/s3-tests) S3 compatibility results.
-
-### Added
-* Versioning support (#122, #242, #263)
-* Ceph S3 compatibility results (#150, #249, #266)
-* Handling `X-Amz-Expected-Bucket-Owner` header (#216)
-* `X-Container-Id` header for `HeadBucket` response (#220)
-* Basic ACL support (#49, #213)
-* Caching (#179, #206, #231, #236, #253)
-* Metadata directive when copying (#191)
-* Bucket name checking (189)
-* Continuation token support (#112, #154, #180)
-* Mapping `LocationConstraint` to `PlacementPolicy` (#89)
-* Tagging support (#196)
-* POST uploading support (#190)
-* Delete marker support (#248)
-* Expiration for access box (#255)
-* AWS CLI credential generating by authmate (#241)
-
-### Changed
-* Default placement policy is now configurable (#218)
-* README is split into different files (#210)
-* Unified error handling (#89, #149, #184)
-* Authmate issue-secret response contains container id (#163)
-* Removed "github.com/nspcc-dev/neofs-node" dependency (#234)
-* Removed GitHub workflow of image publishing (#243)
-* Changed license to AGPLv3 (#264)
-
-### Fixed
-* ListObjects results are now the same for different users (#230)
-* Error response for invalid authentication header is now correct (#199)
-* Saving object metadata (#198)
-* Range header handling (#194)
-* Correct status codes (#118, #262)
-* HeadObject for "directories" (#160)
-* Fetch-owner parameter support (#159)
-
-## 0.16.0 (16 Jul 2021)
-
-With this release we publish S3 gateway source code. It includes various S3
-compatibility improvements, support of bucket management, unified secp256r1
-cryptography with NEP-6 wallet support.
-
-### Fixed
- * Allowed no-sign request (#65)
- * Bearer token attached to all requests (#84)
- * Time format in responses (#133)
- * Max-keys checked in ListObjects (#135)
- * Lost metadat in the objects (#131)
- * Unique bucket name check (#125)
-
-### Added
- * Bucket management operations (#47, #72)
- * Node-specific owner IDs in bearer tokens (#83)
- * AWS CLI usage section in README (#77)
- * List object paging (#97)
- * Lifetime for the tokens in auth-mate (#108)
- * Support of range in GetObject request (#96)
- * Support of NEP-6 wallets instead of binary encoded keys (#92)
- * Support of JSON encoded rules in auth-mate (#71)
- * Support of delimiters in ListObjects (#98)
- * Support of object ETag (#93)
- * Support of time-based conditional CopyObject and GetObject (#94)
-
-### Changed
- * Accesskey format: now `0` used as a delimiter between container ID and object
-   ID instead of `_` (#164)
- * Accessbox is encoded in protobuf format (#48)
- * Authentication uses secp256r1 instead of ed25519 (#75)
- * Improved integration with NeoFS SDK and NeoFS API Go (#78, #88)
- * Optimized object put execution (#155)
-
-### Removed
- * GRPC keepalive options (#73)
-
-## 0.15.0 (10 Jun 2021)
-
-This release brings S3 gateway to the current state of NeoFS and fixes some
-bugs, no new significant features introduced (other than moving here already
-existing authmate component).
-
-New features:
- * authmate was moved into this repository and is now built along with the
-   gateway itself (#46)
-
-Behavior changes:
- * neofs-s3-gate was renamed to neofs-s3-gw (#50)
-
-Improvements:
- * better Makefile (#43, #45, #55)
- * stricter linters (#45)
- * removed non-standard errors package from dependencies (#54)
- * refactoring, reusing new sdk-go component (#60, #62, #63)
- * updated neofs-api-go for compatibility with current NeoFS node 0.21.0 (#60, #68)
- * extended README (#67, #76)
-
-Bugs fixed:
- * wrong (as per AWS specification) access key ID generated (#64)
 
 ## Older versions
 
-Please refer to [Github
+This project is a fork of [NeoFS S3 Gateway](https://github.com/nspcc-dev/neofs-s3-gw) from version v0.26.0.
-releases](https://github.com/nspcc-dev/neofs-s3-gw/releases/) for older
+To see CHANGELOG for older versions, refer to https://github.com/nspcc-dev/neofs-s3-gw/blob/master/CHANGELOG.md.
-releases.
 
-[0.18.0]: https://github.com/nspcc-dev/neofs-s3-gw/compare/v0.17.0...v0.18.0
+[0.27.0]: https://git.frostfs.info/TrueCloudLab/frostfs-s3-gw/compare/b2148cc3...v0.27.0
-[0.19.0]: https://github.com/nspcc-dev/neofs-s3-gw/compare/v0.18.0...v0.19.0
+[Unreleased]: https://git.frostfs.info/TrueCloudLab/frostfs-s3-gw/compare/v0.27.0...master
-[0.20.0]: https://github.com/nspcc-dev/neofs-s3-gw/compare/v0.19.0...v0.20.0
-[0.21.0]: https://github.com/nspcc-dev/neofs-s3-gw/compare/v0.20.0...v0.21.0
-[0.21.1]: https://github.com/nspcc-dev/neofs-s3-gw/compare/v0.21.0...v0.21.1
-[0.22.0]: https://github.com/nspcc-dev/neofs-s3-gw/compare/v0.21.1...v0.22.0
-[0.23.0]: https://github.com/nspcc-dev/neofs-s3-gw/compare/v0.22.0...v0.23.0
-[0.24.0]: https://github.com/nspcc-dev/neofs-s3-gw/compare/v0.23.0...v0.24.0
-[0.25.0]: https://github.com/nspcc-dev/neofs-s3-gw/compare/v0.24.0...v0.25.0
-[Unreleased]: https://github.com/nspcc-dev/neofs-s3-gw/compare/v0.25.0...master

VERSION

@@ -1 +1 @@
-v0.26.0
+v0.27.0

api/handler/delete_test.go

@@ -147,6 +147,81 @@ func TestRemoveDeleteMarker(t *testing.T) {
 	require.True(t, existInMockedFrostFS(tc, bktInfo, objInfo), "object doesn't exist but should")
 }
 
+func TestDeleteMarkerVersioned(t *testing.T) {
+	tc := prepareHandlerContext(t)
+
+	bktName, objName := "bucket-for-removal", "object-to-delete"
+	createVersionedBucketAndObject(t, tc, bktName, objName)
+
+	t.Run("not create new delete marker if last version is delete marker", func(t *testing.T) {
+		deleteMarkerVersion, isDeleteMarker := deleteObject(t, tc, bktName, objName, emptyVersion)
+		require.True(t, isDeleteMarker)
+		versions := listVersions(t, tc, bktName)
+		require.Equal(t, deleteMarkerVersion, versions.DeleteMarker[0].VersionID)
+
+		_, isDeleteMarker = deleteObject(t, tc, bktName, objName, emptyVersion)
+		require.True(t, isDeleteMarker)
+		versions = listVersions(t, tc, bktName)
+		require.Len(t, versions.DeleteMarker, 1)
+		require.Equal(t, deleteMarkerVersion, versions.DeleteMarker[0].VersionID)
+	})
+
+	t.Run("do not create delete marker if object does not exist", func(t *testing.T) {
+		versionsBefore := listVersions(t, tc, bktName)
+		_, isDeleteMarker := deleteObject(t, tc, bktName, "dummy", emptyVersion)
+		require.False(t, isDeleteMarker)
+		versionsAfter := listVersions(t, tc, bktName)
+		require.Equal(t, versionsBefore, versionsAfter)
+	})
+}
+
+func TestDeleteMarkerSuspended(t *testing.T) {
+	tc := prepareHandlerContext(t)
+
+	bktName, objName := "bucket-for-removal", "object-to-delete"
+	bktInfo, _ := createVersionedBucketAndObject(t, tc, bktName, objName)
+	putBucketVersioning(t, tc, bktName, false)
+
+	t.Run("not create new delete marker if last version is delete marker", func(t *testing.T) {
+		deleteMarkerVersion, isDeleteMarker := deleteObject(t, tc, bktName, objName, emptyVersion)
+		require.True(t, isDeleteMarker)
+		require.Equal(t, data.UnversionedObjectVersionID, deleteMarkerVersion)
+
+		deleteMarkerVersion, isDeleteMarker = deleteObject(t, tc, bktName, objName, emptyVersion)
+		require.True(t, isDeleteMarker)
+		require.Equal(t, data.UnversionedObjectVersionID, deleteMarkerVersion)
+
+		versions := listVersions(t, tc, bktName)
+		require.Len(t, versions.DeleteMarker, 1)
+		require.Equal(t, deleteMarkerVersion, versions.DeleteMarker[0].VersionID)
+	})
+
+	t.Run("do not create delete marker if object does not exist", func(t *testing.T) {
+		versionsBefore := listVersions(t, tc, bktName)
+		_, isDeleteMarker := deleteObject(t, tc, bktName, "dummy", emptyVersion)
+		require.False(t, isDeleteMarker)
+		versionsAfter := listVersions(t, tc, bktName)
+		require.Equal(t, versionsBefore, versionsAfter)
+	})
+
+	t.Run("remove last unversioned non delete marker", func(t *testing.T) {
+		objName := "obj3"
+		putObject(t, tc, bktName, objName)
+
+		nodeVersion, err := tc.tree.GetUnversioned(tc.Context(), bktInfo, objName)
+		require.NoError(t, err)
+
+		deleteMarkerVersion, isDeleteMarker := deleteObject(t, tc, bktName, objName, emptyVersion)
+		require.True(t, isDeleteMarker)
+		require.Equal(t, data.UnversionedObjectVersionID, deleteMarkerVersion)
+
+		objVersions := getVersion(listVersions(t, tc, bktName), objName)
+		require.Len(t, objVersions, 0)
+
+		require.False(t, tc.MockedPool().ObjectExists(nodeVersion.OID))
+	})
+}
+
 func TestDeleteObjectCombined(t *testing.T) {
 	tc := prepareHandlerContext(t)
 
@@ -197,7 +272,7 @@ func TestDeleteMarkers(t *testing.T) {
 	deleteObject(t, tc, bktName, objName, emptyVersion)
 
 	versions := listVersions(t, tc, bktName)
-	require.Len(t, versions.DeleteMarker, 3, "invalid delete markers length")
+	require.Len(t, versions.DeleteMarker, 0, "invalid delete markers length")
 	require.Len(t, versions.Version, 0, "versions must be empty")
 
 	require.Len(t, listOIDsFromMockedFrostFS(t, tc, bktName), 0, "shouldn't be any object in frostfs")
@@ -316,6 +391,16 @@ func listVersions(t *testing.T, tc *handlerContext, bktName string) *ListObjects
 	return res
 }
 
+func getVersion(resp *ListObjectsVersionsResponse, objName string) []*ObjectVersionResponse {
+	var res []*ObjectVersionResponse
+	for i, version := range resp.Version {
+		if version.Key == objName {
+			res = append(res, &resp.Version[i])
+		}
+	}
+	return res
+}
+
 func putObject(t *testing.T, tc *handlerContext, bktName, objName string) {
 	body := bytes.NewReader([]byte("content"))
 	w, r := prepareTestPayloadRequest(tc, bktName, objName, body)

api/handler/handlers_test.go

@@ -33,6 +33,7 @@ type handlerContext struct {
 	t       *testing.T
 	h       *handler
 	tp      *layer.TestFrostFS
+	tree    *tree.Tree
 	context context.Context
 }
 
@@ -85,11 +86,13 @@ func prepareHandlerContext(t *testing.T) *handlerContext {
 	var owner user.ID
 	user.IDFromKey(&owner, key.PrivateKey.PublicKey)
 
+	treeMock := NewTreeServiceMock(t)
+
 	layerCfg := &layer.Config{
 		Caches:      layer.DefaultCachesConfigs(zap.NewExample()),
 		AnonKey:     layer.AnonymousKey{Key: key},
 		Resolver:    testResolver,
-		TreeService: NewTreeServiceMock(t),
+		TreeService: treeMock,
 	}
 
 	var pp netmap.PlacementPolicy
@@ -110,6 +113,7 @@ func prepareHandlerContext(t *testing.T) *handlerContext {
 		t:       t,
 		h:       h,
 		tp:      tp,
+		tree:    treeMock,
 		context: context.WithValue(context.Background(), api.BoxData, newTestAccessBox(t, key)),
 	}
 }
@@ -117,7 +121,7 @@ func prepareHandlerContext(t *testing.T) *handlerContext {
 func NewTreeServiceMock(t *testing.T) *tree.Tree {
 	memCli, err := tree.NewTreeServiceClientMemory()
 	require.NoError(t, err)
-	return tree.NewTree(memCli)
+	return tree.NewTree(memCli, zap.NewExample())
 }
 
 func createTestBucket(hc *handlerContext, bktName string) *data.BucketInfo {

api/layer/frostfs_mock.go

@@ -15,6 +15,7 @@ import (
 	"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/creds/accessbox"
 	"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/bearer"
 	"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/checksum"
+	apistatus "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/client/status"
 	"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container"
 	cid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/id"
 	"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/eacl"
@@ -55,6 +56,16 @@ func (t *TestFrostFS) Objects() []*object.Object {
 	return res
 }
 
+func (t *TestFrostFS) ObjectExists(objID oid.ID) bool {
+	for _, obj := range t.objects {
+		if id, _ := obj.ID(); id.Equals(objID) {
+			return true
+		}
+	}
+
+	return false
+}
+
 func (t *TestFrostFS) AddObject(key string, obj *object.Object) {
 	t.objects[key] = obj
 }
@@ -161,7 +172,7 @@ func (t *TestFrostFS) ReadObject(ctx context.Context, prm PrmObjectRead) (*Objec
 		}, nil
 	}
 
-	return nil, fmt.Errorf("object not found %s", addr)
+	return nil, fmt.Errorf("%w: %s", apistatus.ObjectNotFound{}, addr)
 }
 
 func (t *TestFrostFS) CreateObject(ctx context.Context, prm PrmObjectCreate) (oid.ID, error) {

api/layer/layer.go

@@ -561,21 +561,38 @@ func (n *layer) deleteObject(ctx context.Context, bkt *data.BucketInfo, settings
 		return obj
 	}
 
-	var newVersion *data.NodeVersion
+	lastVersion, err := n.getLastNodeVersion(ctx, bkt, obj)
+	if err != nil {
+		obj.Error = err
+		return n.handleNotFoundError(bkt, obj)
+	}
 
 	if settings.VersioningSuspended() {
 		obj.VersionID = data.UnversionedObjectVersionID
 
-		var nodeVersion *data.NodeVersion
+		var nullVersionToDelete *data.NodeVersion
-		if nodeVersion, obj.Error = n.getNodeVersionToDelete(ctx, bkt, obj); obj.Error != nil {
+		if lastVersion.IsUnversioned {
-			return n.handleNotFoundError(bkt, obj)
+			if !lastVersion.IsDeleteMarker() {
+				nullVersionToDelete = lastVersion
+			}
+		} else if nullVersionToDelete, obj.Error = n.getNodeVersionToDelete(ctx, bkt, obj); obj.Error != nil {
+			if !isNotFoundError(obj.Error) {
+				return obj
+			}
 		}
 
-		if obj.DeleteMarkVersion, obj.Error = n.removeOldVersion(ctx, bkt, nodeVersion, obj); obj.Error != nil {
+		if nullVersionToDelete != nil {
-			return obj
+			if obj.DeleteMarkVersion, obj.Error = n.removeOldVersion(ctx, bkt, nullVersionToDelete, obj); obj.Error != nil {
+				return obj
+			}
 		}
 	}
 
+	if lastVersion.IsDeleteMarker() {
+		obj.DeleteMarkVersion = lastVersion.OID.EncodeToString()
+		return obj
+	}
+
 	randOID, err := getRandomOID()
 	if err != nil {
 		obj.Error = fmt.Errorf("couldn't get random oid: %w", err)
@@ -584,7 +601,7 @@ func (n *layer) deleteObject(ctx context.Context, bkt *data.BucketInfo, settings
 
 	obj.DeleteMarkVersion = randOID.EncodeToString()
 
-	newVersion = &data.NodeVersion{
+	newVersion := &data.NodeVersion{
 		BaseNodeVersion: data.BaseNodeVersion{
 			OID:      randOID,
 			FilePath: obj.Name,
@@ -606,8 +623,7 @@ func (n *layer) deleteObject(ctx context.Context, bkt *data.BucketInfo, settings
 }
 
 func (n *layer) handleNotFoundError(bkt *data.BucketInfo, obj *VersionedObject) *VersionedObject {
-	if errors.IsS3Error(obj.Error, errors.ErrNoSuchKey) ||
+	if isNotFoundError(obj.Error) {
-		errors.IsS3Error(obj.Error, errors.ErrNoSuchVersion) {
 		obj.Error = nil
 		n.cache.CleanListCacheEntriesContainingObject(obj.Name, bkt.CID)
 		n.cache.DeleteObjectName(bkt.CID, bkt.Name, obj.Name)
@@ -616,6 +632,11 @@ func (n *layer) handleNotFoundError(bkt *data.BucketInfo, obj *VersionedObject)
 	return obj
 }
 
+func isNotFoundError(err error) bool {
+	return errors.IsS3Error(err, errors.ErrNoSuchKey) ||
+		errors.IsS3Error(err, errors.ErrNoSuchVersion)
+}
+
 func (n *layer) getNodeVersionToDelete(ctx context.Context, bkt *data.BucketInfo, obj *VersionedObject) (*data.NodeVersion, error) {
 	objVersion := &ObjectVersion{
 		BktInfo:               bkt,
@@ -627,6 +648,17 @@ func (n *layer) getNodeVersionToDelete(ctx context.Context, bkt *data.BucketInfo
 	return n.getNodeVersion(ctx, objVersion)
 }
 
+func (n *layer) getLastNodeVersion(ctx context.Context, bkt *data.BucketInfo, obj *VersionedObject) (*data.NodeVersion, error) {
+	objVersion := &ObjectVersion{
+		BktInfo:               bkt,
+		ObjectName:            obj.Name,
+		VersionID:             "",
+		NoErrorOnDeleteMarker: true,
+	}
+
+	return n.getNodeVersion(ctx, objVersion)
+}
+
 func (n *layer) removeOldVersion(ctx context.Context, bkt *data.BucketInfo, nodeVersion *data.NodeVersion, obj *VersionedObject) (string, error) {
 	if nodeVersion.IsDeleteMarker() {
 		return obj.VersionID, nil

cmd/s3-gw/app.go

@@ -113,14 +113,14 @@ func (a *App) init(ctx context.Context) {
 func (a *App) initLayer(ctx context.Context) {
 	a.initResolver()
 
-	treeServiceEndpoint := a.cfg.GetString(cfgTreeServiceEndpoint)
+	treeServiceEndpoint := a.cfg.GetStringSlice(cfgTreeServiceEndpoint)
 	grpcDialOpt := grpc.WithTransportCredentials(insecure.NewCredentials())
-	treeGRPCClient, err := services.NewTreeServiceClientGRPC(ctx, treeServiceEndpoint, a.key, grpcDialOpt)
+	treeGRPCClient, err := services.NewTreeServiceClientGRPC(ctx, treeServiceEndpoint, a.key, a.log, grpcDialOpt)
 	if err != nil {
 		a.log.Fatal("failed to create tree service", zap.Error(err))
 	}
-	treeService := tree.NewTree(treeGRPCClient)
+	treeService := tree.NewTree(treeGRPCClient, a.log)
-	a.log.Info("init tree service", zap.String("endpoint", treeServiceEndpoint))
+	a.log.Info("init tree service", zap.Strings("endpoints", treeGRPCClient.Endpoints()))
 
 	// prepare random key for anonymous requests
 	randomKey, err := keys.NewPrivateKey()
@@ -182,6 +182,7 @@ func (a *App) initAPI(ctx context.Context) {
 
 func (a *App) initMetrics() {
 	a.metrics = metrics.NewAppMetrics(a.log, frostfs.NewPoolStatistic(a.pool), a.cfg.GetBool(cfgPrometheusEnabled))
+	a.metrics.SetHealth(metrics.HealthStatusStarting)
 }
 
 func (a *App) initResolver() {
@@ -369,7 +370,7 @@ func (a *App) Wait() {
 }
 
 func (a *App) setHealthStatus() {
-	a.metrics.SetHealth(1)
+	a.metrics.SetHealth(metrics.HealthStatusReady)
 }
 
 // Serve runs HTTP server to handle S3 API requests.

config/config.env

@@ -42,8 +42,8 @@ S3_GW_CONFIG=/path/to/config/yaml
 # Logger
 S3_GW_LOGGER_LEVEL=debug
 
-# Endpoint of the tree service. Must be provided. Can be one of the node address (from the `peers` section).
+# Endpoints of the tree service. At least one endpoint must be provided. Node addresses (from the `peers` section) can be used.
-S3_GW_TREE_SERVICE=grpc://s01.frostfs.devenv:8080
+S3_GW_TREE_SERVICE=grpc://s01.frostfs.devenv:8080 grpc://s02.frostfs.devenv:8080
 
 # RPC endpoint and order of resolving of bucket names
 S3_GW_RPC_ENDPOINT=http://morph-chain.frostfs.devenv:30333/

config/config.yaml

@@ -44,9 +44,11 @@ listen_domains:
 logger:
   level: debug
 
-# Endpoint of the tree service. Must be provided. Can be one of the node address (from the `peers` section).
+# Endpoints of the tree service. At least one endpoint must be provided. Node addresses (from the `peers` section) can be used.
 tree:
-  service: node1.frostfs:8080
+  service:
+    - node1.frostfs:8080
+    - node2.frostfs:8080
 
 # RPC endpoint and order of resolving of bucket names
 rpc_endpoint: http://morph-chain.frostfs.devenv:30333

docs/configuration.md

@@ -337,14 +337,23 @@ logger:
 
 ### `tree` section
 
+```yaml
+tree:
+  service:
+   - s01.frostfs.devenv:8080
+   - s02.frostfs.devenv:8080
+```
+
+If you use only one endpoint, it can be provided as:
+
 ```yaml
 tree:
   service: s01.frostfs.devenv:8080
 ```
 
-| Parameter | Type     | Default value | Description                                                                                                |
+| Parameter | Type       | Default value | Description                                                                                                                   |
-|-----------|----------|---------------|------------------------------------------------------------------------------------------------------------|
+|-----------|------------|---------------|-------------------------------------------------------------------------------------------------------------------------------|
-| `service` | `string` |               | Endpoint of the tree service. Must be provided. Can be one of the node address (from the `peers` section). |
+| `service` | `[]string` |               | Endpoints of the tree service. At least one endpoint must be provided. Node addresses (from the `peers` section) can be used. |
 
 ### `cache` section
 

go.mod

@@ -3,7 +3,7 @@ module git.frostfs.info/TrueCloudLab/frostfs-s3-gw
 go 1.18
 
 require (
-	git.frostfs.info/TrueCloudLab/frostfs-api-go/v2 v2.11.2-0.20230315095236-9dc375346703
+	git.frostfs.info/TrueCloudLab/frostfs-api-go/v2 v2.15.1-0.20230602142024-4cb0068ddef0
 	git.frostfs.info/TrueCloudLab/frostfs-crypto v0.6.0
 	git.frostfs.info/TrueCloudLab/frostfs-sdk-go v0.0.0-20230329125804-552219b8e130
 	github.com/aws/aws-sdk-go v1.44.6
@@ -17,11 +17,11 @@ require (
 	github.com/prometheus/client_golang v1.13.0
 	github.com/spf13/pflag v1.0.5
 	github.com/spf13/viper v1.15.0
-	github.com/stretchr/testify v1.8.1
+	github.com/stretchr/testify v1.8.2
 	github.com/urfave/cli/v2 v2.3.0
 	go.uber.org/zap v1.24.0
 	golang.org/x/crypto v0.4.0
-	google.golang.org/grpc v1.52.0
+	google.golang.org/grpc v1.53.0
 	google.golang.org/protobuf v1.28.1
 )
 
@@ -31,15 +31,20 @@ require (
 	git.frostfs.info/TrueCloudLab/rfc6979 v0.4.0 // indirect
 	git.frostfs.info/TrueCloudLab/tzhash v1.8.0 // indirect
 	github.com/antlr/antlr4/runtime/Go/antlr/v4 v4.0.0-20221202181307-76fa05c21b12 // indirect
+	github.com/benbjohnson/clock v1.1.0 // indirect
 	github.com/beorn7/perks v1.0.1 // indirect
-	github.com/cespare/xxhash/v2 v2.1.2 // indirect
+	github.com/cenkalti/backoff/v4 v4.2.0 // indirect
+	github.com/cespare/xxhash/v2 v2.2.0 // indirect
 	github.com/cpuguy83/go-md2man/v2 v2.0.2 // indirect
 	github.com/davecgh/go-spew v1.1.1 // indirect
 	github.com/decred/dcrd/dcrec/secp256k1/v4 v4.0.1 // indirect
 	github.com/fsnotify/fsnotify v1.6.0 // indirect
+	github.com/go-logr/logr v1.2.3 // indirect
+	github.com/go-logr/stdr v1.2.2 // indirect
 	github.com/golang/protobuf v1.5.2 // indirect
 	github.com/golang/snappy v0.0.3 // indirect
 	github.com/gorilla/websocket v1.4.2 // indirect
+	github.com/grpc-ecosystem/grpc-gateway/v2 v2.7.0 // indirect
 	github.com/hashicorp/golang-lru v0.6.0 // indirect
 	github.com/hashicorp/golang-lru/v2 v2.0.1 // indirect
 	github.com/hashicorp/hcl v1.0.0 // indirect
@@ -66,15 +71,23 @@ require (
 	github.com/spf13/jwalterweatherman v1.1.0 // indirect
 	github.com/subosito/gotenv v1.4.2 // indirect
 	github.com/urfave/cli v1.22.5 // indirect
+	go.opentelemetry.io/otel v1.14.0 // indirect
+	go.opentelemetry.io/otel/exporters/otlp/internal/retry v1.14.0 // indirect
+	go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.14.0 // indirect
+	go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.14.0 // indirect
+	go.opentelemetry.io/otel/exporters/stdout/stdouttrace v1.14.0 // indirect
+	go.opentelemetry.io/otel/sdk v1.14.0 // indirect
+	go.opentelemetry.io/otel/trace v1.14.0 // indirect
+	go.opentelemetry.io/proto/otlp v0.19.0 // indirect
 	go.uber.org/atomic v1.10.0 // indirect
 	go.uber.org/multierr v1.9.0 // indirect
 	golang.org/x/exp v0.0.0-20221227203929-1b447090c38c // indirect
-	golang.org/x/net v0.4.0 // indirect
+	golang.org/x/net v0.7.0 // indirect
 	golang.org/x/sync v0.1.0 // indirect
-	golang.org/x/sys v0.3.0 // indirect
+	golang.org/x/sys v0.5.0 // indirect
-	golang.org/x/term v0.3.0 // indirect
+	golang.org/x/term v0.5.0 // indirect
-	golang.org/x/text v0.5.0 // indirect
+	golang.org/x/text v0.7.0 // indirect
-	google.golang.org/genproto v0.0.0-20221227171554-f9683d7f8bef // indirect
+	google.golang.org/genproto v0.0.0-20230110181048-76db0878b65f // indirect
 	gopkg.in/ini.v1 v1.67.0 // indirect
 	gopkg.in/yaml.v3 v3.0.1 // indirect
 )

go.sum

@@ -36,8 +36,8 @@ cloud.google.com/go/storage v1.8.0/go.mod h1:Wv1Oy7z6Yz3DshWRJFhqM/UCfaWIRTdp0RX
 cloud.google.com/go/storage v1.10.0/go.mod h1:FLPqc6j+Ki4BU591ie1oL6qBQGu2Bl/tZ9ullr3+Kg0=
 cloud.google.com/go/storage v1.14.0/go.mod h1:GrKmX003DSIwi9o29oFT7YDnHYwZoctc3fOKtUw0Xmo=
 dmitri.shuralyov.com/gpu/mtl v0.0.0-20190408044501-666a987793e9/go.mod h1:H6x//7gZCb22OMCxBHrMx7a5I7Hp++hsVxbQ4BYO7hU=
-git.frostfs.info/TrueCloudLab/frostfs-api-go/v2 v2.11.2-0.20230315095236-9dc375346703 h1:lxe0DtZq/uFZVZu9apx6OcIXCJskQBMd/GVeYGKA3wA=
+git.frostfs.info/TrueCloudLab/frostfs-api-go/v2 v2.15.1-0.20230602142024-4cb0068ddef0 h1:2rgsz9KdOf2obbL/FRCZNUZja83kQZxPuF8IKBDl/bY=
-git.frostfs.info/TrueCloudLab/frostfs-api-go/v2 v2.11.2-0.20230315095236-9dc375346703/go.mod h1:gRd5iE5A84viily6AcNBsSlTx2XgoWrwRDz7z0MayDQ=
+git.frostfs.info/TrueCloudLab/frostfs-api-go/v2 v2.15.1-0.20230602142024-4cb0068ddef0/go.mod h1:sPyITTmQT662ZI38ud2aoE1SUCAr1mO5xV8P4nzLkKI=
 git.frostfs.info/TrueCloudLab/frostfs-contract v0.0.0-20230307110621-19a8ef2d02fb h1:S/TrbOOu9qEXZRZ9/Ddw7crnxbBUQLo68PSzQWYrc9M=
 git.frostfs.info/TrueCloudLab/frostfs-contract v0.0.0-20230307110621-19a8ef2d02fb/go.mod h1:nkR5gaGeez3Zv2SE7aceP0YwxG2FzIB5cGKpQO2vV2o=
 git.frostfs.info/TrueCloudLab/frostfs-crypto v0.6.0 h1:FxqFDhQYYgpe41qsIHVOcdzSVCB8JNSfPG7Uk4r2oSk=
@@ -56,6 +56,7 @@ github.com/CityOfZion/neo-go v0.62.1-pre.0.20191114145240-e740fbe708f8/go.mod h1
 github.com/CityOfZion/neo-go v0.70.1-pre.0.20191209120015-fccb0085941e/go.mod h1:0enZl0az8xA6PVkwzEOwPWVJGqlt/GO4hA4kmQ5Xzig=
 github.com/CityOfZion/neo-go v0.70.1-pre.0.20191212173117-32ac01130d4c/go.mod h1:JtlHfeqLywZLswKIKFnAp+yzezY4Dji9qlfQKB2OD/I=
 github.com/CityOfZion/neo-go v0.71.1-pre.0.20200129171427-f773ec69fb84/go.mod h1:FLI526IrRWHmcsO+mHsCbj64pJZhwQFTLJZu+A4PGOA=
+github.com/OneOfOne/xxhash v1.2.2/go.mod h1:HSdplMjZKSmBqAxg5vPj2TmRDmfkzw+cTzAElWljhcU=
 github.com/Workiva/go-datastructures v1.0.50/go.mod h1:Z+F2Rca0qCsVYDS8z7bAGm8f3UkzuWYS/oBZz5a7VVA=
 github.com/abiosoft/ishell v2.0.0+incompatible/go.mod h1:HQR9AqF2R3P4XXpMpI0NAzgHf/aS6+zVXRj14cVk9qg=
 github.com/abiosoft/ishell/v2 v2.0.2/go.mod h1:E4oTCXfo6QjoCart0QYa5m9w4S+deXs/P/9jA77A9Bs=
@@ -94,11 +95,15 @@ github.com/btcsuite/snappy-go v0.0.0-20151229074030-0bdef8d06723/go.mod h1:8woku
 github.com/btcsuite/snappy-go v1.0.0/go.mod h1:8woku9dyThutzjeg+3xrA5iCpBRH8XEEg3lh6TiUghc=
 github.com/btcsuite/websocket v0.0.0-20150119174127-31079b680792/go.mod h1:ghJtEyQwv5/p4Mg4C0fgbePVuGr935/5ddU9Z3TmDRY=
 github.com/btcsuite/winsvc v1.0.0/go.mod h1:jsenWakMcC0zFBFurPLEAyrnc/teJEM1O46fmI40EZs=
+github.com/cenkalti/backoff/v4 v4.2.0 h1:HN5dHm3WBOgndBH6E8V0q2jIYIR3s9yglV8k/+MN3u4=
+github.com/cenkalti/backoff/v4 v4.2.0/go.mod h1:Y3VNntkOUPxTVeUxJ/G5vcM//AlwfmyYozVcomhLiZE=
 github.com/census-instrumentation/opencensus-proto v0.2.1/go.mod h1:f6KPmirojxKA12rnyqOA5BBL4O983OfeGPqjHWSTneU=
+github.com/cespare/xxhash v1.1.0/go.mod h1:XrSqR1VqqWfGrhpAt58auRo0WTKS1nRRg3ghfAqPWnc=
 github.com/cespare/xxhash/v2 v2.1.0/go.mod h1:dgIUBU3pDso/gPgZ1osOZ0iQf77oPR28Tjxl5dIMyVM=
 github.com/cespare/xxhash/v2 v2.1.1/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
-github.com/cespare/xxhash/v2 v2.1.2 h1:YRXhKfTDauu4ajMg1TPgFO5jnlC2HCbmLXMcTG5cbYE=
 github.com/cespare/xxhash/v2 v2.1.2/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
+github.com/cespare/xxhash/v2 v2.2.0 h1:DC2CZ1Ep5Y4k3ZQ899DldepgrayRUGE6BBZ/cd9Cj44=
+github.com/cespare/xxhash/v2 v2.2.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
 github.com/chzyer/logex v1.1.10/go.mod h1:+Ywpsq7O8HXn0nuIou7OrIPyXbp3wmkHB+jjWRnGsAI=
 github.com/chzyer/readline v0.0.0-20180603132655-2972be24d48e/go.mod h1:nSuG5e5PlCu98SY8svDHJxuZscDgtXS6KTTbou5AhLI=
 github.com/chzyer/test v0.0.0-20180213035817-a1ea475d72b1/go.mod h1:Q3SI9o4m/ZMnBNeIyt5eFwwo7qiLfzFZmjNmxjkiQlU=
@@ -106,7 +111,11 @@ github.com/client9/misspell v0.3.4/go.mod h1:qj6jICC3Q7zFZvVWo7KLAzC3yx5G7kyvSDk
 github.com/cncf/udpa/go v0.0.0-20191209042840-269d4d468f6f/go.mod h1:M8M6+tZqaGXZJjfX53e64911xZQV5JYwmTeXPW+k8Sc=
 github.com/cncf/udpa/go v0.0.0-20200629203442-efcf912fb354/go.mod h1:WmhPx2Nbnhtbo57+VJT5O0JRkEi1Wbu0z5j0R8u5Hbk=
 github.com/cncf/udpa/go v0.0.0-20201120205902-5459f2c99403/go.mod h1:WmhPx2Nbnhtbo57+VJT5O0JRkEi1Wbu0z5j0R8u5Hbk=
+github.com/cncf/udpa/go v0.0.0-20210930031921-04548b0d99d4/go.mod h1:6pvJx4me5XPnfI9Z40ddWsdw2W/uZgQLFXToKeRcDiI=
+github.com/cncf/xds/go v0.0.0-20210312221358-fbca930ec8ed/go.mod h1:eXthEFrGJvWHgFFCl3hGmgk+/aYT6PnTQLykKQRLhEs=
 github.com/cncf/xds/go v0.0.0-20210805033703-aa0b78936158/go.mod h1:eXthEFrGJvWHgFFCl3hGmgk+/aYT6PnTQLykKQRLhEs=
+github.com/cncf/xds/go v0.0.0-20210922020428-25de7278fc84/go.mod h1:eXthEFrGJvWHgFFCl3hGmgk+/aYT6PnTQLykKQRLhEs=
+github.com/cncf/xds/go v0.0.0-20211011173535-cb28da3451f1/go.mod h1:eXthEFrGJvWHgFFCl3hGmgk+/aYT6PnTQLykKQRLhEs=
 github.com/coreos/go-semver v0.3.0/go.mod h1:nnelYz7RCh+5ahJtPPxZlU+153eP4D4r3EedlOD2RNk=
 github.com/cpuguy83/go-md2man/v2 v2.0.0-20190314233015-f79a8a8ca69d/go.mod h1:maD7wRr/U5Z6m/iR4s+kqSMx2CaBsrgA7czyZG/E6dU=
 github.com/cpuguy83/go-md2man/v2 v2.0.2 h1:p1EgwI/C7NhT0JmVkwCD2ZBK8j4aeHQX2pMHHBfMQ6w=
@@ -125,6 +134,7 @@ github.com/envoyproxy/go-control-plane v0.9.1-0.20191026205805-5f8ba28d4473/go.m
 github.com/envoyproxy/go-control-plane v0.9.4/go.mod h1:6rpuAdCZL397s3pYoYcLgu1mIlRU8Am5FuJP05cCM98=
 github.com/envoyproxy/go-control-plane v0.9.7/go.mod h1:cwu0lG7PUMfa9snN8LXBig5ynNVH9qI8YYLbd1fK2po=
 github.com/envoyproxy/go-control-plane v0.9.9-0.20201210154907-fd9021fe5dad/go.mod h1:cXg6YxExXjJnVBQHBLXeUAgxn2UodCpnH306RInaBQk=
+github.com/envoyproxy/go-control-plane v0.9.9-0.20210512163311-63b5d3c536b0/go.mod h1:hliV/p42l8fGbc6Y9bQ70uLwIvmJyVE5k4iMKlh8wCQ=
 github.com/envoyproxy/go-control-plane v0.9.10-0.20210907150352-cf90f659a021/go.mod h1:AFq3mo9L8Lqqiid3OhADV3RfLJnjiw63cSpi+fDTRC0=
 github.com/envoyproxy/protoc-gen-validate v0.1.0/go.mod h1:iSmxcyjqTsJpI2R4NaDN7+kN2VEUnK/pcBlmesArF7c=
 github.com/etcd-io/bbolt v1.3.3/go.mod h1:ZF2nL25h33cCyBtcyWeZ2/I3HQOfTP+0PIEvHjkjCrw=
@@ -149,11 +159,18 @@ github.com/go-logfmt/logfmt v0.3.0/go.mod h1:Qt1PoO58o5twSAckw1HlFXLmHsOX5/0LbT9
 github.com/go-logfmt/logfmt v0.4.0/go.mod h1:3RMwSq7FuexP4Kalkev3ejPJsZTpXXBr9+V4qmtdjCk=
 github.com/go-logfmt/logfmt v0.5.0/go.mod h1:wCYkCAKZfumFQihp8CzCvQ3paCTfi41vtzG1KdI/P7A=
 github.com/go-logfmt/logfmt v0.5.1/go.mod h1:WYhtIu8zTZfxdn5+rREduYbwxfcBr/Vr6KEVveWlfTs=
+github.com/go-logr/logr v1.2.2/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbVz/1A=
+github.com/go-logr/logr v1.2.3 h1:2DntVwHkVopvECVRSlL5PSo9eG+cAkDCuckLubN+rq0=
+github.com/go-logr/logr v1.2.3/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbVz/1A=
+github.com/go-logr/stdr v1.2.2 h1:hSWxHoqTgW2S2qGc0LTAI563KZ5YKYRhT3MFKZMbjag=
+github.com/go-logr/stdr v1.2.2/go.mod h1:mMo/vtBO5dYbehREoey6XUKy/eSumjCCveDpRre4VKE=
 github.com/go-redis/redis v6.10.2+incompatible/go.mod h1:NAIEuMOZ/fxfXJIrKDQDz8wamY7mA7PouImQ2Jvg6kA=
 github.com/go-stack/stack v1.8.0/go.mod h1:v0f6uXyyMGvRgIKkXu+yp6POWl0qKG85gN/melR3HDY=
 github.com/go-yaml/yaml v2.1.0+incompatible/go.mod h1:w2MrLa16VYP0jy6N7M5kHaCkaLENm+P+Tv+MfurjSw0=
 github.com/gogo/protobuf v1.1.1/go.mod h1:r8qH/GZQm5c6nD/R0oafs1akxWv10x8SbQlK7atdtwQ=
 github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q=
+github.com/golang/glog v1.0.0 h1:nfP3RFugxnNRyKgeWd4oI1nYvXpxrx8ck8ZrcizshdQ=
+github.com/golang/glog v1.0.0/go.mod h1:EWib/APOK0SL3dFbYqvxE3UYd8E6s1ouQ7iEp/0LWV4=
 github.com/golang/groupcache v0.0.0-20190702054246-869f871628b6/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
 github.com/golang/groupcache v0.0.0-20191227052852-215e87163ea7/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
 github.com/golang/groupcache v0.0.0-20200121045136-8c9f03a8e57e/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
@@ -229,6 +246,8 @@ github.com/gorilla/mux v1.8.0/go.mod h1:DVbg23sWSpFRCP0SfiEN6jmj59UnW/n46BH5rLB7
 github.com/gorilla/websocket v1.4.2 h1:+/TMaTYc4QFitKJxsQ7Yye35DkWvkdLcvGKqM+x0Ufc=
 github.com/gorilla/websocket v1.4.2/go.mod h1:YR8l580nyteQvAITg2hZ9XVh4b55+EU/adAjf1fMHhE=
 github.com/grpc-ecosystem/grpc-gateway v1.16.0/go.mod h1:BDjrQk3hbvj6Nolgz8mAMFbcEtjT1g+wF4CSlocrBnw=
+github.com/grpc-ecosystem/grpc-gateway/v2 v2.7.0 h1:BZHcxBETFHIdVyhyEfOvn/RdU/QGdLI4y34qQGjGWO0=
+github.com/grpc-ecosystem/grpc-gateway/v2 v2.7.0/go.mod h1:hgWBS7lorOAVIJEQMi4ZsPv9hVvWI6+ch50m39Pf2Ks=
 github.com/hashicorp/golang-lru v0.5.0/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8=
 github.com/hashicorp/golang-lru v0.5.1/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8=
 github.com/hashicorp/golang-lru v0.5.4/go.mod h1:iADmTwqILo4mZ8BN3D2Q6+9jd8WM5uGBxy+E8yxSoD4=
@@ -400,6 +419,7 @@ github.com/shurcooL/sanitized_anchor_name v1.0.0/go.mod h1:1NzhyTcUVG4SuEtjjoZeV
 github.com/sirupsen/logrus v1.2.0/go.mod h1:LxeOpSwHxABJmUn/MG1IvRgCAasNZTLOkJPxbbu5VWo=
 github.com/sirupsen/logrus v1.4.2/go.mod h1:tLMulIdttU9McNUspp0xgXVQah82FyeX6MwdIuYE2rE=
 github.com/sirupsen/logrus v1.6.0/go.mod h1:7uNnSEd1DgxDLC74fIahvMZmmYsHGZGEOFrfsX/uA88=
+github.com/spaolacci/murmur3 v0.0.0-20180118202830-f09979ecbc72/go.mod h1:JwIasOWyU6f++ZhiEuf87xNszmSA2myDM2Kzu9HwQUA=
 github.com/spaolacci/murmur3 v1.1.0 h1:7c1g84S4BPRrfL5Xrdp6fOJ206sU9y293DDHaoy0bLI=
 github.com/spaolacci/murmur3 v1.1.0/go.mod h1:JwIasOWyU6f++ZhiEuf87xNszmSA2myDM2Kzu9HwQUA=
 github.com/spf13/afero v1.9.3 h1:41FoI0fD7OR7mGcKE/aOiLkGreyf8ifIOQmJANWogMk=
@@ -423,8 +443,9 @@ github.com/stretchr/testify v1.5.1/go.mod h1:5W2xD1RspED5o8YsWQXVCued0rvSQ+mT+I5
 github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
 github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
 github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU=
-github.com/stretchr/testify v1.8.1 h1:w7B6lhMri9wdJUVmEZPGGhZzrYTPvgJArz7wNPgYKsk=
 github.com/stretchr/testify v1.8.1/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4=
+github.com/stretchr/testify v1.8.2 h1:+h33VjcLVPDHtOdpUCuF+7gSuG3yGIftsP1YvFihtJ8=
+github.com/stretchr/testify v1.8.2/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4=
 github.com/subosito/gotenv v1.4.2 h1:X1TuBLAMDFbaTAChgCBLu3DU3UPyELpnF2jjJ2cz/S8=
 github.com/subosito/gotenv v1.4.2/go.mod h1:ayKnFf/c6rvx/2iiLrJUk1e6plDbT3edrFNGqEflhK0=
 github.com/syndtr/goleveldb v0.0.0-20180307113352-169b1b37be73/go.mod h1:Z4AUp2Km+PwemOoO/VB5AOx9XSsIItzFjoJlOSiYmn0=
@@ -454,14 +475,30 @@ go.opencensus.io v0.22.2/go.mod h1:yxeiOL68Rb0Xd1ddK5vPZ/oVn4vY4Ynel7k9FzqtOIw=
 go.opencensus.io v0.22.3/go.mod h1:yxeiOL68Rb0Xd1ddK5vPZ/oVn4vY4Ynel7k9FzqtOIw=
 go.opencensus.io v0.22.4/go.mod h1:yxeiOL68Rb0Xd1ddK5vPZ/oVn4vY4Ynel7k9FzqtOIw=
 go.opencensus.io v0.22.5/go.mod h1:5pWMHQbX5EPX2/62yrJeAkowc+lfs/XD7Uxpq3pI6kk=
+go.opentelemetry.io/otel v1.14.0 h1:/79Huy8wbf5DnIPhemGB+zEPVwnN6fuQybr/SRXa6hM=
+go.opentelemetry.io/otel v1.14.0/go.mod h1:o4buv+dJzx8rohcUeRmWUZhqupFvzWis188WlggnNeU=
+go.opentelemetry.io/otel/exporters/otlp/internal/retry v1.14.0 h1:/fXHZHGvro6MVqV34fJzDhi7sHGpX3Ej/Qjmfn003ho=
+go.opentelemetry.io/otel/exporters/otlp/internal/retry v1.14.0/go.mod h1:UFG7EBMRdXyFstOwH028U0sVf+AvukSGhF0g8+dmNG8=
+go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.14.0 h1:TKf2uAs2ueguzLaxOCBXNpHxfO/aC7PAdDsSH0IbeRQ=
+go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.14.0/go.mod h1:HrbCVv40OOLTABmOn1ZWty6CHXkU8DK/Urc43tHug70=
+go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.14.0 h1:ap+y8RXX3Mu9apKVtOkM6WSFESLM8K3wNQyOU8sWHcc=
+go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.14.0/go.mod h1:5w41DY6S9gZrbjuq6Y+753e96WfPha5IcsOSZTtullM=
+go.opentelemetry.io/otel/exporters/stdout/stdouttrace v1.14.0 h1:sEL90JjOO/4yhquXl5zTAkLLsZ5+MycAgX99SDsxGc8=
+go.opentelemetry.io/otel/exporters/stdout/stdouttrace v1.14.0/go.mod h1:oCslUcizYdpKYyS9e8srZEqM6BB8fq41VJBjLAE6z1w=
+go.opentelemetry.io/otel/sdk v1.14.0 h1:PDCppFRDq8A1jL9v6KMI6dYesaq+DFcDZvjsoGvxGzY=
+go.opentelemetry.io/otel/sdk v1.14.0/go.mod h1:bwIC5TjrNG6QDCHNWvW4HLHtUQ4I+VQDsnjhvyZCALM=
+go.opentelemetry.io/otel/trace v1.14.0 h1:wp2Mmvj41tDsyAJXiWDWpfNsOiIyd38fy85pyKcFq/M=
+go.opentelemetry.io/otel/trace v1.14.0/go.mod h1:8avnQLK+CG77yNLUae4ea2JDQ6iT+gozhnZjy/rw9G8=
 go.opentelemetry.io/proto/otlp v0.7.0/go.mod h1:PqfVotwruBrMGOCsRd/89rSnXhoiJIqeYNgFYFoEGnI=
+go.opentelemetry.io/proto/otlp v0.19.0 h1:IVN6GR+mhC4s5yfcTbmzHYODqvWAp3ZedA2SJPI1Nnw=
+go.opentelemetry.io/proto/otlp v0.19.0/go.mod h1:H7XAot3MsfNsj7EXtrA2q5xSNQ10UqI405h3+duxN4U=
 go.uber.org/atomic v1.4.0/go.mod h1:gD2HeocX3+yG+ygLZcrzQJaqmWj9AIm7n08wl/qW/PE=
 go.uber.org/atomic v1.7.0/go.mod h1:fEN4uk6kAWBTFdckzkM89CLk9XfWZrxpCo0nPH17wJc=
 go.uber.org/atomic v1.9.0/go.mod h1:fEN4uk6kAWBTFdckzkM89CLk9XfWZrxpCo0nPH17wJc=
 go.uber.org/atomic v1.10.0 h1:9qC72Qh0+3MqyJbAn8YU5xVq1frD8bn3JtD2oXtafVQ=
 go.uber.org/atomic v1.10.0/go.mod h1:LUxbIzbOniOlMKjJjyPfpl4v+PKK2cNJn91OQbhoJI0=
 go.uber.org/goleak v1.1.10/go.mod h1:8a7PlsEVH3e/a/GLqe5IIrQx6GzcnRmZEufDUTk4A7A=
-go.uber.org/goleak v1.1.11 h1:wy28qYRKZgnJTxGxvye5/wgWr1EKjmUDGYox5mGlRlI=
+go.uber.org/goleak v1.2.1 h1:NBol2c7O1ZokfZ0LEU9K6Whx/KnwvepVetCUhtKja4A=
 go.uber.org/multierr v1.1.0/go.mod h1:wR5kodmAFQ0UK8QlbwjlSNy0Z68gJhDJUG5sjR94q/0=
 go.uber.org/multierr v1.6.0/go.mod h1:cdWPpRnG4AhwMwsgIHip0KRBQjJy5kYEpYjJxpXp9iU=
 go.uber.org/multierr v1.9.0 h1:7fIwc/ZtS0q++VgcfqFDxSBZVv/Xo49/SYnDFupUwlI=
@@ -560,12 +597,13 @@ golang.org/x/net v0.0.0-20201031054903-ff519b6c9102/go.mod h1:sp8m0HH+o8qH0wwXwY
 golang.org/x/net v0.0.0-20201209123823-ac852fbbde11/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg=
 golang.org/x/net v0.0.0-20201224014010-6772e930b67b/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg=
 golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg=
+golang.org/x/net v0.0.0-20210405180319-a5a99cb37ef4/go.mod h1:p54w0d4576C0XHj96bSt6lcn1PtDYWL6XObtHCRCNQM=
 golang.org/x/net v0.0.0-20210525063256-abc453219eb5/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
 golang.org/x/net v0.0.0-20211015210444-4f30a5c0130f/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
 golang.org/x/net v0.0.0-20220127200216-cd36cc0744dd/go.mod h1:CfG3xpIq0wQ8r1q4Su4UZFWDARRcnwPjda9FqA0JpMk=
 golang.org/x/net v0.0.0-20220225172249-27dd8689420f/go.mod h1:CfG3xpIq0wQ8r1q4Su4UZFWDARRcnwPjda9FqA0JpMk=
-golang.org/x/net v0.4.0 h1:Q5QPcMlvfxFTAPV0+07Xz/MpK9NTXu2VDUuy0FeMfaU=
+golang.org/x/net v0.7.0 h1:rJrUqqhjsgNp7KqAIc25s9pZnjU7TUcSY7HcVZjdn1g=
-golang.org/x/net v0.4.0/go.mod h1:MBQ8lrhLObU/6UmLb4fmbmk5OcyYmqtbGd/9yIeKjEE=
+golang.org/x/net v0.7.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs=
 golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
 golang.org/x/oauth2 v0.0.0-20190226205417-e64efc72b421/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
 golang.org/x/oauth2 v0.0.0-20190604053449-0f29369cfe45/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
@@ -576,6 +614,7 @@ golang.org/x/oauth2 v0.0.0-20201109201403-9fd604954f58/go.mod h1:KelEdhl1UZF7XfJ
 golang.org/x/oauth2 v0.0.0-20201208152858-08078c50e5b5/go.mod h1:KelEdhl1UZF7XfJ4dDtk6s++YSgaE7mD/BuKKDLBl4A=
 golang.org/x/oauth2 v0.0.0-20210218202405-ba52d332ba99/go.mod h1:KelEdhl1UZF7XfJ4dDtk6s++YSgaE7mD/BuKKDLBl4A=
 golang.org/x/oauth2 v0.0.0-20210514164344-f6687ab2804c/go.mod h1:KelEdhl1UZF7XfJ4dDtk6s++YSgaE7mD/BuKKDLBl4A=
+golang.org/x/oauth2 v0.0.0-20211104180415-d3ed0bb246c8/go.mod h1:KelEdhl1UZF7XfJ4dDtk6s++YSgaE7mD/BuKKDLBl4A=
 golang.org/x/oauth2 v0.0.0-20220223155221-ee480838109b/go.mod h1:DAh4E804XQdzx2j+YRIaUnCqCV2RuMz24cGBJ5QYIrc=
 golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
 golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
@@ -643,8 +682,10 @@ golang.org/x/sys v0.0.0-20210104204734-6f8348627aad/go.mod h1:h1NjWce9XRLGQEsW7w
 golang.org/x/sys v0.0.0-20210119212857-b64e53b001e4/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20210124154548-22da62e12c0c/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20210225134936-a50acf3fe073/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20210330210617-4fbd30eecc44/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20210423082822-04245dca01da/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20210423185535-09eb48e85fd7/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20210510120138-977fb7262007/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.0.0-20210603081109-ebe580a85c40/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.0.0-20210615035016-665e8c7367d1/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.0.0-20211019181941-9d821ace8654/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
@@ -652,23 +693,24 @@ golang.org/x/sys v0.0.0-20211216021012-1d35b9e2eb4e/go.mod h1:oPkhp1MJrh7nUepCBc
 golang.org/x/sys v0.0.0-20220114195835-da31bd327af9/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.0.0-20220908164124-27713097b956/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
-golang.org/x/sys v0.3.0 h1:w8ZOecv6NaNa/zC8944JTU3vz4u6Lagfk4RPQxv92NQ=
+golang.org/x/sys v0.5.0 h1:MUK/U/4lj1t1oPg0HfuXDN/Z1wv31ZJ/YcPiGccS4DU=
-golang.org/x/sys v0.3.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
+golang.org/x/sys v0.5.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
 golang.org/x/term v0.0.0-20210429154555-c04ba851c2a4/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
 golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
-golang.org/x/term v0.3.0 h1:qoo4akIqOcDME5bhc/NgxUdovd6BSS2uMsVjB56q1xI=
+golang.org/x/term v0.5.0 h1:n2a8QNdAb0sZNpU9R1ALUXBbY+w51fCQDN+7EdxNBsY=
-golang.org/x/term v0.3.0/go.mod h1:q750SLmJuPmVoN1blW3UFBPREJfb1KmY3vwxfr+nFDA=
+golang.org/x/term v0.5.0/go.mod h1:jMB1sMXY+tzblOD4FWmEbocvup2/aLOaQEp7JmGp78k=
 golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
 golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
 golang.org/x/text v0.3.1-0.20180807135948-17ff2d5776d2/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
 golang.org/x/text v0.3.2/go.mod h1:bEr9sfX3Q8Zfm5fL9x+3itogRgK3+ptLWKqgva+5dAk=
 golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
 golang.org/x/text v0.3.4/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
+golang.org/x/text v0.3.5/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
 golang.org/x/text v0.3.6/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
 golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ=
-golang.org/x/text v0.5.0 h1:OLmvp0KP+FVG99Ct/qFiL/Fhk4zp4QQnZ7b2U+5piUM=
+golang.org/x/text v0.7.0 h1:4BRB4x83lYWy72KwLD/qYDuTu7q9PjSagHvijDw7cLo=
-golang.org/x/text v0.5.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8=
+golang.org/x/text v0.7.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8=
 golang.org/x/time v0.0.0-20181108054448-85acf8d2951c/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
 golang.org/x/time v0.0.0-20190308202827-9d24e82272b4/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
 golang.org/x/time v0.0.0-20191024005414-555d28b269f0/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
@@ -791,8 +833,9 @@ google.golang.org/genproto v0.0.0-20201210142538-e3217bee35cc/go.mod h1:FWY/as6D
 google.golang.org/genproto v0.0.0-20201214200347-8c77b98c765d/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no=
 google.golang.org/genproto v0.0.0-20210108203827-ffc7fda8c3d7/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no=
 google.golang.org/genproto v0.0.0-20210226172003-ab064af71705/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no=
-google.golang.org/genproto v0.0.0-20221227171554-f9683d7f8bef h1:uQ2vjV/sHTsWSqdKeLqmwitzgvjMl7o4IdtHwUDXSJY=
+google.golang.org/genproto v0.0.0-20211118181313-81c1377c94b1/go.mod h1:5CzLGKJ67TSI2B9POpiiyGha0AjJvZIUgRMt1dSmuhc=
-google.golang.org/genproto v0.0.0-20221227171554-f9683d7f8bef/go.mod h1:RGgjbofJ8xD9Sq1VVhDM1Vok1vRONV+rg+CjzG4SZKM=
+google.golang.org/genproto v0.0.0-20230110181048-76db0878b65f h1:BWUVssLB0HVOSY78gIdvk1dTVYtT1y8SBWtPYuTJ/6w=
+google.golang.org/genproto v0.0.0-20230110181048-76db0878b65f/go.mod h1:RGgjbofJ8xD9Sq1VVhDM1Vok1vRONV+rg+CjzG4SZKM=
 google.golang.org/grpc v1.19.0/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZiDr8c=
 google.golang.org/grpc v1.20.1/go.mod h1:10oTOabMzJvdu6/UiuZezV6QK5dSlG84ov/aaiqXj38=
 google.golang.org/grpc v1.21.1/go.mod h1:oYelfM1adQP15Ek0mdvEgi9Df8B9CZIaU1084ijfRaM=
@@ -811,9 +854,11 @@ google.golang.org/grpc v1.33.2/go.mod h1:JMHMWHQWaTccqQQlmk3MJZS+GWXOdAesneDmEnv
 google.golang.org/grpc v1.34.0/go.mod h1:WotjhfgOW/POjDeRt8vscBtXq+2VjORFy659qA51WJ8=
 google.golang.org/grpc v1.35.0/go.mod h1:qjiiYl8FncCW8feJPdyg3v6XW24KsRHe+dy9BAGRRjU=
 google.golang.org/grpc v1.36.0/go.mod h1:qjiiYl8FncCW8feJPdyg3v6XW24KsRHe+dy9BAGRRjU=
+google.golang.org/grpc v1.40.0/go.mod h1:ogyxbiOoUXAkP+4+xa6PZSE9DZgIHtSpzjDTB9KAK34=
 google.golang.org/grpc v1.41.0/go.mod h1:U3l9uK9J0sini8mHphKoXyaqDA/8VyGnDee1zzIUK6k=
-google.golang.org/grpc v1.52.0 h1:kd48UiU7EHsV4rnLyOJRuP/Il/UHE7gdDAQ+SZI7nZk=
+google.golang.org/grpc v1.42.0/go.mod h1:k+4IHHFw41K8+bbowsex27ge2rCb65oeWqe4jJ590SU=
-google.golang.org/grpc v1.52.0/go.mod h1:pu6fVzoFb+NBYNAvQL08ic+lvB2IojljRYuun5vorUY=
+google.golang.org/grpc v1.53.0 h1:LAv2ds7cmFV/XTS3XG1NneeENYrXGmorPxsBbptIjNc=
+google.golang.org/grpc v1.53.0/go.mod h1:OnIrk0ipVdj4N5d9IUoFUx72/VlD7+jUsHwZgwSMQpw=
 google.golang.org/protobuf v0.0.0-20200109180630-ec00e32a8dfd/go.mod h1:DFci5gLYBciE7Vtevhsrf46CRTquxDuWsQurQQe4oz8=
 google.golang.org/protobuf v0.0.0-20200221191635-4d8936d0db64/go.mod h1:kwYJMbMJ01Woi6D6+Kah6886xMZcty6N08ah7+eCXa0=
 google.golang.org/protobuf v0.0.0-20200228230310-ab0ca4ff8a60/go.mod h1:cfTl7dwQJ+fmap5saPgwCLgHXTUD7jkjRqWcaiX5VyM=

internal/frostfs/services/client/client.go

@@ -0,0 +1,70 @@
+package client
+
+import (
+	"context"
+	"fmt"
+	"sync"
+
+	grpcService "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/frostfs/services/tree"
+	"google.golang.org/grpc"
+)
+
+type TreeClient struct {
+	mu      sync.RWMutex
+	address string
+	opts    []grpc.DialOption
+	conn    *grpc.ClientConn
+	service grpcService.TreeServiceClient
+	dialed  bool
+}
+
+// NewTreeClient creates new tree client with auto dial.
+func NewTreeClient(addr string, opts ...grpc.DialOption) *TreeClient {
+	return &TreeClient{
+		address: addr,
+		opts:    opts,
+	}
+}
+
+func (c *TreeClient) dial(ctx context.Context) error {
+	c.mu.Lock()
+	defer c.mu.Unlock()
+
+	if c.dialed {
+		return fmt.Errorf("couldn't dial '%s': connection already established", c.address)
+	}
+
+	conn, err := grpc.Dial(c.address, c.opts...)
+	if err != nil {
+		return fmt.Errorf("grpc dial node tree service: %w", err)
+	}
+
+	serviceClient := grpcService.NewTreeServiceClient(conn)
+	if _, err = serviceClient.Healthcheck(ctx, &grpcService.HealthcheckRequest{}); err != nil {
+		return fmt.Errorf("healthcheck tree service: %w", err)
+	}
+
+	c.conn = conn
+	c.service = serviceClient
+	c.dialed = true
+
+	return nil
+}
+
+func (c *TreeClient) TreeClient(ctx context.Context) (grpcService.TreeServiceClient, error) {
+	c.mu.RLock()
+	dialed := c.dialed
+	c.mu.RUnlock()
+
+	if !dialed {
+		if err := c.dial(ctx); err != nil {
+			return nil, err
+		}
+	}
+
+	return c.service, nil
+}
+
+func (c *TreeClient) Address() string {
+	return c.address
+}

internal/frostfs/services/tree_client_grpc.go

@@ -6,14 +6,17 @@ import (
 	"fmt"
 	"io"
 	"strings"
+	"sync/atomic"
 
 	"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api"
 	"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data"
 	"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/creds/accessbox"
+	treeClient "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/frostfs/services/client"
 	grpcService "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/frostfs/services/tree"
 	"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/pkg/service/tree"
 	"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/bearer"
 	"github.com/nspcc-dev/neo-go/pkg/crypto/keys"
+	"go.uber.org/zap"
 	"google.golang.org/grpc"
 )
 
@@ -65,28 +68,61 @@ func (n GetSubTreeResponseBodyWrapper) GetMeta() []tree.Meta {
 	return res
 }
 
-type ServiceClientGRPC struct {
+type TreeClient interface {
-	key     *keys.PrivateKey
+	TreeClient(ctx context.Context) (grpcService.TreeServiceClient, error)
-	conn    *grpc.ClientConn
+	Address() string
-	service grpcService.TreeServiceClient
 }
 
-func NewTreeServiceClientGRPC(ctx context.Context, addr string, key *keys.PrivateKey, grpcOpts ...grpc.DialOption) (*ServiceClientGRPC, error) {
+type ServiceClientGRPC struct {
-	conn, err := grpc.Dial(addr, grpcOpts...)
+	key        *keys.PrivateKey
-	if err != nil {
+	log        *zap.Logger
-		return nil, fmt.Errorf("did not connect: %v", err)
+	clients    []TreeClient
+	startIndex int32
+}
+
+func (c *ServiceClientGRPC) getStartIndex() int {
+	return int(atomic.LoadInt32(&c.startIndex))
+}
+
+func (c *ServiceClientGRPC) setStartIndex(index int) {
+	atomic.StoreInt32(&c.startIndex, int32(index))
+}
+
+func (c *ServiceClientGRPC) Endpoints() []string {
+	res := make([]string, len(c.clients))
+	for i, client := range c.clients {
+		res[i] = client.Address()
+	}
+	return res
+}
+
+func NewTreeServiceClientGRPC(ctx context.Context, endpoints []string, key *keys.PrivateKey, log *zap.Logger, grpcOpts ...grpc.DialOption) (*ServiceClientGRPC, error) {
+	res := &ServiceClientGRPC{
+		key: key,
+		log: log,
 	}
 
-	c := grpcService.NewTreeServiceClient(conn)
+	firstHealthy := -1
-	if _, err = c.Healthcheck(ctx, &grpcService.HealthcheckRequest{}); err != nil {
+
-		return nil, fmt.Errorf("healthcheck: %w", err)
+	res.clients = make([]TreeClient, len(endpoints))
+	for i, addr := range endpoints {
+		res.clients[i] = treeClient.NewTreeClient(addr, grpcOpts...)
+		if _, err := res.clients[i].TreeClient(ctx); err != nil {
+			log.Warn("dial tree", zap.String("address", addr), zap.Error(err))
+			continue
+		}
+		if firstHealthy == -1 {
+			firstHealthy = i
+		}
 	}
 
-	return &ServiceClientGRPC{
+	if firstHealthy == -1 {
-		key:     key,
+		return nil, errors.New("no healthy tree grpc client")
-		conn:    conn,
+	}
-		service: c,
+
-	}, nil
+	res.setStartIndex(firstHealthy)
+
+	return res, nil
 }
 
 func (c *ServiceClientGRPC) GetNodes(ctx context.Context, p *tree.GetNodesParams) ([]tree.NodeResponse, error) {
@@ -112,9 +148,15 @@ func (c *ServiceClientGRPC) GetNodes(ctx context.Context, p *tree.GetNodesParams
 		return nil, err
 	}
 
-	resp, err := c.service.GetNodeByPath(ctx, request)
+	log := c.log.With(zap.String("request_id", api.GetRequestID(ctx)), zap.String("tree", p.TreeID),
-	if err != nil {
+		zap.String("method", "GetNodeByPath"))
-		return nil, handleError("failed to get node by path", err)
+
+	var resp *grpcService.GetNodeByPathResponse
+	if err := c.requestWithRetry(ctx, log, func(client grpcService.TreeServiceClient) (inErr error) {
+		resp, inErr = client.GetNodeByPath(ctx, request)
+		return handleError("failed to get node by path", inErr)
+	}); err != nil {
+		return nil, err
 	}
 
 	res := make([]tree.NodeResponse, len(resp.GetBody().GetNodes()))
@@ -145,9 +187,15 @@ func (c *ServiceClientGRPC) GetSubTree(ctx context.Context, bktInfo *data.Bucket
 		return nil, err
 	}
 
-	cli, err := c.service.GetSubTree(ctx, request)
+	log := c.log.With(zap.String("request_id", api.GetRequestID(ctx)), zap.String("tree", treeID),
-	if err != nil {
+		zap.String("method", "GetSubTree"))
-		return nil, handleError("failed to get sub tree client", err)
+
+	var cli grpcService.TreeService_GetSubTreeClient
+	if err := c.requestWithRetry(ctx, log, func(client grpcService.TreeServiceClient) (inErr error) {
+		cli, inErr = client.GetSubTree(ctx, request)
+		return handleError("failed to get sub tree client", inErr)
+	}); err != nil {
+		return nil, err
 	}
 
 	var subtree []tree.NodeResponse
@@ -183,9 +231,15 @@ func (c *ServiceClientGRPC) AddNode(ctx context.Context, bktInfo *data.BucketInf
 		return 0, err
 	}
 
-	resp, err := c.service.Add(ctx, request)
+	log := c.log.With(zap.String("request_id", api.GetRequestID(ctx)), zap.String("tree", treeID),
-	if err != nil {
+		zap.String("method", "Add"))
-		return 0, handleError("failed to add node", err)
+
+	var resp *grpcService.AddResponse
+	if err := c.requestWithRetry(ctx, log, func(client grpcService.TreeServiceClient) (inErr error) {
+		resp, inErr = client.Add(ctx, request)
+		return handleError("failed to add node", inErr)
+	}); err != nil {
+		return 0, err
 	}
 
 	return resp.GetBody().GetNodeId(), nil
@@ -212,9 +266,15 @@ func (c *ServiceClientGRPC) AddNodeByPath(ctx context.Context, bktInfo *data.Buc
 		return 0, err
 	}
 
-	resp, err := c.service.AddByPath(ctx, request)
+	log := c.log.With(zap.String("request_id", api.GetRequestID(ctx)), zap.String("tree", treeID),
-	if err != nil {
+		zap.String("method", "AddByPath"))
-		return 0, handleError("failed to add node by path", err)
+
+	var resp *grpcService.AddByPathResponse
+	if err := c.requestWithRetry(ctx, log, func(client grpcService.TreeServiceClient) (inErr error) {
+		resp, inErr = client.AddByPath(ctx, request)
+		return handleError("failed to add node by path", inErr)
+	}); err != nil {
+		return 0, err
 	}
 
 	body := resp.GetBody()
@@ -249,11 +309,15 @@ func (c *ServiceClientGRPC) MoveNode(ctx context.Context, bktInfo *data.BucketIn
 		return err
 	}
 
-	if _, err := c.service.Move(ctx, request); err != nil {
+	log := c.log.With(zap.String("request_id", api.GetRequestID(ctx)), zap.String("tree", treeID),
-		return handleError("failed to move node", err)
+		zap.String("method", "Move"))
-	}
 
-	return nil
+	return c.requestWithRetry(ctx, log, func(client grpcService.TreeServiceClient) error {
+		if _, err := client.Move(ctx, request); err != nil {
+			return handleError("failed to move node", err)
+		}
+		return nil
+	})
 }
 
 func (c *ServiceClientGRPC) RemoveNode(ctx context.Context, bktInfo *data.BucketInfo, treeID string, nodeID uint64) error {
@@ -274,11 +338,43 @@ func (c *ServiceClientGRPC) RemoveNode(ctx context.Context, bktInfo *data.Bucket
 		return err
 	}
 
-	if _, err := c.service.Remove(ctx, request); err != nil {
+	log := c.log.With(zap.String("request_id", api.GetRequestID(ctx)), zap.String("tree", treeID),
-		return handleError("failed to remove node", err)
+		zap.String("method", "Remove"))
+
+	return c.requestWithRetry(ctx, log, func(client grpcService.TreeServiceClient) error {
+		if _, err := client.Remove(ctx, request); err != nil {
+			return handleError("failed to remove node", err)
+		}
+		return nil
+	})
+}
+
+func (c *ServiceClientGRPC) requestWithRetry(ctx context.Context, log *zap.Logger, fn func(client grpcService.TreeServiceClient) error) error {
+	var (
+		err error
+		cl  grpcService.TreeServiceClient
+	)
+
+	start := c.getStartIndex()
+	for i := start; i < start+len(c.clients); i++ {
+		index := i % len(c.clients)
+		if cl, err = c.clients[index].TreeClient(ctx); err == nil {
+			err = fn(cl)
+		}
+		if !shouldTryAgain(err) {
+			c.setStartIndex(index)
+			return err
+		}
+		log.Debug("tree request error", zap.String("address", c.clients[index].Address()), zap.Error(err))
 	}
 
-	return nil
+	return err
+}
+
+func shouldTryAgain(err error) bool {
+	return !(err == nil ||
+		errors.Is(err, tree.ErrNodeNotFound) ||
+		errors.Is(err, tree.ErrNodeAccessDenied))
 }
 
 func metaToKV(meta map[string]string) []*grpcService.KeyValue {
@@ -303,6 +399,9 @@ func getBearer(ctx context.Context, bktInfo *data.BucketInfo) []byte {
 }
 
 func handleError(msg string, err error) error {
+	if err == nil {
+		return nil
+	}
 	if strings.Contains(err.Error(), "not found") {
 		return fmt.Errorf("%w: %s", tree.ErrNodeNotFound, err.Error())
 	} else if strings.Contains(err.Error(), "is denied by") {

internal/frostfs/services/tree_client_grpc_signature.go

@@ -7,8 +7,6 @@ import (
 )
 
 func (c *ServiceClientGRPC) signData(buf []byte, f func(key, sign []byte)) error {
-	// crypto package should not be used outside of API libraries (see neofs-node#491).
-	// For now tree service does not include into SDK Client nor SDK Pool, so there is no choice.
 	// When SDK library adopts Tree service client, this should be dropped.
 	sign, err := crypto.Sign(&c.key.PrivateKey, buf)
 	if err != nil {

internal/frostfs/services/tree_client_grpc_test.go

@@ -1,13 +1,32 @@
 package services
 
 import (
+	"context"
 	"errors"
 	"testing"
 
 	"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer"
+	grpcService "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/frostfs/services/tree"
 	"github.com/stretchr/testify/require"
+	"go.uber.org/zap/zaptest"
 )
 
+type treeClientMock struct {
+	address string
+	err     bool
+}
+
+func (t *treeClientMock) TreeClient(context.Context) (grpcService.TreeServiceClient, error) {
+	if t.err {
+		return nil, errors.New("error")
+	}
+	return nil, nil
+}
+
+func (t *treeClientMock) Address() string {
+	return t.address
+}
+
 func TestHandleError(t *testing.T) {
 	defaultError := errors.New("default error")
 	for _, tc := range []struct {
@@ -33,3 +52,78 @@ func TestHandleError(t *testing.T) {
 		})
 	}
 }
+
+func TestRetry(t *testing.T) {
+	ctx := context.Background()
+	log := zaptest.NewLogger(t)
+
+	cl := &ServiceClientGRPC{
+		log: zaptest.NewLogger(t),
+		clients: []TreeClient{
+			&treeClientMock{address: "node0"},
+			&treeClientMock{address: "node1"},
+			&treeClientMock{address: "node2"},
+			&treeClientMock{address: "node3"},
+		},
+	}
+
+	makeFn := func(client grpcService.TreeServiceClient) error {
+		return nil
+	}
+
+	t.Run("first ok", func(t *testing.T) {
+		err := cl.requestWithRetry(ctx, log, makeFn)
+		require.NoError(t, err)
+		require.Equal(t, 0, cl.getStartIndex())
+		resetClients(cl)
+	})
+
+	t.Run("first failed", func(t *testing.T) {
+		setErrors(cl.clients[:1])
+		err := cl.requestWithRetry(ctx, log, makeFn)
+		require.NoError(t, err)
+		require.Equal(t, 1, cl.getStartIndex())
+		resetClients(cl)
+	})
+
+	t.Run("all failed", func(t *testing.T) {
+		setErrors(cl.clients)
+		err := cl.requestWithRetry(ctx, log, makeFn)
+		require.Error(t, err)
+		require.Equal(t, 0, cl.getStartIndex())
+		resetClients(cl)
+	})
+
+	t.Run("round", func(t *testing.T) {
+		setErrors(cl.clients[:2])
+		err := cl.requestWithRetry(ctx, log, makeFn)
+		require.NoError(t, err)
+		require.Equal(t, 2, cl.getStartIndex())
+		resetClientsErrors(cl)
+
+		setErrors(cl.clients[2:])
+		err = cl.requestWithRetry(ctx, log, makeFn)
+		require.NoError(t, err)
+		require.Equal(t, 0, cl.getStartIndex())
+		resetClients(cl)
+	})
+}
+
+func resetClients(cl *ServiceClientGRPC) {
+	resetClientsErrors(cl)
+	cl.setStartIndex(0)
+}
+
+func resetClientsErrors(cl *ServiceClientGRPC) {
+	for _, client := range cl.clients {
+		node := client.(*treeClientMock)
+		node.err = false
+	}
+}
+
+func setErrors(clients []TreeClient) {
+	for _, client := range clients {
+		node := client.(*treeClientMock)
+		node.err = true
+	}
+}

metrics/app.go

@@ -37,7 +37,7 @@ func (m *AppMetrics) SetEnabled(enabled bool) {
 	m.mu.Unlock()
 }
 
-func (m *AppMetrics) SetHealth(status int32) {
+func (m *AppMetrics) SetHealth(status HealthStatus) {
 	if !m.isEnabled() {
 		return
 	}
@@ -48,7 +48,7 @@ func (m *AppMetrics) SetHealth(status int32) {
 func (m *AppMetrics) Shutdown() {
 	m.mu.Lock()
 	if m.enabled {
-		m.gate.State.SetHealth(0)
+		m.gate.State.SetHealth(HealthStatusShuttingDown)
 		m.enabled = false
 	}
 	m.gate.Unregister()

metrics/state.go

@@ -4,6 +4,16 @@ import "github.com/prometheus/client_golang/prometheus"
 
 const stateSubsystem = "state"
 
+// HealthStatus of the gate application.
+type HealthStatus int32
+
+const (
+	HealthStatusUndefined    HealthStatus = 0
+	HealthStatusStarting     HealthStatus = 1
+	HealthStatusReady        HealthStatus = 2
+	HealthStatusShuttingDown HealthStatus = 3
+)
+
 type stateMetrics struct {
 	healthCheck prometheus.Gauge
 }
@@ -27,6 +37,6 @@ func (m stateMetrics) unregister() {
 	prometheus.Unregister(m.healthCheck)
 }
 
-func (m stateMetrics) SetHealth(s int32) {
+func (m stateMetrics) SetHealth(s HealthStatus) {
 	m.healthCheck.Set(float64(s))
 }

pkg/service/tree/tree.go

@@ -4,6 +4,7 @@ import (
 	"context"
 	"errors"
 	"fmt"
+	"sort"
 	"strconv"
 	"strings"
 	"time"
@@ -12,11 +13,13 @@ import (
 	"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer"
 	oid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id"
 	"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/user"
+	"go.uber.org/zap"
 )
 
 type (
 	Tree struct {
 		service ServiceClient
+		log     *zap.Logger
 	}
 
 	// ServiceClient is a client to interact with tree service.
@@ -104,8 +107,11 @@ const (
 )
 
 // NewTree creates instance of Tree using provided address and create grpc connection.
-func NewTree(service ServiceClient) *Tree {
+func NewTree(service ServiceClient, log *zap.Logger) *Tree {
-	return &Tree{service: service}
+	return &Tree{
+		service: service,
+		log:     log,
+	}
 }
 
 type Meta interface {
@@ -811,14 +817,19 @@ func (c *Tree) getUnversioned(ctx context.Context, bktInfo *data.BucketInfo, tre
 		return nil, err
 	}
 
-	if len(nodes) > 1 {
+	if len(nodes) == 0 {
-		return nil, fmt.Errorf("found more than one unversioned node")
-	}
-
-	if len(nodes) != 1 {
 		return nil, layer.ErrNodeNotFound
 	}
 
+	if len(nodes) > 1 {
+		c.log.Debug("found more than one unversioned node", zap.Stringer("cid", bktInfo.CID),
+			zap.String("treeID", treeID), zap.String("filepath", filepath))
+	}
+
+	sort.Slice(nodes, func(i, j int) bool {
+		return nodes[i].Timestamp > nodes[j].Timestamp
+	})
+
 	return nodes[0], nil
 }
 

pkg/service/tree/tree_test.go

@@ -8,6 +8,7 @@ import (
 	cidtest "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/id/test"
 	oidtest "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id/test"
 	"github.com/stretchr/testify/require"
+	"go.uber.org/zap/zaptest"
 )
 
 func TestLockConfigurationEncoding(t *testing.T) {
@@ -102,7 +103,7 @@ func TestTreeServiceSettings(t *testing.T) {
 
 	memCli, err := NewTreeServiceClientMemory()
 	require.NoError(t, err)
-	treeService := NewTree(memCli)
+	treeService := NewTree(memCli, zaptest.NewLogger(t))
 
 	bktInfo := &data.BucketInfo{
 		CID: cidtest.ID(),
@@ -134,7 +135,7 @@ func TestTreeServiceAddVersion(t *testing.T) {
 
 	memCli, err := NewTreeServiceClientMemory()
 	require.NoError(t, err)
-	treeService := NewTree(memCli)
+	treeService := NewTree(memCli, zaptest.NewLogger(t))
 
 	bktInfo := &data.BucketInfo{
 		CID: cidtest.ID(),