Commit Graph

3042 Commits (3ea1d7b7290a2dad4fa2992772006921e848733a)

Author SHA1 Message Date
Evgenii Stratonikov 3ea1d7b729 [#1089] control: Add USER and GROUP targets for local override storage
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-04-16 11:03:50 +00:00
Evgenii Stratonikov 0094186299 [#1089] control: Format proto files with clang-format
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-04-16 11:03:50 +00:00
Evgenii Stratonikov 91e79c98ba [#1089] ape: Provide request actor as an additional target
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-04-16 11:03:50 +00:00
Evgenii Stratonikov e5e0542482 [#1085] log: Move storage log message to constants package
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-04-15 07:57:00 +00:00
Evgenii Stratonikov 5be36924e3 [#41] log: Log storage operations in only in Debug
They are mostly useless unless we need to _debug_ a specific issue.
The amount of logs we produce is too big.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-04-15 07:57:00 +00:00
Airat Arifullin 6a46c6d229 [#1090] tree: Make workaround for APE checks
* Make `verifyClient` method perform APE check if a container
  was created with zero-filled basic ACL.
* Object verbs are used in APE, until tree verbs are introduced.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-04-15 07:45:45 +00:00
Airat Arifullin f4dcb418f2 [#1090] ape: Move ape request and resource implementations to common package
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-04-15 07:45:45 +00:00
Dmitrii Stepanov 40781b3a20 [#1086] engine: Change mode in case of errors async
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-04-10 12:29:43 +00:00
Evgenii Stratonikov 5ef5734c4e Reapply "[#972] Drop x/exp/slices dependency"
This reverts commit 946f2ec2bf.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-04-10 12:09:34 +00:00
Evgenii Stratonikov 669103a33e Reapply "[#972] Use slices.Sort* when useful"
This reverts commit 3359349acb.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-04-10 12:09:34 +00:00
Evgenii Stratonikov 2c4b50a71e Reapply "[#972] Use require.ElementsMatch() where possible"
This reverts commit 7627d08914.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-04-10 12:09:34 +00:00
Evgenii Stratonikov 3dc81cb4fc Reapply "[#972] Use min/max builtins"
This reverts commit dad56d2e98.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-04-10 12:09:34 +00:00
Evgenii Stratonikov d864945961 Reapply "[#972] pilorama: Remove removeDuplicatesInPlace()"
This reverts commit 9f68305c2e.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-04-10 12:09:34 +00:00
Dmitrii Stepanov 1b17258c04 [#1029] metabase: Add refill metrics
DCO action / DCO (pull_request) Successful in 1m22s Details
Vulncheck / Vulncheck (pull_request) Successful in 3m11s Details
Build / Build Components (1.21) (pull_request) Successful in 3m56s Details
Build / Build Components (1.20) (pull_request) Successful in 3m59s Details
Tests and linters / Staticcheck (pull_request) Successful in 5m31s Details
Tests and linters / gopls check (pull_request) Successful in 5m26s Details
Tests and linters / Lint (pull_request) Successful in 6m13s Details
Tests and linters / Tests (1.20) (pull_request) Successful in 8m54s Details
Tests and linters / Tests (1.21) (pull_request) Successful in 9m13s Details
Tests and linters / Tests with -race (pull_request) Successful in 9m30s Details
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-04-10 13:05:44 +03:00
Dmitrii Stepanov e3d9dd6ee8 [#1024] blobovnicza: Copy data on iterate
DB value is only valid while the tx is alive.
But handler may to run something in other goroutine.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-04-10 10:21:11 +03:00
Dmitrii Stepanov 57466594fb [#1024] shard: Resync metabase concurrently
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-04-10 10:21:10 +03:00
Dmitrii Stepanov 1005bf4f56 [#1024] shard: Add refill metabase benchmark
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-04-10 10:21:10 +03:00
Dmitrii Stepanov 76398c06b0 [#1080] metabase: Add StorageID metric
DCO action / DCO (pull_request) Successful in 1m19s Details
Build / Build Components (1.21) (pull_request) Successful in 4m6s Details
Vulncheck / Vulncheck (pull_request) Successful in 4m35s Details
Build / Build Components (1.20) (pull_request) Successful in 7m32s Details
Tests and linters / gopls check (pull_request) Successful in 7m43s Details
Tests and linters / Lint (pull_request) Successful in 9m51s Details
Tests and linters / Staticcheck (pull_request) Successful in 9m53s Details
Tests and linters / Tests (1.20) (pull_request) Successful in 12m48s Details
Tests and linters / Tests (1.21) (pull_request) Successful in 13m11s Details
Tests and linters / Tests with -race (pull_request) Successful in 14m4s Details
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-04-10 10:00:08 +03:00
Dmitrii Stepanov 7b1adfed3e [#1080] metabase: Open bucket for container counter once
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-04-10 10:00:08 +03:00
Dmitrii Stepanov e74bdaa5d5 [#1080] ape: Use value for APE request
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-04-09 18:42:03 +03:00
Dmitrii Stepanov 338d8cbebd [#1080] ape: Do not read object headers before Head/Get
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-04-09 15:27:40 +03:00
Dmitrii Stepanov 5b8200de88 [#984] blobovnicza: Do not fail rebuild on big objects
If blobovnicza contains objects larger than object size parameter
value, then rebuild fails with an error, because there is no such
bucket in database. This commit forces to create bucket on rebuild.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-04-09 11:51:18 +00:00
Anton Nikiforov 2b88361849 [#1062] object: Fix buffer allocation for `PayloadRange`
DCO action / DCO (pull_request) Successful in 5m34s Details
Vulncheck / Vulncheck (pull_request) Successful in 5m14s Details
Build / Build Components (1.20) (pull_request) Successful in 8m56s Details
Build / Build Components (1.21) (pull_request) Successful in 8m56s Details
Tests and linters / gopls check (pull_request) Successful in 9m3s Details
Tests and linters / Staticcheck (pull_request) Successful in 9m39s Details
Tests and linters / Lint (pull_request) Successful in 10m9s Details
Tests and linters / Tests (1.20) (pull_request) Successful in 13m43s Details
Tests and linters / Tests (1.21) (pull_request) Successful in 14m4s Details
Tests and linters / Tests with -race (pull_request) Successful in 14m40s Details
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-04-09 11:59:07 +03:00
Dmitrii Stepanov f5b67c6735 [#1064] policer: Disable EC processing
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-04-09 07:08:53 +00:00
Dmitrii Stepanov 1c5e0f90aa [#1064] putsvc: Add EC put
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-04-09 07:08:53 +00:00
Dmitrii Stepanov 39da643354 [#1064] putsvc: Refactor distributed target
Extract object builder.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-04-09 07:08:53 +00:00
Alexander Chuprov 17f7adb640 [#1065] adm: Add support EC parameters
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2024-04-08 12:27:30 +03:00
Anton Nikiforov 9aa533e59a [#1072] node, ir: Add new config option `kludge_compatibility_mode`
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-04-04 11:05:25 +03:00
Anton Nikiforov d614f04a0a [#1072] Fix gofumpt issues
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-04-03 22:21:14 +03:00
Airat Arifullin 4738508ce2 [#1063] go.mod: Update SDK version
* Update frostfs-sdk and frostfs-api-go versions.
* Refactor depreacted method ReplicaNumberByIndex.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-04-03 10:33:51 +00:00
Evgenii Stratonikov ff4c23f59a [#1070] services/tree: Fix fast listing depth processing
For unsorted `GetSubTree()` we return a single node for depth=1.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-04-02 14:41:31 +00:00
Evgenii Stratonikov 17af91619a [#1070] pilorama: Fix cycling behaviour for sorted listing
In case there are no items left, return empty slice.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-04-02 14:41:31 +00:00
Ekaterina Lebedeva d5194ab2a6 [#949] metabase: fix shard.UpdateID()
Vulncheck / Vulncheck (pull_request) Successful in 1m20s Details
DCO action / DCO (pull_request) Successful in 1m59s Details
Build / Build Components (1.21) (pull_request) Successful in 3m25s Details
Build / Build Components (1.20) (pull_request) Successful in 4m46s Details
Tests and linters / Staticcheck (pull_request) Successful in 6m5s Details
Tests and linters / gopls check (pull_request) Successful in 6m17s Details
Tests and linters / Lint (pull_request) Successful in 7m7s Details
Tests and linters / Tests (1.20) (pull_request) Successful in 8m38s Details
Tests and linters / Tests with -race (pull_request) Successful in 8m51s Details
Tests and linters / Tests (1.21) (pull_request) Successful in 8m56s Details
metabase.Open() now reports metabase mode metric. shard.UpdateID()
needs to read shard ID from metabase => needs to open metabase.
It caused reporting 'shard undefined' metrics. To avoid reporting
wrong metrics metabase.GetShardID() was added which also opens
metabase and does not report metrics.

Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2024-04-01 17:27:34 +03:00
Ekaterina Lebedeva 81a0346a96 [#949] metabase: fix metabase mode metric
It used to always show CLOSED regardless of actual mode.
Now metric represents actual metabase mode of operations.

Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2024-04-01 17:27:34 +03:00
Evgenii Stratonikov e12fcc041d [#1059] services/tree: Fast sorted listing
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-04-01 12:37:34 +00:00
Evgenii Stratonikov f23e38c285 Revert "[#446] engine: Move to read-only on blobstor errors"
DCO action / DCO (pull_request) Successful in 2m14s Details
Build / Build Components (1.20) (pull_request) Successful in 4m7s Details
Vulncheck / Vulncheck (pull_request) Successful in 3m30s Details
Build / Build Components (1.21) (pull_request) Successful in 4m15s Details
Tests and linters / Staticcheck (pull_request) Successful in 5m41s Details
Tests and linters / Lint (pull_request) Successful in 6m6s Details
Tests and linters / gopls check (pull_request) Successful in 6m42s Details
Tests and linters / Tests (1.20) (pull_request) Successful in 7m47s Details
Tests and linters / Tests (1.21) (pull_request) Successful in 8m21s Details
Tests and linters / Tests with -race (pull_request) Successful in 8m20s Details
This reverts commit 69df0d21c2.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-04-01 12:48:30 +03:00
Dmitrii Stepanov 942d83611b [#874] engine: Revert Check object existance concurrently
This reverts commit f526f49995.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-04-01 08:42:34 +00:00
Evgenii Stratonikov fd8cdb9671 [#1057] netmap: Do not iterate over external addresses in Node
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-03-29 20:46:24 +00:00
Ekaterina Lebedeva 0990a9b0bd [#1055] blobstor: fix mode metric
It used to always show CLOSED after setting shard mode
to read-only regardless of actual mode.
Now metric represents actual blobstor mode of operations.

Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2024-03-29 20:44:47 +00:00
Evgenii Stratonikov c7a12ca3d8 [#1054] network: Optimize IsTLSEnabled()
DCO action / DCO (pull_request) Successful in 5m33s Details
Vulncheck / Vulncheck (pull_request) Successful in 5m54s Details
Build / Build Components (1.21) (pull_request) Successful in 10m3s Details
Tests and linters / gopls check (pull_request) Successful in 11m49s Details
Build / Build Components (1.20) (pull_request) Successful in 12m47s Details
Tests and linters / Staticcheck (pull_request) Successful in 13m21s Details
Tests and linters / Lint (pull_request) Successful in 14m21s Details
Tests and linters / Tests (1.20) (pull_request) Successful in 16m39s Details
Tests and linters / Tests with -race (pull_request) Successful in 16m46s Details
Tests and linters / Tests (1.21) (pull_request) Successful in 16m59s Details
No big deal, but it is called multiple times in sorting routine, this
easily results in 20 allocations per group traversal.

```
goos: linux
goarch: amd64
pkg: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/network
cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz
                    │     old     │                 new                 │
                    │   sec/op    │   sec/op     vs base                │
AddressTLSEnabled-8   184.6n ± 1%   103.3n ± 6%  -44.04% (p=0.000 n=10)

                    │    old     │                new                │
                    │    B/op    │   B/op    vs base                 │
AddressTLSEnabled-8   704.0 ± 0%   0.0 ± 0%  -100.00% (p=0.000 n=10)

                    │    old     │                 new                 │
                    │ allocs/op  │ allocs/op   vs base                 │
AddressTLSEnabled-8   1.000 ± 0%   0.000 ± 0%  -100.00% (p=0.000 n=10)
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-03-21 18:48:35 +03:00
Dmitrii Stepanov c09c701613 [#1048] metabase: Fix drop buckets during resync
Vulncheck / Vulncheck (pull_request) Successful in 1m35s Details
DCO action / DCO (pull_request) Successful in 2m12s Details
Build / Build Components (1.21) (pull_request) Successful in 3m57s Details
Build / Build Components (1.20) (pull_request) Successful in 4m45s Details
Tests and linters / Staticcheck (pull_request) Successful in 5m32s Details
Tests and linters / gopls check (pull_request) Successful in 5m33s Details
Tests and linters / Lint (pull_request) Successful in 6m18s Details
Tests and linters / Tests (1.20) (pull_request) Successful in 8m8s Details
Tests and linters / Tests with -race (pull_request) Successful in 8m14s Details
Tests and linters / Tests (1.21) (pull_request) Successful in 8m36s Details
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-03-19 14:28:31 +03:00
Airat Arifullin 6959e617c4 [#1047] object: Set container owner ID property to ape request
* Introduce ContainerOwner field in RequestContext.
* Set ContainerOwner in aclv2 middleware.
* Set PropertyKeyContainerOwnerID for object ape request.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-03-18 15:39:50 +00:00
Airat Arifullin d7be70e93f [#1040] object: Wrap CheckAPE errors to status errors
* All methods should wrap CheckAPE error, if it occurs, to
  status error.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-03-14 07:34:03 +00:00
Anton Nikiforov 5ee5f1df42 [#976] control: Introduce new method `RemoveChainLocalOverridesByTarget`
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-03-13 15:33:19 +03:00
Anton Nikiforov 2d595ec15f [#976] ape: Update dependency
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-03-13 14:54:19 +03:00
Anton Nikiforov 7ed07d2dfd [#976] morph: Implement missing methods for `SwitchRPCGuardedActor`
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-03-13 14:54:19 +03:00
Dmitrii Stepanov 17f5463389 [#1043] cli: Add reset evacuation status command
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-03-13 10:29:45 +00:00
Dmitrii Stepanov 31e2396a5f [#1043] control: Add ResetEvacuationStatus implementation
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-03-13 10:29:45 +00:00
Dmitrii Stepanov 926cdeb072 [#1043] services: Regenerate proto
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-03-13 10:29:45 +00:00
Airat Arifullin 5c252c9193 [#1039] object: Skip APE check for certain request roles
DCO action / DCO (pull_request) Successful in 1m31s Details
Vulncheck / Vulncheck (pull_request) Successful in 2m52s Details
Build / Build Components (1.21) (pull_request) Successful in 3m52s Details
Build / Build Components (1.20) (pull_request) Successful in 4m16s Details
Tests and linters / gopls check (pull_request) Successful in 11m54s Details
Tests and linters / Staticcheck (pull_request) Successful in 12m31s Details
Tests and linters / Tests (1.21) (pull_request) Successful in 12m49s Details
Tests and linters / Tests (1.20) (pull_request) Successful in 13m8s Details
Tests and linters / Tests with -race (pull_request) Successful in 13m14s Details
Tests and linters / Lint (pull_request) Successful in 13m31s Details
* Skip APE check if a role is Container.
* Skip APE check if a role is IR and methods are get-like.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-03-12 16:15:20 +03:00