Commit Graph

3029 Commits (feat/resync_speedup)

Author SHA1 Message Date
Dmitrii Stepanov 1b17258c04 [#1029] metabase: Add refill metrics
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-04-10 13:05:44 +03:00
Dmitrii Stepanov e3d9dd6ee8 [#1024] blobovnicza: Copy data on iterate
DB value is only valid while the tx is alive.
But handler may to run something in other goroutine.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-04-10 10:21:11 +03:00
Dmitrii Stepanov 57466594fb [#1024] shard: Resync metabase concurrently
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-04-10 10:21:10 +03:00
Dmitrii Stepanov 1005bf4f56 [#1024] shard: Add refill metabase benchmark
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-04-10 10:21:10 +03:00
Dmitrii Stepanov 76398c06b0 [#1080] metabase: Add StorageID metric
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-04-10 10:00:08 +03:00
Dmitrii Stepanov 7b1adfed3e [#1080] metabase: Open bucket for container counter once
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-04-10 10:00:08 +03:00
Dmitrii Stepanov e74bdaa5d5 [#1080] ape: Use value for APE request
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-04-09 18:42:03 +03:00
Dmitrii Stepanov 338d8cbebd [#1080] ape: Do not read object headers before Head/Get
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-04-09 15:27:40 +03:00
Dmitrii Stepanov 5b8200de88 [#984] blobovnicza: Do not fail rebuild on big objects
If blobovnicza contains objects larger than object size parameter
value, then rebuild fails with an error, because there is no such
bucket in database. This commit forces to create bucket on rebuild.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-04-09 11:51:18 +00:00
Anton Nikiforov 2b88361849 [#1062] object: Fix buffer allocation for `PayloadRange`
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-04-09 11:59:07 +03:00
Dmitrii Stepanov f5b67c6735 [#1064] policer: Disable EC processing
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-04-09 07:08:53 +00:00
Dmitrii Stepanov 1c5e0f90aa [#1064] putsvc: Add EC put
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-04-09 07:08:53 +00:00
Dmitrii Stepanov 39da643354 [#1064] putsvc: Refactor distributed target
Extract object builder.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-04-09 07:08:53 +00:00
Alexander Chuprov 17f7adb640 [#1065] adm: Add support EC parameters
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2024-04-08 12:27:30 +03:00
Anton Nikiforov 9aa533e59a [#1072] node, ir: Add new config option `kludge_compatibility_mode`
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-04-04 11:05:25 +03:00
Anton Nikiforov d614f04a0a [#1072] Fix gofumpt issues
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-04-03 22:21:14 +03:00
Airat Arifullin 4738508ce2 [#1063] go.mod: Update SDK version
* Update frostfs-sdk and frostfs-api-go versions.
* Refactor depreacted method ReplicaNumberByIndex.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-04-03 10:33:51 +00:00
Evgenii Stratonikov ff4c23f59a [#1070] services/tree: Fix fast listing depth processing
For unsorted `GetSubTree()` we return a single node for depth=1.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-04-02 14:41:31 +00:00
Evgenii Stratonikov 17af91619a [#1070] pilorama: Fix cycling behaviour for sorted listing
In case there are no items left, return empty slice.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-04-02 14:41:31 +00:00
Ekaterina Lebedeva d5194ab2a6 [#949] metabase: fix shard.UpdateID()
metabase.Open() now reports metabase mode metric. shard.UpdateID()
needs to read shard ID from metabase => needs to open metabase.
It caused reporting 'shard undefined' metrics. To avoid reporting
wrong metrics metabase.GetShardID() was added which also opens
metabase and does not report metrics.

Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2024-04-01 17:27:34 +03:00
Ekaterina Lebedeva 81a0346a96 [#949] metabase: fix metabase mode metric
It used to always show CLOSED regardless of actual mode.
Now metric represents actual metabase mode of operations.

Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2024-04-01 17:27:34 +03:00
Evgenii Stratonikov e12fcc041d [#1059] services/tree: Fast sorted listing
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-04-01 12:37:34 +00:00
Evgenii Stratonikov f23e38c285 Revert "[#446] engine: Move to read-only on blobstor errors"
This reverts commit 69df0d21c2.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-04-01 12:48:30 +03:00
Dmitrii Stepanov 942d83611b [#874] engine: Revert Check object existance concurrently
This reverts commit f526f49995.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-04-01 08:42:34 +00:00
Evgenii Stratonikov fd8cdb9671 [#1057] netmap: Do not iterate over external addresses in Node
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-03-29 20:46:24 +00:00
Ekaterina Lebedeva 0990a9b0bd [#1055] blobstor: fix mode metric
It used to always show CLOSED after setting shard mode
to read-only regardless of actual mode.
Now metric represents actual blobstor mode of operations.

Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2024-03-29 20:44:47 +00:00
Evgenii Stratonikov c7a12ca3d8 [#1054] network: Optimize IsTLSEnabled()
No big deal, but it is called multiple times in sorting routine, this
easily results in 20 allocations per group traversal.

```
goos: linux
goarch: amd64
pkg: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/network
cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz
                    │     old     │                 new                 │
                    │   sec/op    │   sec/op     vs base                │
AddressTLSEnabled-8   184.6n ± 1%   103.3n ± 6%  -44.04% (p=0.000 n=10)

                    │    old     │                new                │
                    │    B/op    │   B/op    vs base                 │
AddressTLSEnabled-8   704.0 ± 0%   0.0 ± 0%  -100.00% (p=0.000 n=10)

                    │    old     │                 new                 │
                    │ allocs/op  │ allocs/op   vs base                 │
AddressTLSEnabled-8   1.000 ± 0%   0.000 ± 0%  -100.00% (p=0.000 n=10)
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-03-21 18:48:35 +03:00
Dmitrii Stepanov c09c701613 [#1048] metabase: Fix drop buckets during resync
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-03-19 14:28:31 +03:00
Airat Arifullin 6959e617c4 [#1047] object: Set container owner ID property to ape request
* Introduce ContainerOwner field in RequestContext.
* Set ContainerOwner in aclv2 middleware.
* Set PropertyKeyContainerOwnerID for object ape request.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-03-18 15:39:50 +00:00
Airat Arifullin d7be70e93f [#1040] object: Wrap CheckAPE errors to status errors
* All methods should wrap CheckAPE error, if it occurs, to
  status error.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-03-14 07:34:03 +00:00
Anton Nikiforov 5ee5f1df42 [#976] control: Introduce new method `RemoveChainLocalOverridesByTarget`
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-03-13 15:33:19 +03:00
Anton Nikiforov 2d595ec15f [#976] ape: Update dependency
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-03-13 14:54:19 +03:00
Anton Nikiforov 7ed07d2dfd [#976] morph: Implement missing methods for `SwitchRPCGuardedActor`
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-03-13 14:54:19 +03:00
Dmitrii Stepanov 17f5463389 [#1043] cli: Add reset evacuation status command
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-03-13 10:29:45 +00:00
Dmitrii Stepanov 31e2396a5f [#1043] control: Add ResetEvacuationStatus implementation
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-03-13 10:29:45 +00:00
Dmitrii Stepanov 926cdeb072 [#1043] services: Regenerate proto
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-03-13 10:29:45 +00:00
Airat Arifullin 5c252c9193 [#1039] object: Skip APE check for certain request roles
* Skip APE check if a role is Container.
* Skip APE check if a role is IR and methods are get-like.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-03-12 16:15:20 +03:00
Anton Nikiforov 3195142d67 [#959] writecache: Avoid manipulation with cache in `DEGRADED` mode
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-03-11 18:35:41 +00:00
Dmitrii Stepanov d433b49265 [#973] node: Resolve perfsprint linter
`fmt.Errorf can be replaced with errors.New` and `fmt.Sprintf can be replaced with string addition`

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-03-11 17:55:50 +03:00
Dmitrii Stepanov 66a26b7775 [#973] node: Resolve revive: unused-parameter linter
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-03-11 17:11:49 +03:00
Dmitrii Stepanov 0882840bf5 [#634] shard: Add writecache inhume tests
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-03-06 13:12:34 +03:00
Anton Nikiforov ae5bb87e70 Revert "[#866] Use TTL for blobovnicza tree cache"
This reverts commit d9cbb16bd3.

Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-03-01 19:29:33 +03:00
Dmitrii Stepanov d6534fd755 [#1016] frostfs-node: Fix gopls issues
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-03-01 12:13:43 +03:00
Airat Arifullin 93bf9acbc2 [#898] control: Remove removed flag from RemoveChainLocalOverrideResponse
* Remove removed flag in service.proto for RemoveChainLocalOverrideResponse.
* Regenerate control API.
* Return error only if RemoveOverride returns non-NotFound code.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-02-28 19:07:07 +00:00
Airat Arifullin 75a1a95c2c [#986] tree: Skip ACL checks if basicACL mask is unset
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-02-28 19:05:57 +00:00
Airat Arifullin b1d171c261 [#986] container: Interpret APE NoRuleFound as request deny
* If APE check returns NoRuleFound, then it is taken for request deny.
* Add more unit-test for ape container middleware.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-02-28 19:05:57 +00:00
Airat Arifullin 7cc368e188 [#986] object: Introduce soft ape checks
* Soft APE check means that APE should allow request even
  it gets status NoRuleFound for a request. Otherwise,
  it is interpreted as Deny.
* Soft APE check is performed if basic ACL mask is not set.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-02-28 19:05:57 +00:00
Dmitrii Stepanov 918613546f [#1008] metabase: Do not update storageID on put
There may be a race condition between put an object and
flushing the writecache:
1. Put object to the writecache
2. Writecache flushes object to the blobstore and sets blobstore's
storageID
3. Put object to the metabase, set writecache's storageID

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-02-28 11:01:50 +03:00
Dmitrii Stepanov 2ad433dbcb [#1005] engine: Drop shards weights
Unused.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-02-26 17:25:05 +03:00
Evgenii Stratonikov 4b13b85173 [#1000] morph: Fix batch size in TraverseIterator()
Initial prefetch size can be arbitrary an restricted only by VM/RPC
limits. For TraverseIterator() there is an explicit check on the
server-side, though.
Introduced in df055fead5.
Refs #931.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-02-23 06:24:40 +00:00