44df67492f
[ #1493 ] metabase: Split inhumeTx() into 2 functions
...
No functional changes.
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-11-14 06:47:04 +00:00
1e6f132b4e
[ #1493 ] metabase: Pass InhumePrm by value
...
Unify with the other code, no functional changes.
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-11-14 06:47:04 +00:00
6dc0dc6691
[ #1493 ] shard: Take mode mutex in HandleExpiredTombstones()
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-11-14 06:47:04 +00:00
7fc6101bec
[ #1491 ] engine/test: Rework engine test utils
...
Tests and linters / Run gofumpt (pull_request) Successful in 2m24s
DCO action / DCO (pull_request) Successful in 2m37s
Pre-commit hooks / Pre-commit (pull_request) Successful in 3m46s
Vulncheck / Vulncheck (pull_request) Successful in 3m46s
Build / Build Components (pull_request) Successful in 4m4s
Tests and linters / Staticcheck (pull_request) Successful in 4m1s
Tests and linters / gopls check (pull_request) Successful in 4m14s
Tests and linters / Lint (pull_request) Successful in 4m49s
Tests and linters / Tests with -race (pull_request) Successful in 5m25s
Tests and linters / Tests (pull_request) Successful in 5m46s
- Remove `testNewShard` and `setInitializedShards` because they
violated the default engine workflow. The correct workflow is:
first use `New()`, followed by `Open()`, and then `Init()`. As a
result, adding new logic to `(*StorageEngine).Init` caused several
tests to fail with a panic when attempting to access uninitialized
resources. Now, all engines created with the test utils must be
initialized manually. The new helper method `prepare` can be used
for that purpose.
- Additionally, `setInitializedShards` hardcoded the shard worker
pool size, which prevented it from being configured in tests and
benchmarks. This has been fixed as well.
- Ensure engine initialization is done wherever it was missing.
- Refactor `setShardsNumOpts`, `setShardsNumAdditionalOpts`, and
`setShardsNum`. Make them all depend on `setShardsNumOpts`.
Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-11-13 14:42:53 +03:00
7ef36749d0
[ #1491 ] engine/test: Move BenchmarkExists to exists_test.go
...
Move `BenchmarkExists` from `engine_test.go` to `exists_test.go`
for better organization and clarity.
Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-11-13 14:09:29 +03:00
c6066d6ee4
[ #1491 ] engine/test: Use more suitable testing utils here and there
...
Use `setShardsNum` instead of `setInitializedShards` wherever possible.
Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-11-13 14:09:29 +03:00
7429553266
[ #1437 ] node: Fix contextcheck linter
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-11-13 10:36:10 +03:00
16598553d9
[ #1437 ] shard: Fix contextcheck linter
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-11-13 10:36:09 +03:00
62b5181618
[ #1437 ] blobovnicza: Fix contextcheck linter
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-11-13 10:36:08 +03:00
6db46257c0
[ #1437 ] node: Use ctx for logging
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-11-13 10:36:07 +03:00
fd004add00
[ #1492 ] metabase: Fix import formatting
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-11-13 07:30:25 +00:00
8ed7a676d5
[ #1492 ] metabase: Ensure Unmarshal() is called on a cloned slice
...
The slice returned from bucket.Get() is only valid during the tx
lifetime. Cloning it is not necessary everywhere, but better safe than
sorry.
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-11-13 07:30:25 +00:00
b451de94c8
[ #1492 ] metabase: Fix typo in objData
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-11-13 07:30:25 +00:00
9bd05e94c8
[ #1449 ] tree: Add ApplyBatch method
...
Concurrent Apply can lead to child node applies before parent, so
undo/redo operations will perform. This leads to performance degradation
in case of tree with many sublevels.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-11-12 12:11:07 +00:00
8a57c78f5f
[ #1484 ] engine: Fix engine metrics
...
Tests and linters / Run gofumpt (pull_request) Successful in 2m38s
Pre-commit hooks / Pre-commit (pull_request) Successful in 2m51s
DCO action / DCO (pull_request) Successful in 2m53s
Vulncheck / Vulncheck (pull_request) Successful in 3m44s
Tests and linters / gopls check (pull_request) Successful in 3m53s
Build / Build Components (pull_request) Successful in 4m15s
Tests and linters / Staticcheck (pull_request) Successful in 4m56s
Tests and linters / Tests with -race (pull_request) Successful in 5m5s
Tests and linters / Lint (pull_request) Successful in 5m47s
Tests and linters / Tests (pull_request) Successful in 5m51s
1. Add forgotten metrics for client requests
2. Include execIfNotBlocked into metrics
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-11-11 12:59:20 +03:00
9b13a18aac
[ #1479 ] go.mod: Bump frostfs-sdk-go version
...
* Update version within go.mod;
* Fix deprecated frostfs-api-go/v2 package and use frostfs-sdk-go/api
instead.
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-11-08 10:43:19 +03:00
15102e6dfd
[ #1471 ] Replace sort.Slice in some places
...
`slices.SortFunc` doesn't use reflection and is a bit faster.
I have done some micro-benchmarks for `[]NodeInfo`:
```
$ benchstat -col "/func" out
goos: linux
goarch: amd64
pkg: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/pilorama
cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz
│ sort.Slice │ slices.SortFunc │
│ sec/op │ sec/op vs base │
Sort-8 2.130µ ± 2% 1.253µ ± 2% -41.20% (p=0.000 n=10)
```
Haven't included them, though, as they I don't see them being used a
lot.
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-11-06 08:07:32 +00:00
c2effcc61c
[ #1465 ] Makefile: Update golangci-lint, fix warnings
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-10-31 06:39:59 +00:00
7ac0852364
[ #1459 ] .golangci.yml: Add intrange linter, fix issues
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-10-30 15:18:22 +00:00
b0c5def2d9
[ #1433 ] shard/test: Use WithDisabledGC() option where possible
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-10-17 14:38:07 +03:00
41038b2ec0
[ #1431 ] node: Fix 'empty slice declaration using a literal'
...
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2024-10-15 08:53:59 +00:00
d83879d4b8
[ #1431 ] node: Fix comment format
...
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2024-10-15 08:53:59 +00:00
f6582081a4
[ #1431 ] obj_storage/metabase: Delete unused variable
...
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2024-10-15 08:53:59 +00:00
00b1cecfb7
[ #1431 ] obj_storage/shard: Fix visibility of 'newMetricStore'
...
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2024-10-15 08:53:59 +00:00
d53732f663
[ #1431 ] engine: Delete always false condition
...
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2024-10-15 08:53:59 +00:00
3012286452
[ #1431 ] metabase: Fix unreachable code
...
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2024-10-15 08:53:59 +00:00
936ebbb8e5
[ #1423 ] metabase: Hide BucketName form upper levels
...
Tests and linters / Run gofumpt (pull_request) Successful in 2m11s
DCO action / DCO (pull_request) Successful in 2m28s
Pre-commit hooks / Pre-commit (pull_request) Successful in 4m17s
Vulncheck / Vulncheck (pull_request) Successful in 4m25s
Tests and linters / Lint (pull_request) Successful in 4m37s
Tests and linters / gopls check (pull_request) Successful in 4m34s
Tests and linters / Tests with -race (pull_request) Successful in 4m37s
Build / Build Components (pull_request) Successful in 4m49s
Tests and linters / Staticcheck (pull_request) Successful in 6m1s
Tests and linters / Tests (pull_request) Successful in 7m12s
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-10-08 18:41:16 +03:00
c065d55ca3
[ #1412 ] metabase: Drop logging inside transaction
...
DCO action / DCO (pull_request) Successful in 1m13s
Tests and linters / Run gofumpt (pull_request) Successful in 1m29s
Vulncheck / Vulncheck (pull_request) Successful in 2m0s
Pre-commit hooks / Pre-commit (pull_request) Successful in 2m10s
Build / Build Components (pull_request) Successful in 2m22s
Tests and linters / gopls check (pull_request) Successful in 2m36s
Tests and linters / Staticcheck (pull_request) Successful in 3m14s
Tests and linters / Lint (pull_request) Successful in 3m31s
Tests and linters / Tests (pull_request) Successful in 4m16s
Tests and linters / Tests with -race (pull_request) Successful in 5m55s
This could lead to hang the db.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-10-08 11:41:52 +03:00
fe9f664b57
[ #1412 ] metabase: Drop empty user attribute buckets on upgrade
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-10-08 11:41:52 +03:00
87f4b934d1
[ #1412 ] metabase: Run bucket drop steps on upgrade concurrently
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-10-08 11:41:52 +03:00
8093e145b3
[ #1412 ] adm: Resolve container type by metabase upgrade
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-10-08 11:41:51 +03:00
3da168f8cf
[ #1412 ] shard: Resolve container is indexed on metabase resync
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-10-08 11:41:51 +03:00
4572fa4874
[ #1412 ] searchSvc: Check container is indexed
...
For non S3 containers it is expected to use attributes index for some
attributes.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-10-08 11:41:50 +03:00
1efa64ee72
[ #1412 ] metabase: Add search by indexed attributes
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-10-08 11:41:50 +03:00
be744ae3e6
[ #1412 ] metabase: Index attributes for indexed containers
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-10-08 11:41:49 +03:00
1b520f7973
[ #1412 ] engine: Add IsIndexedContainer flag
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-10-08 11:41:49 +03:00
899cd55c27
[ #1412 ] engine: PutPrm refactoring
...
Use fields instead of methods.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-10-08 11:41:48 +03:00
fc032838c0
[ #1215 ] blobstor/test: Cover iteration behaviour
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-10-07 12:44:27 +00:00
2f710d8f94
[ #1414 ] metabase: Check parameter for CountAliveObjectsInBucket
...
DCO action / DCO (pull_request) Successful in 1m33s
Tests and linters / Run gofumpt (pull_request) Successful in 1m28s
Pre-commit hooks / Pre-commit (pull_request) Successful in 2m10s
Vulncheck / Vulncheck (pull_request) Successful in 2m6s
Build / Build Components (pull_request) Successful in 2m32s
Tests and linters / gopls check (pull_request) Successful in 2m44s
Tests and linters / Staticcheck (pull_request) Successful in 3m6s
Tests and linters / Lint (pull_request) Successful in 3m35s
Tests and linters / Tests (pull_request) Successful in 4m21s
Tests and linters / Tests with -race (pull_request) Successful in 5m44s
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-10-04 17:11:15 +03:00
4dc9a1b300
[ #1413 ] engine: Remove error counting methods from Shard
...
Tests and linters / Run gofumpt (pull_request) Successful in 2m4s
DCO action / DCO (pull_request) Successful in 2m22s
Pre-commit hooks / Pre-commit (pull_request) Successful in 4m10s
Vulncheck / Vulncheck (pull_request) Successful in 4m5s
Build / Build Components (pull_request) Successful in 4m31s
Tests and linters / Staticcheck (pull_request) Successful in 4m21s
Tests and linters / gopls check (pull_request) Successful in 4m43s
Tests and linters / Lint (pull_request) Successful in 4m58s
Tests and linters / Tests (pull_request) Successful in 6m36s
Tests and linters / Tests with -race (pull_request) Successful in 7m41s
All error counting and hangling logic is present on the engine level.
Currently, we pass engine metrics with shard ID metric to shard, then
export 3 methods to manipulate these metrics.
In this commits all methods are removed and error counter is tracked on
the engine level exlusively.
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-10-04 15:10:17 +03:00
963faa615a
[ #1413 ] engine: Cleanup shard error reporting
...
- `reportShardErrorBackground()` no longer differs from
`reportShardError()`, reflect this in its name;
- reuse common pieces of code to make it simpler.
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-10-04 15:10:17 +03:00
9a87acb87a
[ #1410 ] engine: Provide the default implementation to MetricsRegister
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-10-03 08:23:06 +00:00
9206ce5cd2
[ #1410 ] shard: Provide the default implementation for MetricsWriter
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-10-03 08:23:06 +00:00
6c46044c9c
[ #1410 ] shard: Move MetricsWriter interface to a separate file
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-10-03 08:23:06 +00:00
01e3944b31
[ #1408 ] metabase: Fix tests
...
DCO action / DCO (pull_request) Successful in 1m3s
Tests and linters / Run gofumpt (pull_request) Successful in 1m21s
Pre-commit hooks / Pre-commit (pull_request) Successful in 2m27s
Vulncheck / Vulncheck (pull_request) Successful in 2m20s
Build / Build Components (pull_request) Successful in 2m41s
Tests and linters / gopls check (pull_request) Successful in 2m57s
Tests and linters / Staticcheck (pull_request) Successful in 3m5s
Tests and linters / Lint (pull_request) Successful in 3m37s
Tests and linters / Tests (pull_request) Successful in 4m31s
Tests and linters / Tests with -race (pull_request) Successful in 6m22s
No need to specify container ID for objects created with `testutil.GenerateObjectWithCID`.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-10-02 14:41:40 +03:00
434048e8d9
[ #1408 ] metabase: Fix EC search with slow and fast filters
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-10-02 14:36:59 +03:00
d0ed29b3c7
[ #1350 ] node: Add ability to evacuate objects from REP 1 only
...
Tests and linters / Run gofumpt (pull_request) Successful in 50s
DCO action / DCO (pull_request) Successful in 1m41s
Pre-commit hooks / Pre-commit (pull_request) Successful in 1m53s
Vulncheck / Vulncheck (pull_request) Successful in 2m10s
Build / Build Components (pull_request) Successful in 2m43s
Tests and linters / gopls check (pull_request) Successful in 3m0s
Tests and linters / Staticcheck (pull_request) Successful in 3m5s
Tests and linters / Lint (pull_request) Successful in 3m52s
Tests and linters / Tests (pull_request) Successful in 4m40s
Tests and linters / Tests with -race (pull_request) Successful in 6m17s
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-09-27 15:41:17 +03:00
401c398704
[ #1388 ] metabase: Drop unused
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-25 08:55:38 +00:00
004ff9e9bf
[ #1388 ] blobstor: Drop unused
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-25 08:55:38 +00:00
63a567a1de
[ #1388 ] engine: Drop unused
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-25 08:55:38 +00:00
