Commit graph

3434 commits

Author SHA1 Message Date
69c63006da
[#1496] morph: Move tx waiter to morph package
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-11-15 16:36:07 +03:00
d77a218f7c [#1493] metabase: Merge Inhume() and DropGraves() for tombstones
DropGraves() is only used to drop gravemarks after a tombstone
removal. Thus, it makes sense to do Inhume() and DropGraves() in one
transaction. It has less overhead and no unexpected problems in case
of sudden power failure.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-11-14 06:47:04 +00:00
44df67492f [#1493] metabase: Split inhumeTx() into 2 functions
No functional changes.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-11-14 06:47:04 +00:00
1e6f132b4e [#1493] metabase: Pass InhumePrm by value
Unify with the other code, no functional changes.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-11-14 06:47:04 +00:00
6dc0dc6691 [#1493] shard: Take mode mutex in HandleExpiredTombstones()
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-11-14 06:47:04 +00:00
7fc6101bec
[#1491] engine/test: Rework engine test utils
All checks were successful
Tests and linters / Run gofumpt (pull_request) Successful in 2m24s
DCO action / DCO (pull_request) Successful in 2m37s
Pre-commit hooks / Pre-commit (pull_request) Successful in 3m46s
Vulncheck / Vulncheck (pull_request) Successful in 3m46s
Build / Build Components (pull_request) Successful in 4m4s
Tests and linters / Staticcheck (pull_request) Successful in 4m1s
Tests and linters / gopls check (pull_request) Successful in 4m14s
Tests and linters / Lint (pull_request) Successful in 4m49s
Tests and linters / Tests with -race (pull_request) Successful in 5m25s
Tests and linters / Tests (pull_request) Successful in 5m46s
- Remove `testNewShard` and `setInitializedShards` because they
violated the default engine workflow. The correct workflow is:
first use `New()`, followed by `Open()`, and then `Init()`. As a
result, adding new logic to `(*StorageEngine).Init` caused several
tests to fail with a panic when attempting to access uninitialized
resources. Now, all engines created with the test utils must be
initialized manually. The new helper method `prepare` can be used
for that purpose.
- Additionally, `setInitializedShards` hardcoded the shard worker
pool size, which prevented it from being configured in tests and
benchmarks. This has been fixed as well.
- Ensure engine initialization is done wherever it was missing.
- Refactor `setShardsNumOpts`, `setShardsNumAdditionalOpts`, and
`setShardsNum`. Make them all depend on `setShardsNumOpts`.

Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-11-13 14:42:53 +03:00
7ef36749d0
[#1491] engine/test: Move BenchmarkExists to exists_test.go
Move `BenchmarkExists` from `engine_test.go` to `exists_test.go`
for better organization and clarity.

Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-11-13 14:09:29 +03:00
c6066d6ee4
[#1491] engine/test: Use more suitable testing utils here and there
Use `setShardsNum` instead of `setInitializedShards` wherever possible.

Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-11-13 14:09:29 +03:00
612b34d570
[#1437] logger: Add caller skip to log original caller position
All checks were successful
Tests and linters / Run gofumpt (pull_request) Successful in 1m36s
DCO action / DCO (pull_request) Successful in 1m50s
Vulncheck / Vulncheck (pull_request) Successful in 2m19s
Pre-commit hooks / Pre-commit (pull_request) Successful in 2m34s
Build / Build Components (pull_request) Successful in 2m49s
Tests and linters / gopls check (pull_request) Successful in 3m6s
Tests and linters / Staticcheck (pull_request) Successful in 3m15s
Tests and linters / Lint (pull_request) Successful in 3m56s
Tests and linters / Tests (pull_request) Successful in 4m44s
Tests and linters / Tests with -race (pull_request) Successful in 5m36s
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-11-13 10:36:12 +03:00
7429553266
[#1437] node: Fix contextcheck linter
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-11-13 10:36:10 +03:00
6921a89061
[#1437] ir: Fix contextcheck linters
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-11-13 10:36:10 +03:00
16598553d9
[#1437] shard: Fix contextcheck linter
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-11-13 10:36:09 +03:00
c139892117
[#1437] ir: Fix contextcheck linter
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-11-13 10:36:09 +03:00
62b5181618
[#1437] blobovnicza: Fix contextcheck linter
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-11-13 10:36:08 +03:00
6db46257c0
[#1437] node: Use ctx for logging
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-11-13 10:36:07 +03:00
c16dae8b4d
[#1437] logger: Use context to log trace id
Signed-off-by: Dmitrii Stepanov
2024-11-13 10:36:07 +03:00
fd004add00 [#1492] metabase: Fix import formatting
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-11-13 07:30:25 +00:00
8ed7a676d5 [#1492] metabase: Ensure Unmarshal() is called on a cloned slice
The slice returned from bucket.Get() is only valid during the tx
lifetime. Cloning it is not necessary everywhere, but better safe than
sorry.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-11-13 07:30:25 +00:00
b451de94c8 [#1492] metabase: Fix typo in objData
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-11-13 07:30:25 +00:00
46fef276b4 [#1449] tree: Log tree sync with Info level
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-11-12 12:11:07 +00:00
9bd05e94c8 [#1449] tree: Add ApplyBatch method
Concurrent Apply can lead to child node applies before parent, so
undo/redo operations will perform. This leads to performance degradation
in case of tree with many sublevels.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-11-12 12:11:07 +00:00
3324c26fd8 [#1483] morph: Remove container.GetEACL()
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-11-12 12:10:51 +00:00
a692298533 [#1483] node: Remove eACL cache
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-11-12 12:10:51 +00:00
8a57c78f5f
[#1484] engine: Fix engine metrics
All checks were successful
Tests and linters / Run gofumpt (pull_request) Successful in 2m38s
Pre-commit hooks / Pre-commit (pull_request) Successful in 2m51s
DCO action / DCO (pull_request) Successful in 2m53s
Vulncheck / Vulncheck (pull_request) Successful in 3m44s
Tests and linters / gopls check (pull_request) Successful in 3m53s
Build / Build Components (pull_request) Successful in 4m15s
Tests and linters / Staticcheck (pull_request) Successful in 4m56s
Tests and linters / Tests with -race (pull_request) Successful in 5m5s
Tests and linters / Lint (pull_request) Successful in 5m47s
Tests and linters / Tests (pull_request) Successful in 5m51s
1. Add forgotten metrics for client requests
2. Include execIfNotBlocked into metrics

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-11-11 12:59:20 +03:00
c82c753e9f [#1480] objsvc: Remove useless stream wrappers
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-11-08 12:01:14 +00:00
f666898e5d [#1480] objsvc: Remove EACL checks
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-11-08 12:01:14 +00:00
b1a31281e4 [#1480] ape: Remove SoftAPECheck flag
Previous release was EACL-compatible.
Starting from now all EACL should've been migrated to APE chains.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-11-08 12:01:14 +00:00
764450d04a [#1479] tree: Regenerate service protobufs
All checks were successful
Tests and linters / Run gofumpt (pull_request) Successful in 1m27s
DCO action / DCO (pull_request) Successful in 1m41s
Vulncheck / Vulncheck (pull_request) Successful in 2m18s
Pre-commit hooks / Pre-commit (pull_request) Successful in 2m34s
Build / Build Components (pull_request) Successful in 2m44s
Tests and linters / Staticcheck (pull_request) Successful in 3m2s
Tests and linters / gopls check (pull_request) Successful in 3m6s
Tests and linters / Lint (pull_request) Successful in 3m56s
Tests and linters / Tests (pull_request) Successful in 4m44s
Tests and linters / Tests with -race (pull_request) Successful in 6m31s
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-11-08 10:43:44 +03:00
755cae3f19 [#1479] control: Regenerate protobufs for service
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-11-08 10:43:44 +03:00
9b13a18aac [#1479] go.mod: Bump frostfs-sdk-go version
* Update version within go.mod;
* Fix deprecated frostfs-api-go/v2 package and use frostfs-sdk-go/api
  instead.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-11-08 10:43:19 +03:00
ef64930fef
[#1477] ape: Fix EC chunk test
All checks were successful
Tests and linters / Run gofumpt (pull_request) Successful in 1m33s
DCO action / DCO (pull_request) Successful in 1m46s
Vulncheck / Vulncheck (pull_request) Successful in 2m16s
Pre-commit hooks / Pre-commit (pull_request) Successful in 2m35s
Build / Build Components (pull_request) Successful in 2m50s
Tests and linters / Staticcheck (pull_request) Successful in 3m1s
Tests and linters / gopls check (pull_request) Successful in 3m6s
Tests and linters / Lint (pull_request) Successful in 3m53s
Tests and linters / Tests (pull_request) Successful in 4m43s
Tests and linters / Tests with -race (pull_request) Successful in 6m18s
Initially, this test was a check that only the container node can
assemble an EC object. But the implementation of this test was wrong.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-11-07 16:00:20 +03:00
c8fb154151
[#1475] Remove container estimation code
All checks were successful
Tests and linters / Run gofumpt (pull_request) Successful in 1m38s
Vulncheck / Vulncheck (pull_request) Successful in 5m20s
Build / Build Components (pull_request) Successful in 6m9s
Tests and linters / gopls check (pull_request) Successful in 6m1s
Tests and linters / Lint (pull_request) Successful in 6m46s
Tests and linters / Tests with -race (pull_request) Successful in 9m35s
Tests and linters / Tests (pull_request) Successful in 2m6s
Tests and linters / Staticcheck (pull_request) Successful in 2m27s
DCO action / DCO (pull_request) Successful in 1m1s
Pre-commit hooks / Pre-commit (pull_request) Successful in 1m45s
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-11-06 15:32:33 +03:00
7edec9193c [#1451] placement: Return copy of slice from container nodes cache
Nodes from cache could be changed by traverser, if no objectID specified.
So it is required to return copy of cache's slice.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-11-06 08:18:10 +00:00
3cf6ea745d [#1451] ec: Check all parts are saved
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-11-06 08:18:10 +00:00
9a77527f46 [#1451] ape: Drop unused
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-11-06 08:18:10 +00:00
5b1ba8e23d [#1451] ape: Perform strict APE checks for EC parts
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-11-06 08:18:10 +00:00
9902965ff4 [#1451] writer: Sign EC parts with node's private key
As EC put request may be processed only by container node, so sign requests
with current node private to not to perform APE checks.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-11-06 08:18:10 +00:00
33ad753302 [#1473] policer: Add tracing span
To filter HEAD requests from policer.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-11-06 08:15:11 +00:00
15102e6dfd [#1471] Replace sort.Slice in some places
`slices.SortFunc` doesn't use reflection and is a bit faster.
I have done some micro-benchmarks for `[]NodeInfo`:
```
$ benchstat -col "/func" out
goos: linux
goarch: amd64
pkg: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/pilorama
cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz
       │ sort.Slice  │           slices.SortFunc           │
       │   sec/op    │   sec/op     vs base                │
Sort-8   2.130µ ± 2%   1.253µ ± 2%  -41.20% (p=0.000 n=10)
```

Haven't included them, though, as they I don't see them being used a
lot.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-11-06 08:07:32 +00:00
6c45a17af6
[#1467] node: Break notary deposit wait after VUB
All checks were successful
DCO action / DCO (pull_request) Successful in 1m39s
Vulncheck / Vulncheck (pull_request) Successful in 2m24s
Pre-commit hooks / Pre-commit (pull_request) Successful in 2m35s
Tests and linters / Run gofumpt (pull_request) Successful in 2m30s
Build / Build Components (pull_request) Successful in 2m42s
Tests and linters / gopls check (pull_request) Successful in 3m3s
Tests and linters / Staticcheck (pull_request) Successful in 3m19s
Tests and linters / Lint (pull_request) Successful in 3m52s
Tests and linters / Tests (pull_request) Successful in 4m39s
Tests and linters / Tests with -race (pull_request) Successful in 5m44s
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-11-01 16:42:02 +03:00
d19ab43500
[#1462] node: Add off-cpu profiler
All checks were successful
Tests and linters / Run gofumpt (pull_request) Successful in 1m24s
DCO action / DCO (pull_request) Successful in 1m35s
Vulncheck / Vulncheck (pull_request) Successful in 2m17s
Pre-commit hooks / Pre-commit (pull_request) Successful in 2m49s
Build / Build Components (pull_request) Successful in 2m54s
Tests and linters / gopls check (pull_request) Successful in 3m5s
Tests and linters / Staticcheck (pull_request) Successful in 3m15s
Tests and linters / Lint (pull_request) Successful in 4m14s
Tests and linters / Tests (pull_request) Successful in 5m35s
Tests and linters / Tests with -race (pull_request) Successful in 5m43s
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-10-31 11:32:13 +03:00
c2effcc61c [#1465] Makefile: Update golangci-lint, fix warnings
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-10-31 06:39:59 +00:00
7ac0852364 [#1459] .golangci.yml: Add intrange linter, fix issues
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-10-30 15:18:22 +00:00
d28a5d2d7a
[#1448] container/ape: Ignore an error when getting a role
All checks were successful
DCO action / DCO (pull_request) Successful in 1m37s
Vulncheck / Vulncheck (pull_request) Successful in 2m8s
Pre-commit hooks / Pre-commit (pull_request) Successful in 2m38s
Tests and linters / Run gofumpt (pull_request) Successful in 2m35s
Tests and linters / Staticcheck (pull_request) Successful in 2m56s
Build / Build Components (pull_request) Successful in 3m4s
Tests and linters / gopls check (pull_request) Successful in 3m12s
Tests and linters / Lint (pull_request) Successful in 3m48s
Tests and linters / Tests (pull_request) Successful in 4m38s
Tests and linters / Tests with -race (pull_request) Successful in 6m20s
When getting a role in the APE checker for the container services,
an error may be returned if network maps of the previous two epochs
don't have enough nodes to fulfil a container placement policy.
It's a logical error, so we should ignore it.

Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-10-30 12:51:57 +03:00
87ac3c5279 [#1458] object: Make patch not set key before target construction
* `SignRequestPrivateKey` field should be initialized either within
  `newUntrustedTarget` or within `newTrustedTarget`. Otherwise, all
  requests are signed by local node key that makes impossible to perform
  patch on non-container node.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-10-29 15:20:28 +00:00
d5ee6d3039
[#1456] morph: Use DialerSource interface instead of internal struct
All checks were successful
DCO action / DCO (pull_request) Successful in 1m35s
Tests and linters / Run gofumpt (pull_request) Successful in 1m33s
Vulncheck / Vulncheck (pull_request) Successful in 2m12s
Pre-commit hooks / Pre-commit (pull_request) Successful in 2m48s
Build / Build Components (pull_request) Successful in 2m53s
Tests and linters / gopls check (pull_request) Successful in 2m59s
Tests and linters / Staticcheck (pull_request) Successful in 3m26s
Tests and linters / Lint (pull_request) Successful in 3m55s
Tests and linters / Tests (pull_request) Successful in 8m46s
Tests and linters / Tests with -race (pull_request) Successful in 8m46s
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-10-29 17:34:14 +03:00
81f4cdbb91 [#1439] object: Sort nodes by priority metrics to compute GET request
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-10-29 08:05:09 +00:00
3cd7d23f10 [#1439] node: Reduce usage of netmapAPI.NodeInfo
Remove outdated code from `netmap` service.

Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-10-29 08:05:09 +00:00
012af5cc38 [#1406] tree: Add unit-tests for ape check
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-10-29 08:04:23 +00:00
eb5336d5ff [#1406] tree: Use delete verb instead put for Remove
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-10-29 08:04:23 +00:00
bc8d79ddf9
[#1447] services/tree: Move relaying code to a separate function
All checks were successful
Tests and linters / Run gofumpt (pull_request) Successful in 3m7s
DCO action / DCO (pull_request) Successful in 3m18s
Vulncheck / Vulncheck (pull_request) Successful in 3m22s
Tests and linters / Staticcheck (pull_request) Successful in 4m25s
Pre-commit hooks / Pre-commit (pull_request) Successful in 5m11s
Tests and linters / Lint (pull_request) Successful in 5m13s
Build / Build Components (pull_request) Successful in 5m19s
Tests and linters / gopls check (pull_request) Successful in 6m12s
Tests and linters / Tests (pull_request) Successful in 6m58s
Tests and linters / Tests with -race (pull_request) Successful in 6m59s
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-10-24 10:01:03 +03:00
65a4320c75 [#1441] services/tree: Use grpc.WaitForReady option when creating client
All checks were successful
DCO action / DCO (pull_request) Successful in 1m10s
Tests and linters / Run gofumpt (pull_request) Successful in 1m28s
Vulncheck / Vulncheck (pull_request) Successful in 2m11s
Pre-commit hooks / Pre-commit (pull_request) Successful in 2m21s
Build / Build Components (pull_request) Successful in 2m33s
Tests and linters / gopls check (pull_request) Successful in 2m48s
Tests and linters / Staticcheck (pull_request) Successful in 3m7s
Tests and linters / Lint (pull_request) Successful in 3m30s
Tests and linters / Tests (pull_request) Successful in 4m25s
Tests and linters / Tests with -race (pull_request) Successful in 6m4s
Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-10-23 11:45:44 +00:00
9a260c2e64 [#1441] network/cache: Use grpc.WaitForReady option when creating client
Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-10-23 11:45:44 +00:00
8b6ec57c61 [#1440] sdnotify: Fix status for RELOADING
All checks were successful
DCO action / DCO (pull_request) Successful in 1m36s
Vulncheck / Vulncheck (pull_request) Successful in 2m16s
Pre-commit hooks / Pre-commit (pull_request) Successful in 2m28s
Tests and linters / Run gofumpt (pull_request) Successful in 2m31s
Build / Build Components (pull_request) Successful in 2m43s
Tests and linters / Staticcheck (pull_request) Successful in 2m56s
Tests and linters / gopls check (pull_request) Successful in 3m7s
Tests and linters / Lint (pull_request) Successful in 3m43s
Tests and linters / Tests (pull_request) Successful in 4m38s
Tests and linters / Tests with -race (pull_request) Successful in 5m51s
Before:
```
RELOADING=1
MONOTONIC_USEC=17951246687
STATUS=RELOADING=1
MONOTONIC_USEC=17951246687
```
After:
```
RELOADING=1
MONOTONIC_USEC=17951246687
STATUS=RELOADING
```

Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-10-21 14:25:08 +03:00
c0a2f20eee [#1422] multinet: Add metrics
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-10-17 13:15:08 +00:00
ef38420623 [#1422] ir: Add dialer source
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-10-17 13:15:08 +00:00
fbdfd503e4 [#1422] morph: Add dialer source support
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-10-17 13:15:08 +00:00
5b653aa65f [#1422] morph: Drop single client as not used
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-10-17 13:15:08 +00:00
e314f328c4 [#1422] tree: Use dialer source for tree service connections
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-10-17 13:15:08 +00:00
6c96cc2af6 [#1422] node: Use dialer source for SDK cache
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-10-17 13:15:08 +00:00
b42bcdc6fa
[#1433] services/object: Put object before auxiliary info
All checks were successful
DCO action / DCO (pull_request) Successful in 1m24s
Tests and linters / Run gofumpt (pull_request) Successful in 1m18s
Vulncheck / Vulncheck (pull_request) Successful in 1m30s
Pre-commit hooks / Pre-commit (pull_request) Successful in 2m20s
Build / Build Components (pull_request) Successful in 2m30s
Tests and linters / gopls check (pull_request) Successful in 2m50s
Tests and linters / Staticcheck (pull_request) Successful in 3m22s
Tests and linters / Lint (pull_request) Successful in 3m55s
Tests and linters / Tests (pull_request) Successful in 6m3s
Tests and linters / Tests with -race (pull_request) Successful in 6m2s
Consider the following operations ordering:
1. Inhume(with tombstone A) --> add tombstone mark for an object
2. --> new epoch arives
3. --> GCMark is added for a tombstone A, because it is unavailable
4. Put(A) --> return error, because the object already has a GCMark

It is possible, and I have successfully reproduced it with a test on the
shard level. However, the error is related to the specific
_ordering_ of operations with engine. And triggering race-conditions like
this is only possible on a shard level currently, so no tests are
written.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-10-17 14:43:13 +03:00
b0c5def2d9
[#1433] shard/test: Use WithDisabledGC() option where possible
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-10-17 14:38:07 +03:00
90f3669399 [#1342] network/cache: Add node address to error multiClient
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2024-10-15 14:05:33 +00:00
41038b2ec0 [#1431] node: Fix 'empty slice declaration using a literal'
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2024-10-15 08:53:59 +00:00
d83879d4b8 [#1431] node: Fix comment format
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2024-10-15 08:53:59 +00:00
f6582081a4 [#1431] obj_storage/metabase: Delete unused variable
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2024-10-15 08:53:59 +00:00
00b1cecfb7 [#1431] obj_storage/shard: Fix visibility of 'newMetricStore'
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2024-10-15 08:53:59 +00:00
63466d71b2 [#1431] engine: Delete unused constants
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2024-10-15 08:53:59 +00:00
d53732f663 [#1431] engine: Delete always false condition
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2024-10-15 08:53:59 +00:00
3012286452 [#1431] metabase: Fix unreachable code
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2024-10-15 08:53:59 +00:00
714ff784fa [#1431] objsvc: Use specific values in message about address mismatch
All checks were successful
Tests and linters / Run gofumpt (pull_request) Successful in 1m30s
DCO action / DCO (pull_request) Successful in 1m42s
Vulncheck / Vulncheck (pull_request) Successful in 2m3s
Pre-commit hooks / Pre-commit (pull_request) Successful in 2m19s
Build / Build Components (pull_request) Successful in 2m34s
Tests and linters / gopls check (pull_request) Successful in 2m49s
Tests and linters / Staticcheck (pull_request) Successful in 3m15s
Tests and linters / Lint (pull_request) Successful in 3m40s
Tests and linters / Tests (pull_request) Successful in 4m31s
Tests and linters / Tests with -race (pull_request) Successful in 5m53s
This makes troubleshooting failed operations much easier

Signed-off-by: Vitaliy Potyarkin <v.potyarkin@yadro.com>
2024-10-15 07:34:03 +00:00
acd6eb1815 [#1427] object: Fix Put for EC object when node unavailable
All checks were successful
DCO action / DCO (pull_request) Successful in 34s
Tests and linters / Run gofumpt (pull_request) Successful in 43s
Pre-commit hooks / Pre-commit (pull_request) Successful in 2m6s
Tests and linters / Staticcheck (pull_request) Successful in 2m13s
Vulncheck / Vulncheck (pull_request) Successful in 2m9s
Build / Build Components (pull_request) Successful in 2m23s
Tests and linters / Lint (pull_request) Successful in 3m24s
Tests and linters / Tests (pull_request) Successful in 3m36s
Tests and linters / gopls check (pull_request) Successful in 3m46s
Tests and linters / Tests with -race (pull_request) Successful in 4m19s
There might be situation when context canceled earlier than traverser move to another part of the nodes.
To avoid this, need to wait for the result from concurrent put at each traverser iteration.

Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-10-11 16:53:30 +03:00
1134760271
[#1425] services/tree: Remove eACL mentions from bearer token parsing errors
All checks were successful
DCO action / DCO (pull_request) Successful in 4m45s
Tests and linters / Run gofumpt (pull_request) Successful in 4m47s
Tests and linters / Staticcheck (pull_request) Successful in 6m5s
Tests and linters / Lint (pull_request) Successful in 6m43s
Vulncheck / Vulncheck (pull_request) Successful in 6m47s
Build / Build Components (pull_request) Successful in 6m59s
Tests and linters / gopls check (pull_request) Successful in 8m32s
Tests and linters / Tests with -race (pull_request) Successful in 9m5s
Tests and linters / Tests (pull_request) Successful in 9m24s
Pre-commit hooks / Pre-commit (pull_request) Successful in 1m28s
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-10-09 10:56:49 +03:00
02bb7159a5
[#1425] services/tree: Remove eACL processing
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-10-09 10:56:48 +03:00
cc5360a578
[#1425] morph/event: Rename eacl_test.go
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-10-09 10:56:48 +03:00
4190fba86d
[#1425] Remove SetEACL-related code
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-10-09 10:56:48 +03:00
936ebbb8e5 [#1423] metabase: Hide BucketName form upper levels
All checks were successful
Tests and linters / Run gofumpt (pull_request) Successful in 2m11s
DCO action / DCO (pull_request) Successful in 2m28s
Pre-commit hooks / Pre-commit (pull_request) Successful in 4m17s
Vulncheck / Vulncheck (pull_request) Successful in 4m25s
Tests and linters / Lint (pull_request) Successful in 4m37s
Tests and linters / gopls check (pull_request) Successful in 4m34s
Tests and linters / Tests with -race (pull_request) Successful in 4m37s
Build / Build Components (pull_request) Successful in 4m49s
Tests and linters / Staticcheck (pull_request) Successful in 6m1s
Tests and linters / Tests (pull_request) Successful in 7m12s
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-10-08 18:41:16 +03:00
c065d55ca3
[#1412] metabase: Drop logging inside transaction
All checks were successful
DCO action / DCO (pull_request) Successful in 1m13s
Tests and linters / Run gofumpt (pull_request) Successful in 1m29s
Vulncheck / Vulncheck (pull_request) Successful in 2m0s
Pre-commit hooks / Pre-commit (pull_request) Successful in 2m10s
Build / Build Components (pull_request) Successful in 2m22s
Tests and linters / gopls check (pull_request) Successful in 2m36s
Tests and linters / Staticcheck (pull_request) Successful in 3m14s
Tests and linters / Lint (pull_request) Successful in 3m31s
Tests and linters / Tests (pull_request) Successful in 4m16s
Tests and linters / Tests with -race (pull_request) Successful in 5m55s
This could lead to hang the db.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-10-08 11:41:52 +03:00
fe9f664b57
[#1412] metabase: Drop empty user attribute buckets on upgrade
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-10-08 11:41:52 +03:00
87f4b934d1
[#1412] metabase: Run bucket drop steps on upgrade concurrently
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-10-08 11:41:52 +03:00
8093e145b3
[#1412] adm: Resolve container type by metabase upgrade
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-10-08 11:41:51 +03:00
3da168f8cf
[#1412] shard: Resolve container is indexed on metabase resync
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-10-08 11:41:51 +03:00
4572fa4874
[#1412] searchSvc: Check container is indexed
For non S3 containers it is expected to use attributes index for some
attributes.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-10-08 11:41:50 +03:00
1efa64ee72
[#1412] metabase: Add search by indexed attributes
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-10-08 11:41:50 +03:00
be744ae3e6
[#1412] metabase: Index attributes for indexed containers
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-10-08 11:41:49 +03:00
1b520f7973
[#1412] engine: Add IsIndexedContainer flag
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-10-08 11:41:49 +03:00
899cd55c27
[#1412] engine: PutPrm refactoring
Use fields instead of methods.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-10-08 11:41:48 +03:00
fc032838c0 [#1215] blobstor/test: Cover iteration behaviour
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-10-07 12:44:27 +00:00
2f710d8f94 [#1414] metabase: Check parameter for CountAliveObjectsInBucket
All checks were successful
DCO action / DCO (pull_request) Successful in 1m33s
Tests and linters / Run gofumpt (pull_request) Successful in 1m28s
Pre-commit hooks / Pre-commit (pull_request) Successful in 2m10s
Vulncheck / Vulncheck (pull_request) Successful in 2m6s
Build / Build Components (pull_request) Successful in 2m32s
Tests and linters / gopls check (pull_request) Successful in 2m44s
Tests and linters / Staticcheck (pull_request) Successful in 3m6s
Tests and linters / Lint (pull_request) Successful in 3m35s
Tests and linters / Tests (pull_request) Successful in 4m21s
Tests and linters / Tests with -race (pull_request) Successful in 5m44s
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-10-04 17:11:15 +03:00
4dc9a1b300 [#1413] engine: Remove error counting methods from Shard
All checks were successful
Tests and linters / Run gofumpt (pull_request) Successful in 2m4s
DCO action / DCO (pull_request) Successful in 2m22s
Pre-commit hooks / Pre-commit (pull_request) Successful in 4m10s
Vulncheck / Vulncheck (pull_request) Successful in 4m5s
Build / Build Components (pull_request) Successful in 4m31s
Tests and linters / Staticcheck (pull_request) Successful in 4m21s
Tests and linters / gopls check (pull_request) Successful in 4m43s
Tests and linters / Lint (pull_request) Successful in 4m58s
Tests and linters / Tests (pull_request) Successful in 6m36s
Tests and linters / Tests with -race (pull_request) Successful in 7m41s
All error counting and hangling logic is present on the engine level.
Currently, we pass engine metrics with shard ID metric to shard, then
export 3 methods to manipulate these metrics.
In this commits all methods are removed and error counter is tracked on
the engine level exlusively.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-10-04 15:10:17 +03:00
963faa615a [#1413] engine: Cleanup shard error reporting
- `reportShardErrorBackground()` no longer differs from
  `reportShardError()`, reflect this in its name;
- reuse common pieces of code to make it simpler.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-10-04 15:10:17 +03:00
9a87acb87a [#1410] engine: Provide the default implementation to MetricsRegister
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-10-03 08:23:06 +00:00
9206ce5cd2 [#1410] shard: Provide the default implementation for MetricsWriter
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-10-03 08:23:06 +00:00
6c46044c9c [#1410] shard: Move MetricsWriter interface to a separate file
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-10-03 08:23:06 +00:00
01e3944b31 [#1408] metabase: Fix tests
All checks were successful
DCO action / DCO (pull_request) Successful in 1m3s
Tests and linters / Run gofumpt (pull_request) Successful in 1m21s
Pre-commit hooks / Pre-commit (pull_request) Successful in 2m27s
Vulncheck / Vulncheck (pull_request) Successful in 2m20s
Build / Build Components (pull_request) Successful in 2m41s
Tests and linters / gopls check (pull_request) Successful in 2m57s
Tests and linters / Staticcheck (pull_request) Successful in 3m5s
Tests and linters / Lint (pull_request) Successful in 3m37s
Tests and linters / Tests (pull_request) Successful in 4m31s
Tests and linters / Tests with -race (pull_request) Successful in 6m22s
No need to specify container ID for objects created with `testutil.GenerateObjectWithCID`.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-10-02 14:41:40 +03:00
434048e8d9 [#1408] metabase: Fix EC search with slow and fast filters
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-10-02 14:36:59 +03:00
9c5ddc4dfe [#1407] tree: Set ContainerOwner in parameter for CheckAPE
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-10-02 07:21:02 +00:00
d0ed29b3c7 [#1350] node: Add ability to evacuate objects from REP 1 only
All checks were successful
Tests and linters / Run gofumpt (pull_request) Successful in 50s
DCO action / DCO (pull_request) Successful in 1m41s
Pre-commit hooks / Pre-commit (pull_request) Successful in 1m53s
Vulncheck / Vulncheck (pull_request) Successful in 2m10s
Build / Build Components (pull_request) Successful in 2m43s
Tests and linters / gopls check (pull_request) Successful in 3m0s
Tests and linters / Staticcheck (pull_request) Successful in 3m5s
Tests and linters / Lint (pull_request) Successful in 3m52s
Tests and linters / Tests (pull_request) Successful in 4m40s
Tests and linters / Tests with -race (pull_request) Successful in 6m17s
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-09-27 15:41:17 +03:00
5f22ba6f38 [#1397] object: Correctly set namespace before APE check
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-09-27 11:43:29 +00:00
a5e1aa22c9 [#1394] putSvc: Fix relay
All checks were successful
Tests and linters / Run gofumpt (pull_request) Successful in 1m44s
DCO action / DCO (pull_request) Successful in 1m58s
Pre-commit hooks / Pre-commit (pull_request) Successful in 2m37s
Vulncheck / Vulncheck (pull_request) Successful in 3m11s
Tests and linters / Staticcheck (pull_request) Successful in 3m26s
Tests and linters / Lint (pull_request) Successful in 3m33s
Build / Build Components (pull_request) Successful in 3m42s
Tests and linters / gopls check (pull_request) Successful in 4m8s
Tests and linters / Tests (pull_request) Successful in 5m46s
Tests and linters / Tests with -race (pull_request) Successful in 7m31s
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-25 17:28:03 +03:00