Commit graph

2986 commits

Author SHA1 Message Date
7a9db5bcdd [#569] writecache: Do not wait modeMtx if mode changes
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-29 16:05:37 +03:00
32c282ca10 [#569] writecache: Refactor flush
Make single RUnlock call instead of two.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-29 16:05:37 +03:00
0cb0fc1735 [#569] writecache: Allow to seal writecache after flush
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-29 16:05:37 +03:00
b118734909 [#890] getsvc: Log node PK
Some checks failed
DCO action / DCO (pull_request) Successful in 13m55s
Vulncheck / Vulncheck (pull_request) Successful in 16m51s
Build / Build Components (1.20) (pull_request) Successful in 17m52s
Build / Build Components (1.21) (pull_request) Successful in 18m9s
Tests and linters / Tests (1.20) (pull_request) Failing after 24m34s
Tests and linters / Tests (1.21) (pull_request) Failing after 25m56s
Tests and linters / Staticcheck (pull_request) Successful in 26m46s
Tests and linters / Lint (pull_request) Successful in 28m6s
Tests and linters / Tests with -race (pull_request) Failing after 33m35s
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-29 14:24:13 +03:00
764f70634d [#881] containerSvc: Add APE validation
All checks were successful
Vulncheck / Vulncheck (pull_request) Successful in 2m40s
DCO action / DCO (pull_request) Successful in 2m27s
Build / Build Components (1.21) (pull_request) Successful in 3m22s
Tests and linters / Lint (pull_request) Successful in 4m52s
Tests and linters / Staticcheck (pull_request) Successful in 4m46s
Build / Build Components (1.20) (pull_request) Successful in 4m54s
Tests and linters / Tests (1.20) (pull_request) Successful in 11m59s
Tests and linters / Tests (1.21) (pull_request) Successful in 12m38s
Tests and linters / Tests with -race (pull_request) Successful in 13m10s
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-27 11:05:34 +03:00
8180a0664f [#887] node: Drop badger writecache implementation
All checks were successful
Vulncheck / Vulncheck (pull_request) Successful in 6m1s
Build / Build Components (1.21) (pull_request) Successful in 7m37s
Build / Build Components (1.20) (pull_request) Successful in 7m52s
Tests and linters / Staticcheck (pull_request) Successful in 8m56s
Tests and linters / Lint (pull_request) Successful in 9m26s
Tests and linters / Tests (1.21) (pull_request) Successful in 15m5s
Tests and linters / Tests with -race (pull_request) Successful in 15m7s
DCO action / DCO (pull_request) Successful in 1m1s
Tests and linters / Tests (1.20) (pull_request) Successful in 4m1s
Badger implementation isn't tested and works not well,
but requires human resources to maintain.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-22 13:00:54 +03:00
eab981bf1a [#876] Fix linter error
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-12-21 18:21:37 +03:00
d9cbb16bd3 [#866] Use TTL for blobovnicza tree cache
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-12-19 16:36:28 +00:00
7d7cf05575 [#851] ape: Initialize and use policy contract interface
* Replace inmemory policy contract mock by initialized
  policy contract interface.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-12-18 11:06:35 +00:00
bdd43f6211 [#869] object: Pass just CID to chain router
* Do not convert CID from request to native-schema resource
  format - this step is unneccessary for APE.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-12-14 11:01:20 +00:00
2d4c0a0f4a [#552] Add systemd notifications to ir service
All checks were successful
DCO action / DCO (pull_request) Successful in 3m19s
Vulncheck / Vulncheck (pull_request) Successful in 3m25s
Build / Build Components (1.21) (pull_request) Successful in 4m27s
Build / Build Components (1.20) (pull_request) Successful in 4m41s
Tests and linters / Staticcheck (pull_request) Successful in 5m5s
Tests and linters / Lint (pull_request) Successful in 5m57s
Tests and linters / Tests (1.20) (pull_request) Successful in 13m25s
Tests and linters / Tests (1.21) (pull_request) Successful in 14m13s
Tests and linters / Tests with -race (pull_request) Successful in 16m15s
Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2023-12-13 17:51:41 +03:00
eca7ac9f0d [#552] Add sdnotify package
To avoid using third-party dependencies.

Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2023-12-13 17:49:26 +03:00
05f8f49289 [#552] gofumpt changes
Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2023-12-13 15:02:25 +03:00
7eb46404a1 [#863] blobovnicza: Fix counters
All checks were successful
Vulncheck / Vulncheck (pull_request) Successful in 2m27s
DCO action / DCO (pull_request) Successful in 2m39s
Build / Build Components (1.21) (pull_request) Successful in 3m38s
Build / Build Components (1.20) (pull_request) Successful in 3m45s
Tests and linters / Staticcheck (pull_request) Successful in 4m15s
Tests and linters / Lint (pull_request) Successful in 5m47s
Tests and linters / Tests (1.20) (pull_request) Successful in 11m49s
Tests and linters / Tests (1.21) (pull_request) Successful in 12m16s
Tests and linters / Tests with -race (pull_request) Successful in 12m31s
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-13 13:34:29 +03:00
11add38e87 [#857] golangci: Add protogetter linter
All checks were successful
DCO action / DCO (pull_request) Successful in 1m38s
Vulncheck / Vulncheck (pull_request) Successful in 3m2s
Build / Build Components (1.21) (pull_request) Successful in 3m51s
Build / Build Components (1.20) (pull_request) Successful in 4m3s
Tests and linters / Staticcheck (pull_request) Successful in 5m17s
Tests and linters / Lint (pull_request) Successful in 6m11s
Tests and linters / Tests (1.20) (pull_request) Successful in 11m7s
Tests and linters / Tests (1.21) (pull_request) Successful in 11m32s
Tests and linters / Tests with -race (pull_request) Successful in 12m22s
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-12 16:27:02 +03:00
94ffe8bb45 [#857] golangci: Add testifylint linter
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-12 16:27:02 +03:00
5d7833c89b [#857] golangci: Add perfsprint linter
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-12 16:27:02 +03:00
3b7c0362a8 [#861] shard: Fix Delete object
All checks were successful
DCO action / DCO (pull_request) Successful in 12m22s
Build / Build Components (1.20) (pull_request) Successful in 12m44s
Build / Build Components (1.21) (pull_request) Successful in 13m2s
Vulncheck / Vulncheck (pull_request) Successful in 13m3s
Tests and linters / Staticcheck (pull_request) Successful in 15m32s
Tests and linters / Lint (pull_request) Successful in 16m26s
Tests and linters / Tests (1.21) (pull_request) Successful in 26m11s
Tests and linters / Tests with -race (pull_request) Successful in 28m29s
Tests and linters / Tests (1.20) (pull_request) Successful in 3m28s
It is possible that object doesn't exist in metabase.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-12 14:25:40 +03:00
681b2c5fd4 [#825] policer: Do not drop required linking objects
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-12 11:04:03 +00:00
a3ef7b58b4 [#755] innerring: Check container owner namespace
All checks were successful
Vulncheck / Vulncheck (pull_request) Successful in 1m12s
DCO action / DCO (pull_request) Successful in 2m43s
Build / Build Components (1.21) (pull_request) Successful in 4m8s
Build / Build Components (1.20) (pull_request) Successful in 4m18s
Tests and linters / Staticcheck (pull_request) Successful in 4m19s
Tests and linters / Lint (pull_request) Successful in 5m30s
Tests and linters / Tests (1.20) (pull_request) Successful in 12m2s
Tests and linters / Tests (1.21) (pull_request) Successful in 12m24s
Tests and linters / Tests with -race (pull_request) Successful in 13m4s
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-12 12:36:34 +03:00
1cd2bfe51a [#755] morph: Drop FrostFSID contract usage
Unused.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-12 12:36:34 +03:00
70e0c1e082 [#841] ir: Execute netmap.addPeerIR only for state online
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-12-11 13:14:41 +00:00
0f45e3d344 [#804] ape: Implement boltdb storage for local overrides
All checks were successful
DCO action / DCO (pull_request) Successful in 2m10s
Vulncheck / Vulncheck (pull_request) Successful in 3m26s
Build / Build Components (1.20) (pull_request) Successful in 5m41s
Build / Build Components (1.21) (pull_request) Successful in 5m44s
Tests and linters / Staticcheck (pull_request) Successful in 7m10s
Tests and linters / Lint (pull_request) Successful in 8m14s
Tests and linters / Tests (1.21) (pull_request) Successful in 14m24s
Tests and linters / Tests (1.20) (pull_request) Successful in 14m41s
Tests and linters / Tests with -race (pull_request) Successful in 14m38s
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-12-07 19:08:41 +03:00
e361e017f3 [#842] control: Pass target instead resource name
* Update policy-engine package version in go.mod, go.sum.
* Refactor CheckIfRequestPermitted: pass container target
  instead container ID.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-12-07 14:21:55 +00:00
39060382a1 [#842] control: Recieve target in gRPC methods for APE managing
* Introduce Target type and pass it to all gRPC methods
  for APE chain managing instead CID.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-12-07 14:21:55 +00:00
db49ad16cc [#826] blobovniczatree: Do not create DB's on init
All checks were successful
DCO action / DCO (pull_request) Successful in 4m10s
Build / Build Components (1.21) (pull_request) Successful in 7m30s
Vulncheck / Vulncheck (pull_request) Successful in 7m5s
Tests and linters / Staticcheck (pull_request) Successful in 10m6s
Tests and linters / Lint (pull_request) Successful in 10m26s
Build / Build Components (1.20) (pull_request) Successful in 12m39s
Tests and linters / Tests (1.20) (pull_request) Successful in 16m55s
Tests and linters / Tests (1.21) (pull_request) Successful in 17m13s
Tests and linters / Tests with -race (pull_request) Successful in 17m14s
Blobovniczas will be created on write requests.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-07 15:37:33 +03:00
ad0697adc4 [#661] blobovnicza: Compute size with record size
To get more accurate size of blobovnicza use record
size (lenght of key + lenght of data).

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-07 15:37:33 +03:00
e54dc3dc7c [#698] blobovnicza: Store counter values
Blobovnicza initialization take a long time because of bucket
Stat() call. So now blobovnicza stores counters in META bucket.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-07 15:37:33 +03:00
5e8c08da3e [#661] blobstore: Add address to error logs
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-07 15:37:33 +03:00
8911656b1a [#661] metrcis: Add rebuild percent metric
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-07 15:37:33 +03:00
2407e5f5ff [#661] blobovniczatree: Do not sort DB's and indicies
Put stores object to next active DB, so there is no need to sort DBs.
In addition, it adds unnecessary DB openings.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-07 15:37:33 +03:00
c6a739e746 [#661] blobovniczatree: Make Rebuild concurrent for objects
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-07 15:37:33 +03:00
f1c7905263 [#661] blobovniczatree: Make Rebuild concurrent
Different DBs can be rebuild concurrently.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-07 15:37:33 +03:00
d4d905ecc6 [#661] metrics: Add blobovniczatree rebuild metrics
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-07 15:37:33 +03:00
b2769ca3de [#661] blobovniczatree: Make Rebuild failover safe
Now move info stores in blobovnicza, so in case of failover
rebuild completes previous operation first.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-07 15:37:33 +03:00
da4fee2d0b [#698] blobovniczatree: Init blobovniczas concurrently
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-07 15:37:33 +03:00
422226da18 [#661] blobovniczatree: Add Rebuild implementation
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-07 15:37:32 +03:00
a531eaf8bc [#661] blobstor: Add Rebuild implementation
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-07 15:37:32 +03:00
c1667a11d2 [#661] blobovniczatree: Allow to change depth or width
Now it is possible to change depth or with of blobovniczatree.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-07 15:37:32 +03:00
484eb59893 [#661] blobovniczatree: Use .db extension for db files
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-07 15:37:32 +03:00
44552a849b [#661] shard: Add blobstor rebuilder
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-07 15:37:32 +03:00
a478050639 [#838] metabase: Resolve funlen linter
All checks were successful
DCO action / DCO (pull_request) Successful in 1m59s
Vulncheck / Vulncheck (pull_request) Successful in 3m0s
Build / Build Components (1.21) (pull_request) Successful in 3m40s
Tests and linters / Lint (pull_request) Successful in 4m57s
Tests and linters / Staticcheck (pull_request) Successful in 4m38s
Build / Build Components (1.20) (pull_request) Successful in 5m17s
Tests and linters / Tests (1.20) (pull_request) Successful in 7m1s
Tests and linters / Tests (1.21) (pull_request) Successful in 7m39s
Tests and linters / Tests with -race (pull_request) Successful in 7m46s
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-06 15:44:21 +03:00
d30ab5f29e [#838] metabase: Count user objects
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-06 15:44:21 +03:00
f314da4af3 [#838] metabase: Add user object type counter
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-06 15:44:21 +03:00
29550fe600 [#838] shard: Refactor updateMetrics method
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-06 15:44:21 +03:00
b2c63e57ba [#651] engine/test: Speedup StorageEngine_Inhume
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2023-11-30 13:19:43 +00:00
445ebcc0e7 [#651] shard/test: Speedup Shard_Delete
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2023-11-30 13:19:43 +00:00
2302e5d342 [#651] shard/test: Refactor Shard_Delete
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2023-11-30 13:19:43 +00:00
26e4f7005c [#741] treesvc: Refactor tree sync
Fix linter issues.
Add error logging.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-30 12:45:02 +00:00
b21be1abdd [#741] treesvc: Do not update sync height if some node is unavailable
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-30 12:45:02 +00:00
b215817e14 [#741] treesvc: Remove unused height variables
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-30 12:45:02 +00:00
c516c7c5f4 [#821] node: Pass user.ID by value
All checks were successful
DCO action / DCO (pull_request) Successful in 3m45s
Build / Build Components (1.21) (pull_request) Successful in 5m18s
Build / Build Components (1.20) (pull_request) Successful in 5m28s
Tests and linters / Tests (1.20) (pull_request) Successful in 7m30s
Tests and linters / Tests (1.21) (pull_request) Successful in 7m42s
Tests and linters / Lint (pull_request) Successful in 8m25s
Vulncheck / Vulncheck (pull_request) Successful in 9m22s
Tests and linters / Staticcheck (pull_request) Successful in 10m57s
Tests and linters / Tests with -race (pull_request) Successful in 16m53s
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-23 10:21:07 +03:00
07390ad4e3 [#715] node: Unify config parameter names
All checks were successful
DCO action / DCO (pull_request) Successful in 2m54s
Vulncheck / Vulncheck (pull_request) Successful in 3m11s
Tests and linters / Staticcheck (pull_request) Successful in 3m55s
Build / Build Components (1.21) (pull_request) Successful in 3m51s
Build / Build Components (1.20) (pull_request) Successful in 4m6s
Tests and linters / Tests (1.21) (pull_request) Successful in 5m10s
Tests and linters / Lint (pull_request) Successful in 5m24s
Tests and linters / Tests (1.20) (pull_request) Successful in 6m56s
Tests and linters / Tests with -race (pull_request) Successful in 7m46s
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-22 17:13:50 +03:00
8d18fa159e [#667] writecache: Fix flush test
All checks were successful
DCO action / DCO (pull_request) Successful in 1m28s
Vulncheck / Vulncheck (pull_request) Successful in 3m55s
Tests and linters / Staticcheck (pull_request) Successful in 4m15s
Build / Build Components (1.21) (pull_request) Successful in 4m40s
Build / Build Components (1.20) (pull_request) Successful in 4m54s
Tests and linters / Tests (1.20) (pull_request) Successful in 5m32s
Tests and linters / Lint (pull_request) Successful in 5m59s
Tests and linters / Tests (1.21) (pull_request) Successful in 6m59s
Tests and linters / Tests with -race (pull_request) Successful in 6m57s
Allow to disable background flusher for testing purposes.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-17 17:45:43 +03:00
02454df14a [#809] client: Refactor PrmInit, PrmDial usage
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-11-17 13:37:03 +00:00
76ff26039c [#96] node: Drop neo-go's slices package
All checks were successful
DCO action / DCO (pull_request) Successful in 2m13s
Build / Build Components (1.20) (pull_request) Successful in 5m48s
Build / Build Components (1.21) (pull_request) Successful in 5m47s
Vulncheck / Vulncheck (pull_request) Successful in 5m47s
Tests and linters / Tests (1.21) (pull_request) Successful in 7m33s
Tests and linters / Staticcheck (pull_request) Successful in 7m37s
Tests and linters / Lint (pull_request) Successful in 8m8s
Tests and linters / Tests (1.20) (pull_request) Successful in 10m38s
Tests and linters / Tests with -race (pull_request) Successful in 11m22s
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-17 13:24:04 +03:00
47286ebf32 [#805] pilorama: Fix TreeDrop
* If treeID is empty then deleting buckets for cursor may get
  invalidated. So, buckets should be gathered before deleting.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-11-17 10:21:35 +00:00
29fe8c41f3 [#655] storage: Drop ErrorHandler
All checks were successful
DCO action / DCO (pull_request) Successful in 1m42s
Build / Build Components (1.21) (pull_request) Successful in 3m23s
Vulncheck / Vulncheck (pull_request) Successful in 3m25s
Build / Build Components (1.20) (pull_request) Successful in 4m16s
Tests and linters / Staticcheck (pull_request) Successful in 4m24s
Tests and linters / Lint (pull_request) Successful in 5m2s
Tests and linters / Tests (1.20) (pull_request) Successful in 5m36s
Tests and linters / Tests (1.21) (pull_request) Successful in 6m42s
Tests and linters / Tests with -race (pull_request) Successful in 7m18s
The only one usage was for logging.
Now logging performed by storage anyway.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-16 17:27:38 +03:00
137e987a4e [#655] storage: Drop LazyHandler
LazyHandler is implemented and used incorrectly.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-16 17:27:38 +03:00
4d5be5ccb5 [#811] ape: Update policy-engine module version and rebase
All checks were successful
DCO action / DCO (pull_request) Successful in 4m23s
Vulncheck / Vulncheck (pull_request) Successful in 5m31s
Build / Build Components (1.21) (pull_request) Successful in 7m33s
Build / Build Components (1.20) (pull_request) Successful in 7m40s
Tests and linters / Staticcheck (pull_request) Successful in 8m22s
Tests and linters / Lint (pull_request) Successful in 9m23s
Tests and linters / Tests with -race (pull_request) Successful in 11m20s
Tests and linters / Tests (1.21) (pull_request) Successful in 11m32s
Tests and linters / Tests (1.20) (pull_request) Successful in 11m41s
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-11-16 11:31:37 +03:00
a952a406a2 [#506] container: Use uint64 for epoch type
It is `uint64` in netmap source interfaces and other code.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-11-14 19:00:11 +00:00
f04806ccd3 [#506] container: Use user.ID in DeletionInfo response
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-11-14 19:00:11 +00:00
8088063195 [#787] netmap: Refactor NewEpoch method
All checks were successful
DCO action / DCO (pull_request) Successful in 3m58s
Vulncheck / Vulncheck (pull_request) Successful in 4m51s
Build / Build Components (1.21) (pull_request) Successful in 5m47s
Build / Build Components (1.20) (pull_request) Successful in 5m57s
Tests and linters / Tests (1.20) (pull_request) Successful in 6m25s
Tests and linters / Staticcheck (pull_request) Successful in 6m31s
Tests and linters / Lint (pull_request) Successful in 7m15s
Tests and linters / Tests (1.21) (pull_request) Successful in 7m16s
Tests and linters / Tests with -race (pull_request) Successful in 8m30s
Split for user and control methods.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-13 17:22:31 +03:00
c8a62ffedd [#787] morph: Calculate VUB and nonce when hash is nil
All checks were successful
DCO action / DCO (pull_request) Successful in 6m41s
Vulncheck / Vulncheck (pull_request) Successful in 7m54s
Build / Build Components (1.21) (pull_request) Successful in 9m29s
Build / Build Components (1.20) (pull_request) Successful in 9m46s
Tests and linters / Staticcheck (pull_request) Successful in 12m42s
Tests and linters / Tests (1.20) (pull_request) Successful in 13m51s
Tests and linters / Tests (1.21) (pull_request) Successful in 13m50s
Tests and linters / Tests with -race (pull_request) Successful in 14m26s
Tests and linters / Lint (pull_request) Successful in 15m21s
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-13 17:13:03 +03:00
2393d13e4d [#787] morph: Return VUB for IR service calls
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-13 17:13:03 +03:00
518f3baf41 [#787] morph: Return VUB from Invoke method
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-13 17:13:03 +03:00
bdfa523487 [#787] proto: Add VUB field for IR service
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-13 17:13:03 +03:00
0f75e48138 [#796] policer: Fix tombstone objects replication
Tombstone objects must be replicated to all container nodes.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-09 13:39:33 +03:00
7cdae4f660 [#792] proto: Regenerate with fixed version
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-09 10:09:13 +00:00
9133b4389e [#788] objectsvc: Fix formatting (gofumpt)
All checks were successful
DCO action / DCO (pull_request) Successful in 3m19s
Vulncheck / Vulncheck (pull_request) Successful in 3m40s
Build / Build Components (1.21) (pull_request) Successful in 4m17s
Build / Build Components (1.20) (pull_request) Successful in 4m32s
Tests and linters / Staticcheck (pull_request) Successful in 4m46s
Tests and linters / Tests (1.21) (pull_request) Successful in 5m9s
Tests and linters / Lint (pull_request) Successful in 5m28s
Tests and linters / Tests (1.20) (pull_request) Successful in 5m24s
Tests and linters / Tests with -race (pull_request) Successful in 7m38s
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-09 10:27:32 +03:00
1b22801eed [#788] engine: Fix flaky tests
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-09 10:25:46 +03:00
3534d6d05b [#794] objectsvc: Return accidentally removed acl checks for Head
All checks were successful
DCO action / DCO (pull_request) Successful in 1m42s
Vulncheck / Vulncheck (pull_request) Successful in 3m23s
Build / Build Components (1.21) (pull_request) Successful in 4m22s
Build / Build Components (1.20) (pull_request) Successful in 5m44s
Tests and linters / Staticcheck (pull_request) Successful in 6m3s
Tests and linters / Lint (pull_request) Successful in 6m35s
Tests and linters / Tests (1.20) (pull_request) Successful in 8m32s
Tests and linters / Tests with -race (pull_request) Successful in 8m47s
Tests and linters / Tests (1.21) (pull_request) Successful in 8m54s
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-11-08 17:13:58 +03:00
66848d3288 [#770] cli: Add methods to work with APE rules via control svc
* Add methods to frostfs-cli
* Implement rpc in control service

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-11-08 13:34:03 +00:00
8e11ef46b8 [#770] object: Introduce ape chain checker for object svc
* Introduce Request type converted from RequestInfo type
  to implement policy-engine's Request interface
* Implement basic ape checker to check if a request is
  permitted to be performed
* Make put handlers use APE checker instead EACL

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-11-08 13:34:03 +00:00
5ec73fe8a0 [#770] node: Introduce ape chain source
* Provide methods to access rule chains with access
  policy engine (APE) chain source
* Initialize apeChainSource within object service
  initialization
* Share apeChainSource with control service
* Implement dummy apeChainSource instance based on
  in-memory implementation

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-11-08 13:34:03 +00:00
3a2c319b87 [#770] control: Generate gRPC methods to manipulate APE chains
* Define new types and gRPC methods to manipulate APE chains
  in control service.
* Stub gRPC handlers for the generated methods.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-11-08 13:34:03 +00:00
70ab1ebd54 [#763] metrics: Add container_objects_total metric
All checks were successful
DCO action / DCO (pull_request) Successful in 3m54s
Build / Build Components (1.20) (pull_request) Successful in 4m58s
Build / Build Components (1.21) (pull_request) Successful in 5m16s
Vulncheck / Vulncheck (pull_request) Successful in 9m54s
Tests and linters / Lint (pull_request) Successful in 10m57s
Tests and linters / Tests (1.21) (pull_request) Successful in 12m40s
Tests and linters / Staticcheck (pull_request) Successful in 12m34s
Tests and linters / Tests with -race (pull_request) Successful in 12m48s
Tests and linters / Tests (1.20) (pull_request) Successful in 13m19s
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-08 12:30:57 +03:00
9c98fa6152 [#763] metabase: Add container objects counter
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-08 12:30:57 +03:00
226e84d782 [#684] node: Add skipped objects count to evacuation result
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-07 12:17:11 +00:00
523fb3ca51 [#684] proto: Add skipped count to evacuation status response
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-07 12:17:11 +00:00
74c91eeef5 [#777] client: Refactor PrmContainerList, PrmObjectSearch usage
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-11-06 06:50:11 +00:00
20d6132f31 [#531] signSvc: Add SetMarshaledData method call
All checks were successful
Vulncheck / Vulncheck (pull_request) Successful in 1m25s
DCO action / DCO (pull_request) Successful in 1m36s
Build / Build Components (1.21) (pull_request) Successful in 2m40s
Build / Build Components (1.20) (pull_request) Successful in 3m2s
Tests and linters / Tests (1.21) (pull_request) Successful in 5m41s
Tests and linters / Staticcheck (pull_request) Successful in 5m32s
Tests and linters / Lint (pull_request) Successful in 5m59s
Tests and linters / Tests (1.20) (pull_request) Successful in 6m3s
Tests and linters / Tests with -race (pull_request) Successful in 6m6s
To reduce memory allocations add `SetMarshaledData` method call
to return already marshalled data in next `StableMarshal` calls.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-02 17:34:33 +03:00
a26483fc30 [#749] morph: Fix panic when closing morph client
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-11-01 10:48:10 +00:00
c80b46fad3 [#754] blobstor: Estimate compressability
All checks were successful
DCO action / DCO (pull_request) Successful in 1m59s
Vulncheck / Vulncheck (pull_request) Successful in 3m31s
Build / Build Components (1.20) (pull_request) Successful in 4m37s
Build / Build Components (1.21) (pull_request) Successful in 4m33s
Tests and linters / Tests (1.20) (pull_request) Successful in 4m54s
Tests and linters / Staticcheck (pull_request) Successful in 4m49s
Tests and linters / Tests with -race (pull_request) Successful in 5m9s
Tests and linters / Lint (pull_request) Successful in 6m4s
Tests and linters / Tests (1.21) (pull_request) Successful in 6m9s
Now it is possible to enable compressability estimation.
If data is likely uncompressable, it should reduce CPU time and memory.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-01 11:24:32 +03:00
05b508f79a [#772] proto: Fix file ending
All checks were successful
DCO action / DCO (pull_request) Successful in 3m47s
Vulncheck / Vulncheck (pull_request) Successful in 4m47s
Build / Build Components (1.21) (pull_request) Successful in 6m5s
Build / Build Components (1.20) (pull_request) Successful in 6m13s
Tests and linters / Tests (1.20) (pull_request) Successful in 13m9s
Tests and linters / Staticcheck (pull_request) Successful in 13m17s
Tests and linters / Lint (pull_request) Successful in 13m49s
Tests and linters / Tests (1.21) (pull_request) Successful in 14m12s
Tests and linters / Tests with -race (pull_request) Successful in 4m30s
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-10-31 17:03:04 +03:00
79088baa06 [#772] node: Apply gofumpt
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-10-31 17:03:03 +03:00
00aa6d9749 [#633] shard/test: Fix TestCounters()
Introduced in 362f24953a, forgotten to be changed because test
generator didn't provide payload size.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-10-31 12:53:28 +00:00
b8f79f4227 [#633] shard/test: Fix race conditions in TestCounters()
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-10-31 12:53:28 +00:00
261d281154 [#762] go.mod: Update sdk-go
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-10-31 11:22:29 +00:00
869518be0a [#728] writecache: Fix Badger writecache race.
All checks were successful
Build / Build Components (1.21) (pull_request) Successful in 1m47s
DCO action / DCO (pull_request) Successful in 2m1s
Vulncheck / Vulncheck (pull_request) Successful in 3m5s
Build / Build Components (1.20) (pull_request) Successful in 3m58s
Tests and linters / Tests (1.20) (pull_request) Successful in 5m43s
Tests and linters / Staticcheck (pull_request) Successful in 5m49s
Tests and linters / Lint (pull_request) Successful in 6m20s
Tests and linters / Tests with -race (pull_request) Successful in 6m8s
Tests and linters / Tests (1.21) (pull_request) Successful in 3m6s
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-10-30 18:36:41 +03:00
d4b6ebe7e7 [#725] writecache: Fix metric values
All checks were successful
DCO action / DCO (pull_request) Successful in 1m33s
Build / Build Components (1.21) (pull_request) Successful in 3m11s
Build / Build Components (1.20) (pull_request) Successful in 3m22s
Vulncheck / Vulncheck (pull_request) Successful in 3m34s
Tests and linters / Tests (1.20) (pull_request) Successful in 6m0s
Tests and linters / Tests (1.21) (pull_request) Successful in 6m1s
Tests and linters / Staticcheck (pull_request) Successful in 6m4s
Tests and linters / Tests with -race (pull_request) Successful in 3m58s
Tests and linters / Lint (pull_request) Successful in 11m43s
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-10-27 12:22:29 +03:00
121f5c4dd8 [#757] ir: Do not exclude node in maintenance mode from netmap
All checks were successful
Build / Build Components (1.20) (pull_request) Successful in 6m20s
DCO action / DCO (pull_request) Successful in 6m45s
Vulncheck / Vulncheck (pull_request) Successful in 6m20s
Build / Build Components (1.21) (pull_request) Successful in 8m34s
Tests and linters / Tests (1.20) (pull_request) Successful in 9m21s
Tests and linters / Tests (1.21) (pull_request) Successful in 9m17s
Tests and linters / Staticcheck (pull_request) Successful in 9m15s
Tests and linters / Lint (pull_request) Successful in 9m42s
Tests and linters / Tests with -race (pull_request) Successful in 11m34s
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-10-26 10:50:32 +03:00
9f7c2d8810 [#752] innerring: Simplify keyPosition()
All checks were successful
DCO action / DCO (pull_request) Successful in 3m0s
Vulncheck / Vulncheck (pull_request) Successful in 3m10s
Build / Build Components (1.20) (pull_request) Successful in 4m3s
Build / Build Components (1.21) (pull_request) Successful in 4m8s
Tests and linters / Staticcheck (pull_request) Successful in 5m13s
Tests and linters / Lint (pull_request) Successful in 5m44s
Tests and linters / Tests (1.20) (pull_request) Successful in 7m0s
Tests and linters / Tests (1.21) (pull_request) Successful in 7m7s
Tests and linters / Tests with -race (pull_request) Successful in 7m50s
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-10-25 16:06:44 +03:00
cddc58ace2 [#752] innerring: Optimize keyPosition()
```
goos: linux
goarch: amd64
pkg: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/innerring
cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz
              │      old       │                 new                 │
              │     sec/op     │   sec/op     vs base                │
KeyPosition-8   2771.50n ± 10%   40.32n ± 4%  -98.55% (p=0.000 n=10)

              │     old      │                  new                  │
              │     B/op     │     B/op      vs base                 │
KeyPosition-8   1.531Ki ± 0%   0.000Ki ± 0%  -100.00% (p=0.000 n=10)

              │    old     │                new                 │
              │ allocs/op  │ allocs/op  vs base                 │
KeyPosition-8   21.00 ± 0%   0.00 ± 0%  -100.00% (p=0.000 n=10)
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-10-25 16:06:44 +03:00
0a9830564f [#752] morph: Adopt neo-go RPC statuses
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-10-25 16:06:44 +03:00
6950312967 [#752] morph: Drop loop copy kludges
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-10-25 16:06:44 +03:00
fe1acf9e9a [#750] morph: Remove deprecated channel use
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-10-25 07:57:05 +00:00
559ad58ab1 [#642] writecache: Remove usage of close channel in bbolt
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-10-24 15:57:50 +00:00
c0b86f2d93 [#642] writecache: Remove usage of close channel in badger
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-10-24 15:57:50 +00:00
b0cf100427 [#49] node: React on SIGHUP only when node in READY state
Add more info in logs when node is going to shut down,
but initialization process still in progress.

Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-10-24 15:55:29 +00:00
58b6224dd8 [#747] client: Refactor PrmObjectPutInit usage
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-10-20 11:55:40 +00:00
12b7cf2533 [#747] client: Refactor PrmObjectPutSingle usage
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-10-20 11:55:40 +00:00
dc4d27201b [#733] morph: Fix delete container signature check
Some checks failed
DCO action / DCO (pull_request) Successful in 3m26s
Vulncheck / Vulncheck (pull_request) Successful in 4m22s
Build / Build Components (1.21) (pull_request) Successful in 5m30s
Build / Build Components (1.20) (pull_request) Successful in 6m4s
Tests and linters / Staticcheck (pull_request) Successful in 6m52s
Tests and linters / Tests with -race (pull_request) Failing after 7m16s
Tests and linters / Lint (pull_request) Successful in 9m46s
Tests and linters / Tests (1.20) (pull_request) Successful in 9m44s
Tests and linters / Tests (1.21) (pull_request) Successful in 9m41s
Committed invalid condition, it was just for debug.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-10-19 18:07:37 +03:00
189dbb01be [#733] frostfs-cli: Add control ir remove-container
All checks were successful
DCO action / DCO (pull_request) Successful in 3m51s
Vulncheck / Vulncheck (pull_request) Successful in 3m49s
Build / Build Components (1.21) (pull_request) Successful in 4m34s
Build / Build Components (1.20) (pull_request) Successful in 6m21s
Tests and linters / Tests (1.21) (pull_request) Successful in 9m1s
Tests and linters / Tests (1.20) (pull_request) Successful in 9m12s
Tests and linters / Staticcheck (pull_request) Successful in 6m40s
Tests and linters / Tests with -race (pull_request) Successful in 9m21s
Tests and linters / Lint (pull_request) Successful in 9m39s
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-10-19 16:22:18 +03:00
f2437f7ae9 [#734] shard: Fix Delete method
All checks were successful
DCO action / DCO (pull_request) Successful in 1m36s
Build / Build Components (1.20) (pull_request) Successful in 3m46s
Vulncheck / Vulncheck (pull_request) Successful in 3m16s
Tests and linters / Staticcheck (pull_request) Successful in 4m42s
Tests and linters / Tests (1.20) (pull_request) Successful in 6m27s
Tests and linters / Tests (1.21) (pull_request) Successful in 6m35s
Tests and linters / Tests with -race (pull_request) Successful in 6m33s
Build / Build Components (1.21) (pull_request) Successful in 13m7s
Tests and linters / Lint (pull_request) Successful in 19m14s
Due to the flushing data from the writecache to the storage
and simultaneous deletion, a partial deletion situation is possible.
So as a solution, deletion is allowed only when the object is in storage,
because object will be deleted from writecache by flush goroutine.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-10-16 17:00:18 +03:00
f26233b47a [#734] metabase: Include UpdateStorageID in metrics and traces
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-10-16 17:00:18 +03:00
7e0c5a55de [#734] writecache: Fix flush
Now UpdateStorageID doesn't return error in case of logical error.
If object is in graveyard or GC market, it is still required to
update storage ID.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-10-16 17:00:17 +03:00
3a997d1207 [#680] metrics: Initialize log metrics together with services
All checks were successful
DCO action / DCO (pull_request) Successful in 3m31s
Vulncheck / Vulncheck (pull_request) Successful in 3m17s
Build / Build Components (1.21) (pull_request) Successful in 3m56s
Tests and linters / Staticcheck (pull_request) Successful in 5m13s
Tests and linters / Lint (pull_request) Successful in 5m50s
Tests and linters / Tests (1.20) (pull_request) Successful in 7m31s
Build / Build Components (1.20) (pull_request) Successful in 7m44s
Tests and linters / Tests (1.21) (pull_request) Successful in 7m32s
Tests and linters / Tests with -race (pull_request) Successful in 7m33s
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-10-11 17:08:03 +03:00
994f48f8bb [#680] metrics: Export log and morph with script
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-10-11 17:08:03 +03:00
aca11d7474 [#735] policer: Allow to provide metrics from the outside
Some checks failed
DCO action / DCO (pull_request) Successful in 3m9s
Vulncheck / Vulncheck (pull_request) Successful in 3m28s
Build / Build Components (1.21) (pull_request) Successful in 4m17s
Build / Build Components (1.20) (pull_request) Successful in 4m35s
Tests and linters / Lint (pull_request) Successful in 7m19s
Tests and linters / Tests with -race (pull_request) Failing after 9m47s
Tests and linters / Tests (1.21) (pull_request) Successful in 8m0s
Tests and linters / Tests (1.20) (pull_request) Successful in 11m19s
Tests and linters / Staticcheck (pull_request) Successful in 11m4s
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-10-11 15:14:13 +03:00
5e229dc248 [#701] metrics: add metric to evaluate policer performance
Add processed objects counter in policerMetrics,
add policer field to NodeMetrics

Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2023-10-09 19:02:08 +00:00
4caa934eea [#729] containersvc: Remove load announcement
IR code was removed in 8879c6ea.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-10-09 19:01:13 +00:00
d07afd803c [#726] writecache: Fix small object flush for Badger
All checks were successful
DCO action / DCO (pull_request) Successful in 2m25s
Vulncheck / Vulncheck (pull_request) Successful in 2m41s
Build / Build Components (1.21) (pull_request) Successful in 3m2s
Build / Build Components (1.20) (pull_request) Successful in 3m17s
Tests and linters / Staticcheck (pull_request) Successful in 4m26s
Tests and linters / Tests (1.20) (pull_request) Successful in 6m9s
Tests and linters / Tests (1.21) (pull_request) Successful in 18m15s
Tests and linters / Lint (pull_request) Successful in 20m21s
Tests and linters / Tests with -race (pull_request) Successful in 18m2s
Do not marshal object twice.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-10-06 11:32:50 +03:00
997ac7cd8d [#726] writecache: Fix small object flush for BBolt
Do not marshal object twice.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-10-06 11:32:44 +03:00
c1e4130020 [#146] node: Add trace_id to logs
All checks were successful
Vulncheck / Vulncheck (pull_request) Successful in 3m7s
DCO action / DCO (pull_request) Successful in 3m36s
Build / Build Components (1.21) (pull_request) Successful in 3m29s
Build / Build Components (1.20) (pull_request) Successful in 3m37s
Tests and linters / Staticcheck (pull_request) Successful in 4m39s
Tests and linters / Lint (pull_request) Successful in 5m2s
Tests and linters / Tests (1.21) (pull_request) Successful in 6m27s
Tests and linters / Tests with -race (pull_request) Successful in 6m29s
Tests and linters / Tests (1.20) (pull_request) Successful in 9m19s
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2023-09-27 11:05:27 +03:00
3889e829e6 [#667] writecache: Add logs for report error func in tests
Signed-off-by: Airat Arifullin a.arifullin@yadro.com
2023-09-14 17:02:54 +00:00
c6af4a3ec8 [#679] engine: Do not increase error counter on meta mismatch
All checks were successful
DCO action / DCO (pull_request) Successful in 1m43s
Vulncheck / Vulncheck (pull_request) Successful in 2m0s
Tests and linters / Staticcheck (pull_request) Successful in 2m54s
Build / Build Components (1.20) (pull_request) Successful in 2m49s
Build / Build Components (1.21) (pull_request) Successful in 2m44s
Tests and linters / Tests (1.21) (pull_request) Successful in 3m43s
Tests and linters / Lint (pull_request) Successful in 3m58s
Tests and linters / Tests (1.20) (pull_request) Successful in 23m28s
Tests and linters / Tests with -race (pull_request) Successful in 29m0s
It was introduced in 69e1e6ca to help node determine faulty shards.
However, the situation is possible in a real-life scenario:
1. Object O is evacuated from shard A to B.
2. Shard A is unmounted because of lower-level errors.
3. We now have object in meta on A and in blobstor on B. Technically we
   have it in meta on shard B too, but we still got the error if B goes
   to a degraded mode.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-14 10:39:18 +03:00
8a81af5a3b [#653] Add context parameter to Open functions
All checks were successful
DCO action / DCO (pull_request) Successful in 1m38s
Build / Build Components (1.20) (pull_request) Successful in 4m22s
Build / Build Components (1.21) (pull_request) Successful in 4m25s
Vulncheck / Vulncheck (pull_request) Successful in 4m56s
Tests and linters / Lint (pull_request) Successful in 6m1s
Tests and linters / Tests (1.20) (pull_request) Successful in 7m43s
Tests and linters / Staticcheck (pull_request) Successful in 8m1s
Tests and linters / Tests (1.21) (pull_request) Successful in 8m14s
Tests and linters / Tests with -race (pull_request) Successful in 8m32s
Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2023-09-07 18:03:29 +03:00
a716db99db [#668] shard/test: Do not alter rootPath option
Supposedly, this was added to allow creating 2 different shards without
subtest. Now we use t.TempDir() everywhere, so this should not be a
problem.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-07 07:39:39 +00:00
36759f8434 [#668] shard/test: Properly check event processing
See https://git.frostfs.info/TrueCloudLab/frostfs-node/actions/runs/1594/jobs/2

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-07 07:39:39 +00:00
39879fa868 [#668] shard/test: Add dontRelease options
Most of the time we would like to close shard with minor exceptions.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-07 07:39:39 +00:00
c661ba1312 [#668] shard/test: Use sane defaults in the test constructor
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-07 07:39:39 +00:00
268adb79cb [#668] shard/test: Simplify shard construction
newCustomShard() has many parameters but only the first is obligatory.
`enableWriteCache` is left as-is, because it directly affects the
functionality.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-07 07:39:39 +00:00
429f941cda [#668] shard/test: Release shard in t.Cleanup()
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-07 07:39:39 +00:00
382eb8a485 [#668] shard/test: Disable GC where it is not needed
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-07 07:39:39 +00:00
42696016de [#668] shard: Close stopChannel in GC
It is done once, but now we could read it from multiple places.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-07 07:39:39 +00:00
bdecfbc1be [#668] shard/test: Move tests to the main package
Semantic patch (also, duplicate definitions are removed):
```
@@
var e identifier
@@
-import "git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard"

-shard.e
+e
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-07 07:39:39 +00:00
aa23c6a83a [#668] shard/test: Remove subtest from TestCounters
Otherwise, individual tests cannot be run.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-07 07:39:39 +00:00
da8f384324 [#668] shard/test: Fix typo in existence
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-07 07:39:39 +00:00
aeeb8193d2 [#676] node: Fix header source creation when checking eacl
Some checks failed
DCO action / DCO (pull_request) Successful in 2m55s
Build / Build Components (1.20) (pull_request) Successful in 4m53s
Vulncheck / Vulncheck (pull_request) Successful in 4m36s
Tests and linters / Staticcheck (pull_request) Successful in 6m35s
Tests and linters / Tests (1.21) (pull_request) Successful in 7m7s
Tests and linters / Tests (1.20) (pull_request) Successful in 7m47s
Tests and linters / Tests with -race (pull_request) Failing after 10m7s
Build / Build Components (1.21) (pull_request) Successful in 11m3s
Tests and linters / Lint (pull_request) Successful in 17m34s
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-09-06 17:06:54 +03:00
88d50e4c77 [#656] policer: Add "bad" testcase
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-09-06 08:04:59 +00:00
a54b4472de [#674] network: Close connections on address updates
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-04 15:34:24 +03:00
7456c8556a [#536] blobovnicza: Add blobovniczatree DB cache
All checks were successful
DCO action / DCO (pull_request) Successful in 1m33s
Vulncheck / Vulncheck (pull_request) Successful in 2m43s
Build / Build Components (1.20) (pull_request) Successful in 3m40s
Tests and linters / Staticcheck (pull_request) Successful in 3m59s
Tests and linters / Tests (1.20) (pull_request) Successful in 5m44s
Tests and linters / Tests (1.21) (pull_request) Successful in 6m3s
Tests and linters / Tests with -race (pull_request) Successful in 6m28s
Build / Build Components (1.21) (pull_request) Successful in 12m34s
Tests and linters / Lint (pull_request) Successful in 12m51s
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-09-01 13:53:11 +03:00
c672f59ab8 [#536] blobovnicza: Drop cache
Each blobovnicza instance is opened
while is in use.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-09-01 13:51:26 +03:00
b9b86d2ec8 [#666] shard/test: Fix data race in metrics tests
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-31 08:39:42 +00:00
4dff9555f1 [#568] writecache: Improve flushing scheme for badger
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-08-30 17:22:28 +00:00
806cc13d9f [#658] client: Refactor PrmObjectGet/Head/Range usage
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-08-30 17:13:23 +00:00
1daef2ceeb [#660] writecache: Fix remaining addr2key uses
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-08-30 17:12:33 +00:00
fe5aa06a75 [#665] node: Bind length of copies number to number of replicas
Allow to use one digit in copies number array for backward compatibility.

Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-08-30 17:11:55 +00:00
55b82e744b [#529] objectcore: Use common sender classifier
All checks were successful
DCO action / DCO (pull_request) Successful in 2m19s
Vulncheck / Vulncheck (pull_request) Successful in 3m5s
Build / Build Components (1.21) (pull_request) Successful in 4m8s
Build / Build Components (1.20) (pull_request) Successful in 4m24s
Tests and linters / Tests (1.20) (pull_request) Successful in 4m57s
Tests and linters / Staticcheck (pull_request) Successful in 4m43s
Tests and linters / Tests (1.21) (pull_request) Successful in 5m2s
Tests and linters / Lint (pull_request) Successful in 5m21s
Tests and linters / Tests with -race (pull_request) Successful in 6m17s
Use common sender classifier for ACL service and format validator.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-08-29 10:33:06 +03:00
ae81d6660a [#529] objectcore: Fix object content validation
There are old objects where the owner of the object
may not match the one who issued the token.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-08-29 10:33:06 +03:00
ab2614ec2d [#528] objectcore: Validate token issuer
Add token issuer against object owner validation.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-08-29 10:10:10 +03:00
4ea0df77d0 [#574] policer: Check if the container was really removed
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-08-28 14:21:38 +00:00
554ff2c06b [#574] core: Extend Source interface with DeletionInfo method
* Introduce common method EverExisted
* Define DeletionInfo for struct that must implement Source
* Refactor tree srv

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-08-28 14:21:38 +00:00
9072772a09 [#649] shard/test: Increase GC remover interval
This was set in #348 to speed up tests.
It seems 100ms doesn't increase overall test time,
but it reduces the amount of logs by 100x factor.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-25 10:10:25 +00:00
c4db8e7690 [#637] shard/test: Fix data race
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-25 10:10:25 +00:00
f8ba60aa0c [#648] objsvc/delete: Handle errors in Go style
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-25 09:45:35 +00:00
d2084ece41 [#648] objsvc/delete: Remove redundant logs
We never propagate delete requests to the container node, because
tombstone broadcast is done via PUT. No need to pollute logs.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-25 09:45:35 +00:00
40b556fc19 [#647] objsvc/search: Improve testing coverage
All checks were successful
DCO action / DCO (pull_request) Successful in 1m32s
Build / Build Components (1.20) (pull_request) Successful in 3m54s
Tests and linters / Staticcheck (pull_request) Successful in 3m44s
Tests and linters / Tests (1.21) (pull_request) Successful in 4m18s
Tests and linters / Tests (1.20) (pull_request) Successful in 4m35s
Tests and linters / Lint (pull_request) Successful in 4m42s
Vulncheck / Vulncheck (pull_request) Successful in 5m11s
Tests and linters / Tests with -race (pull_request) Successful in 6m23s
Build / Build Components (1.21) (pull_request) Successful in 8m38s
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-25 10:40:01 +03:00
4db2cbc927 [#647] objsvc/search: Wrap in uniqueIDWriter during parameter setting
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-25 10:40:01 +03:00