TrueCloudLab/frostfs-node
20
1
Fork 27
Code Issues 103 Pull requests 4 Projects Releases 1 Activity Actions
5211 commits 9 branches 93 tags 50 MiB
Commit graph

64 commits

Author SHA1 Message Date
Aleksey Savchuk
72470d6b48
[#1555] local_object_storage: Rename method GetLocked -> GetLocks 
Renamed to better reflect the method's purpose of returning locks
for the specified object.

Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
 2024-12-11 15:06:37 +03:00
Aleksey Savchuk
281d65435e
[#1450] engine: Group object by shard before Inhume
All checks were successful
Tests and linters / Run gofumpt (pull_request) Successful in 2m1s
Details
DCO action / DCO (pull_request) Successful in 2m20s
Details
Vulncheck / Vulncheck (pull_request) Successful in 2m23s
Details
Build / Build Components (pull_request) Successful in 3m15s
Details
Pre-commit hooks / Pre-commit (pull_request) Successful in 3m21s
Details
Tests and linters / gopls check (pull_request) Successful in 3m21s
Details
Tests and linters / Staticcheck (pull_request) Successful in 3m49s
Details
Tests and linters / Lint (pull_request) Successful in 4m37s
Details
Tests and linters / Tests (pull_request) Successful in 5m0s
Details
Tests and linters / Tests with -race (pull_request) Successful in 6m12s
Details
Tests and linters / Run gofumpt (push) Successful in 1m46s
Details
Vulncheck / Vulncheck (push) Successful in 2m36s
Details
Pre-commit hooks / Pre-commit (push) Successful in 2m55s
Details
Build / Build Components (push) Successful in 3m8s
Details
Tests and linters / gopls check (push) Successful in 3m30s
Details
Tests and linters / Staticcheck (push) Successful in 3m36s
Details
Tests and linters / Lint (push) Successful in 4m31s
Details
Tests and linters / Tests (push) Successful in 4m56s
Details
Tests and linters / Tests with -race (push) Successful in 6m7s
Details 
```
goos: linux
goarch: amd64
pkg: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/engine
cpu: 12th Gen Intel(R) Core(TM) i5-1235U
                                 │   old.txt    │              new.txt                │
                                 │    sec/op    │   sec/op     vs base                │
InhumeMultipart/objects=1-12        11.42m ± 1%   10.71m ± 0%   -6.27% (p=0.000 n=10)
InhumeMultipart/objects=10-12       113.5m ± 0%   100.9m ± 3%  -11.08% (p=0.000 n=10)
InhumeMultipart/objects=100-12     1135.4m ± 1%   681.3m ± 2%  -40.00% (p=0.000 n=10)
InhumeMultipart/objects=1000-12     11.358 ± 0%    1.089 ± 1%  -90.41% (p=0.000 n=10)
InhumeMultipart/objects=10000-12   113.251 ± 0%    1.645 ± 1%  -98.55% (p=0.000 n=10)
geomean                              1.136        265.5m       -76.63%
```

Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
 2024-12-04 10:09:00 +03:00
Aleksey Savchuk
748edd1999
[#1450] engine: Return shard-level error if object is expired on inhume 
Since we have errors defined on the shard-level, it looks strage that we
check an error againt the shard-level error `ErrLockObjectRemoval`, but
then return the metabase-level error. Let's return the same shard-level
error instead.

Since we have errors defined on the shard-level

Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
 2024-12-04 10:06:57 +03:00
Dmitrii Stepanov
7429553266
[#1437] node: Fix contextcheck linter 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2024-11-13 10:36:10 +03:00
Dmitrii Stepanov
6db46257c0
[#1437] node: Use ctx for logging 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2024-11-13 10:36:07 +03:00
Dmitrii Stepanov
8a57c78f5f
[#1484] engine: Fix engine metrics
All checks were successful
Tests and linters / Run gofumpt (pull_request) Successful in 2m38s
Details
Pre-commit hooks / Pre-commit (pull_request) Successful in 2m51s
Details
DCO action / DCO (pull_request) Successful in 2m53s
Details
Vulncheck / Vulncheck (pull_request) Successful in 3m44s
Details
Tests and linters / gopls check (pull_request) Successful in 3m53s
Details
Build / Build Components (pull_request) Successful in 4m15s
Details
Tests and linters / Staticcheck (pull_request) Successful in 4m56s
Details
Tests and linters / Tests with -race (pull_request) Successful in 5m5s
Details
Tests and linters / Lint (pull_request) Successful in 5m47s
Details
Tests and linters / Tests (pull_request) Successful in 5m51s
Details 
1. Add forgotten metrics for client requests
2. Include execIfNotBlocked into metrics

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2024-11-11 12:59:20 +03:00
Evgenii Stratonikov
9a87acb87a [#1410] engine: Provide the default implementation to MetricsRegister 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2024-10-03 08:23:06 +00:00
Dmitrii Stepanov
eeca796d2e [#1295] engine: Log object address in case of error 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2024-08-06 09:45:43 +03:00
Anton Nikiforov
92e19feb57 [#1147] node: Use public fields for shard.ExistsPrm 
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
 2024-05-30 08:13:04 +00:00
Anton Nikiforov
6130650bb6 [#1147] node: Implement Lock\Delete requests for EC object 
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
 2024-05-30 08:13:04 +00:00
Dmitrii Stepanov
c681354afd [#938] engine: Fix container count removal 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2024-01-30 12:37:48 +03:00
Dmitrii Stepanov
4b8b4da681 [#864] engine: Drop container count metric if container removed
All checks were successful
DCO action / DCO (pull_request) Successful in 1m30s
Details
Build / Build Components (1.21) (pull_request) Successful in 3m29s
Details
Build / Build Components (1.20) (pull_request) Successful in 3m53s
Details
Tests and linters / Lint (pull_request) Successful in 4m31s
Details
Tests and linters / Tests (1.20) (pull_request) Successful in 5m1s
Details
Tests and linters / Staticcheck (pull_request) Successful in 4m51s
Details
Tests and linters / Tests (1.21) (pull_request) Successful in 5m13s
Details
Tests and linters / Tests with -race (pull_request) Successful in 8m34s
Details
Vulncheck / Vulncheck (pull_request) Successful in 58s
Details 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2024-01-10 10:45:32 +03:00
Dmitrii Stepanov
d75e7e9a21 [#864] engine: Drop container size metric if container deleted 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2024-01-10 10:44:54 +03:00
Alexander Chuprov
c1e4130020 [#146] node: Add trace_id to logs
All checks were successful
Vulncheck / Vulncheck (pull_request) Successful in 3m7s
Details
DCO action / DCO (pull_request) Successful in 3m36s
Details
Build / Build Components (1.21) (pull_request) Successful in 3m29s
Details
Build / Build Components (1.20) (pull_request) Successful in 3m37s
Details
Tests and linters / Staticcheck (pull_request) Successful in 4m39s
Details
Tests and linters / Lint (pull_request) Successful in 5m2s
Details
Tests and linters / Tests (1.21) (pull_request) Successful in 6m27s
Details
Tests and linters / Tests with -race (pull_request) Successful in 6m29s
Details
Tests and linters / Tests (1.20) (pull_request) Successful in 9m19s
Details 
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
 2023-09-27 11:05:27 +03:00
Alejandro Lopez
5b7e4a51b7 [#481] Update frostfs-sdk-go and error pointer receivers 
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
 2023-08-09 10:26:53 +00:00
Dmitrii Stepanov
1b364d8cf4 [#424] metrics: Refactor engine metrics 
Use histogram vector to measure request duration.
Fix naming like in Prometheus best practice.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-06-14 14:53:32 +03:00
Dmitrii Stepanov
74578052f9 [#412] node: Replace tracing package 
Use observability module.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-06-01 13:23:11 +00:00
Dmitrii Stepanov
d62c6e4ce6 [#242] node: Add tracing spans 
Add tracing spans for PUT requests.
Add tracing spans for DELETE requests.
Add tracing spans for SELECT requests.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-14 10:25:53 +00:00
Evgenii Stratonikov
0e31c12e63 [#240] logs: Move log messages to constants 
Drop duplicate entities.
Format entities.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-04-14 05:06:09 +00:00
Dmitrii Stepanov
0920d848d0 [#135] get-object: Add tracing spans 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-12 06:52:00 +00:00
Dmitrii Stepanov
23575e1ac0 [#210] policier: Resolve contextcheck linter 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-05 14:55:52 +03:00
Dmitrii Stepanov
9f0bce5c15 [#183] gc: Fix drop expired locked simple objects 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-03-30 15:33:42 +03:00
Pavel Karpy
97e201993b [#184] *: Unify done contexts handling 
If `ctx.Done()`, return `ctx.Err()` in every function that returns an error.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
 2023-03-29 10:34:54 +00:00
Pavel Karpy
f006f3b342 [#67] node: Make engine's IsLocked public 
It will allow reusing that method in expiration checks.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
 2023-03-16 16:20:45 +03:00
Alex Vanin
20de74a505 Rename package name 
Due to source code relocation from GitHub.

Signed-off-by: Alex Vanin <a.vanin@yadro.com>
 2023-03-07 16:38:26 +03:00
Pavel Karpy
923f84722a Move to frostfs-node 
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
 2022-12-28 15:04:29 +03:00
Pavel Karpy
3b61cb4f49 [#1502] engine: Check all shards for LOCK'ing before inhuming 
It allows keeping all the locked objects safe after metabase
resynchronization. Currently, all `LOCK` objects are broadcast to all nodes
in a container, it guarantees `LOCK` object presence in a regular situation.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-11-19 11:01:04 +03:00
Evgenii Stratonikov
6557f5d249 [#1839] engine: Handle Inhume errors properly 
If shard is in read-only or degraded mode, there is no need to increase
error counter.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
 2022-10-07 14:25:52 +03:00
Evgenii Stratonikov
af56574849 [#1819] engine: Fix error counter in Inhume 
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
 2022-10-04 10:11:52 +03:00
Evgenii Stratonikov
5321f8ef9c [#1786] engine: Unify parameter setters 
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
 2022-09-15 10:28:48 +03:00
Pavel Karpy
30341f2192 [#1687] *: Perform go fmt using go v1.19 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-08-22 18:59:57 +03:00
Pavel Karpy
156ba85326 [#1634] node: Do not return expired objects 
If an object has not been marked for removal by the GC in the current epoch
yet but has already expired, respond with `ErrObjectNotFound` api status.
Also, optimize shard iteration: a node must stop any iteration if the object
 is found but gonna be removed soon.
All the checks are performed by the Metabase.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-08-04 16:31:49 +03:00
Evgenii Stratonikov
d8ba954aff [#1559] shard: Use Set prefix for parameter setting 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-07-21 17:56:06 +03:00
Pavel Karpy
cb0bb7207c [#1461] shard: Add a separate ErrLockObjectRemoval 
Do not return `meta.ErrLockObjectRemoval` from shard's methods, add shard's
own error for that.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-07-18 11:42:25 +03:00
Pavel Karpy
558cc1193a [#1461] engine: Clarify force removal 
Document force removal behaviour in all the Storage engine parts.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-07-18 11:42:25 +03:00
Pavel Karpy
51afcc1182 [#1461] engine, policer: Force remove objects w/o container 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-07-18 11:42:25 +03:00
Pavel Karpy
18ec5d7c8e [#1461] meta: Return error on lock object removal 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-07-18 11:42:25 +03:00
Pavel Karpy
fed9e6679d [#1461] node: Unlock locked object on its lock removal 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-07-18 11:42:25 +03:00
Pavel Karpy
08bf8a68f1 [#1460] engine: Do not use pointers as the results 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-06-06 18:03:12 +03:00
Pavel Karpy
babd382ba5 [#1418] engine: Do not use pointers as parameters 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-06-03 07:35:17 +03:00
Pavel Karpy
6e752f36dc [#1418] shard: Do not use pointers as parameters 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-06-03 07:35:17 +03:00
Leonard Lyubich
1c30414a6c [#1454] Upgrade NeoFS SDK Go module with new IDs 
Core changes:
 * avoid package-colliding variable naming
 * avoid using pointers to IDs where unnecessary
 * avoid using `idSDK` import alias pattern
 * use `EncodeToString` for protocol string calculation and `String` for
  printing

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-06-01 17:41:45 +03:00
Elizaveta Chichindaeva
cc7a723d77 [#1320] English Check 
Signed-off-by: Elizaveta Chichindaeva <elizaveta@nspcc.ru>
 2022-05-11 10:40:02 +03:00
Pavel Karpy
7799f8e4cf [#1318] engine: Change tombstone clear process 
- Delete objects physically on tombstone's arrival;
- Store information about tombstones in the Graveyard;
- Clear Graveyard every epoch based on the information about TS in the
network.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-04-29 16:38:52 +03:00
Leonard Lyubich
70ffdf3478 [#1247] object: Return NOT_FOUND and ALREADY_REMOVED statuses 
Replace `ErrNotFound`/`ErrAlreadyRemoved` error from
`pkg/core/object` package with `ObjectNotFound`/`ObjectAlreadyRemoved`
one from `apistatus` package. These errors are returned by storage
node's server as NeoFS API statuses.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-03-17 16:34:00 +03:00
Leonard Lyubich
9dff07200c [#1181] local storage: Process expired locks similar to tombstones 
There is a need to process expired `LOCK` objects similar to `TOMBSTONE`
ones: we collect them on `Shard`, notify all other shards about
expiration so they could unlock the objects, and only after that mark
lockers as garbage.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-03-15 13:03:23 +03:00
Leonard Lyubich
97b6ec8150 [#1175] storage engine: Return status error on Inhume of locked object 
Make `StorageEngine.Inhume` to forward first encountered
`apistatus.ObjectLocked` error during shard processing.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-03-15 13:03:23 +03:00
Evgenii Stratonikov
c1530dec5e [#1115] shard/gc: provide parsed addresses in HandleExpiredTombstones 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-02-11 12:58:59 +03:00
Pavel Karpy
1667ec9e6d [#1131] *: Adopt SDK changes 
`object.Address` has been moved to `object/address`
`object.ID` has been moved to `object/id`

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-02-08 09:45:38 +03:00
Evgenii Stratonikov
6ad2624552 [#1118] engine: allow to set error threshold 
There are certain errors which are not expected during usual node
operation and which tell us that something is wrong with the shard.
To prevent possible data corruption, move shard in read-only mode after
amount of errors exceeded some threshold. By default no actions are performed.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-02-03 15:14:27 +03:00