e3573de6db
[ #930 ] gc: Stop internal activity by context
...
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-01-31 08:30:34 +00:00
4b8b4da681
[ #864 ] engine: Drop container count metric if container removed
...
DCO action / DCO (pull_request) Successful in 1m30s
Build / Build Components (1.21) (pull_request) Successful in 3m29s
Build / Build Components (1.20) (pull_request) Successful in 3m53s
Tests and linters / Lint (pull_request) Successful in 4m31s
Tests and linters / Tests (1.20) (pull_request) Successful in 5m1s
Tests and linters / Staticcheck (pull_request) Successful in 4m51s
Tests and linters / Tests (1.21) (pull_request) Successful in 5m13s
Tests and linters / Tests with -race (pull_request) Successful in 8m34s
Vulncheck / Vulncheck (pull_request) Successful in 58s
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-01-10 10:45:32 +03:00
d75e7e9a21
[ #864 ] engine: Drop container size metric if container deleted
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-01-10 10:44:54 +03:00
dfd62ca6b1
[ #864 ] metabase: Refactor delete/inhume
...
Available -> Logic, Raw -> Phy for delete/inhume results.
Use single counter instead of vectors.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-01-09 09:59:42 +03:00
d30ab5f29e
[ #838 ] metabase: Count user objects
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-06 15:44:21 +03:00
07390ad4e3
[ #715 ] node: Unify config parameter names
...
DCO action / DCO (pull_request) Successful in 2m54s
Vulncheck / Vulncheck (pull_request) Successful in 3m11s
Tests and linters / Staticcheck (pull_request) Successful in 3m55s
Build / Build Components (1.21) (pull_request) Successful in 3m51s
Build / Build Components (1.20) (pull_request) Successful in 4m6s
Tests and linters / Tests (1.21) (pull_request) Successful in 5m10s
Tests and linters / Lint (pull_request) Successful in 5m24s
Tests and linters / Tests (1.20) (pull_request) Successful in 6m56s
Tests and linters / Tests with -race (pull_request) Successful in 7m46s
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-22 17:13:50 +03:00
70ab1ebd54
[ #763 ] metrics: Add container_objects_total metric
...
DCO action / DCO (pull_request) Successful in 3m54s
Build / Build Components (1.20) (pull_request) Successful in 4m58s
Build / Build Components (1.21) (pull_request) Successful in 5m16s
Vulncheck / Vulncheck (pull_request) Successful in 9m54s
Tests and linters / Lint (pull_request) Successful in 10m57s
Tests and linters / Tests (1.21) (pull_request) Successful in 12m40s
Tests and linters / Staticcheck (pull_request) Successful in 12m34s
Tests and linters / Tests with -race (pull_request) Successful in 12m48s
Tests and linters / Tests (1.20) (pull_request) Successful in 13m19s
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-08 12:30:57 +03:00
f2437f7ae9
[ #734 ] shard: Fix Delete method
...
DCO action / DCO (pull_request) Successful in 1m36s
Build / Build Components (1.20) (pull_request) Successful in 3m46s
Vulncheck / Vulncheck (pull_request) Successful in 3m16s
Tests and linters / Staticcheck (pull_request) Successful in 4m42s
Tests and linters / Tests (1.20) (pull_request) Successful in 6m27s
Tests and linters / Tests (1.21) (pull_request) Successful in 6m35s
Tests and linters / Tests with -race (pull_request) Successful in 6m33s
Build / Build Components (1.21) (pull_request) Successful in 13m7s
Tests and linters / Lint (pull_request) Successful in 19m14s
Due to the flushing data from the writecache to the storage
and simultaneous deletion, a partial deletion situation is possible.
So as a solution, deletion is allowed only when the object is in storage,
because object will be deleted from writecache by flush goroutine.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-10-16 17:00:18 +03:00
36759f8434
[ #668 ] shard/test: Properly check event processing
...
See https://git.frostfs.info/TrueCloudLab/frostfs-node/actions/runs/1594/jobs/2
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-07 07:39:39 +00:00
382eb8a485
[ #668 ] shard/test: Disable GC where it is not needed
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-07 07:39:39 +00:00
42696016de
[ #668 ] shard: Close stopChannel in GC
...
It is done once, but now we could read it from multiple places.
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-07 07:39:39 +00:00
033eaf77e1
[ #496 ] node: Fix linter importas
...
Build / Build Components (1.20) (pull_request) Successful in 3m52s
Build / Build Components (1.19) (pull_request) Successful in 4m1s
ci/woodpecker/pr/pre-commit Pipeline was successful
Tests and linters / Tests with -race (pull_request) Successful in 5m36s
Tests and linters / Tests (1.20) (pull_request) Successful in 5m55s
Tests and linters / Lint (pull_request) Successful in 14m40s
Tests and linters / Tests (1.19) (pull_request) Successful in 15m29s
ci/woodpecker/push/pre-commit Pipeline was successful
Standardize the alias of the
import frostfs-sdk-go/object as objectSDK.
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2023-07-06 15:36:41 +03:00
059e9e88a2
[ #373 ] metabase: Add metrics
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-21 15:13:26 +03:00
3220c4df9f
[ #376 ] metrics: Add GC metrics
...
ci/woodpecker/push/pre-commit Pipeline was successful
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-31 10:22:12 +00:00
365a7ca0f4
[ #366 ] node: Stop GC once termination signal received
...
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/push/pre-commit Pipeline was successful
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-05-29 09:35:08 +03:00
20a489bdb5
[ #393 ] gc: Use defer to mark handler done
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-26 12:14:02 +00:00
2613351008
[ #387 ] gc: Cancel GC is change mode requested
...
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/push/pre-commit Pipeline was successful
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-25 09:38:16 +03:00
869fcbf591
[ #332 ] gc: Fix expired complex object deletion
...
ci/woodpecker/push/pre-commit Pipeline was successful
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-16 12:44:57 +00:00
a181c9e434
[ #332 ] gc: Add additional logging
...
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/push/pre-commit Pipeline was successful
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-10 17:34:40 +03:00
d62c6e4ce6
[ #242 ] node: Add tracing spans
...
Add tracing spans for PUT requests.
Add tracing spans for DELETE requests.
Add tracing spans for SELECT requests.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-14 10:25:53 +00:00
0e31c12e63
[ #240 ] logs: Move log messages to constants
...
Drop duplicate entities.
Format entities.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-14 05:06:09 +00:00
8e5a0dcf27
[ #204 ] gc: Fix GC handlers start
...
ci/woodpecker/push/pre-commit Pipeline was successful
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-04 06:48:27 +00:00
a7c79c773a
[ #168 ] node: Refactor node config
...
Resolve containedctx linter for cfg
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-31 09:32:59 +03:00
9f0bce5c15
[ #183 ] gc: Fix drop expired locked simple objects
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-30 15:33:42 +03:00
5059dcc19d
[ #145 ] shard-gc: Delete expired objects after locks
...
GC deletes expired locks and objects sequentially. Expired locks and
objects are now being deleted concurrently in batches. Added a config
parameter that controls the number of concurrent workers and batch size.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-21 11:31:08 +03:00
20de74a505
Rename package name
...
Due to source code relocation from GitHub.
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2023-03-07 16:38:26 +03:00
362f24953a
[ #47 ] shard: Switch container size metric from physical to logical capacity
...
Signed-off-by: Artem Tataurov <a.tataurov@yadro.com>
2023-02-17 12:03:42 +03:00
cb016d53a6
[ #1 ] Fix comments and error messages
...
Signed-off-by: Stanislav Bogatyrev <s.bogatyrev@yadro.com>
2023-02-06 17:41:14 +03:00
Pavel Karpy
eea2892109
[ #1956 ] node: Lock shard's mode on its methods switch
...
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2022-12-30 11:07:35 +03:00
Pavel Karpy
923f84722a
Move to frostfs-node
...
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2022-12-28 15:04:29 +03:00
Evgenii Stratonikov
1e6588e761
[ #1974 ] shard: Do not panic in degraded mode
...
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-26 12:41:12 +03:00
Evgenii Stratonikov
713fdab177
[ #1907 ] shard: Return from Close after GC has stopped
...
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-25 11:54:45 +03:00
Evgenii Stratonikov
f769fc83fc
[ #1869 ] shard: Embed gcCfg as raw struct
...
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-24 13:28:26 +03:00
Pavel Karpy
f037022a7a
[ #1770 ] logger: Refactor Logger component
...
Make it store its internal `zap.Logger`'s level. Also, make all the
components to accept internal `logger.Logger` instead of `zap.Logger`; it
will simplify future refactor.
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-12 18:11:05 +03:00
Pavel Karpy
2d7166f8d0
[ #1770 ] shard: Move NewEpoch event routing on SE level
...
It will allow dynamic shard management. Closing a shard does not allow
removing event handlers.
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-04 10:08:55 +03:00
Pavel Karpy
431e331373
[ #1658 ] shard: Update metric counters
...
Use meta's operation results to change the metrics. Support typed object
counters.
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-13 21:32:37 +04:00
Evgenii Stratonikov
339864b720
[ #1559 ] local_object_storage: Move shard.Mode to a separate package
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 17:56:06 +03:00
Evgenii Stratonikov
d8ba954aff
[ #1559 ] shard: Use Set prefix for parameter setting
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 17:56:06 +03:00
Evgenii Stratonikov
34d319fed2
[ #1559 ] metabase: Use Set prefix for parameter setting
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 17:56:06 +03:00
Pavel Karpy
fed9e6679d
[ #1461 ] node: Unlock locked object on its lock removal
...
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-18 11:42:25 +03:00
Pavel Karpy
6e752f36dc
[ #1418 ] shard: Do not use pointers as parameters
...
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-03 07:35:17 +03:00
Pavel Karpy
e265ce2d52
[ #1418 ] meta: Do not use pointers as parameters
...
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-03 07:35:17 +03:00
Leonard Lyubich
1c30414a6c
[ #1454 ] Upgrade NeoFS SDK Go module with new IDs
...
Core changes:
* avoid package-colliding variable naming
* avoid using pointers to IDs where unnecessary
* avoid using `idSDK` import alias pattern
* use `EncodeToString` for protocol string calculation and `String` for
printing
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-06-01 17:41:45 +03:00
Pavel Karpy
2aec5736e0
[ #1318 ] node/gc: Fix tombstones handling
...
Handle only expired tombstones.
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-29 16:38:52 +03:00
Pavel Karpy
7799f8e4cf
[ #1318 ] engine: Change tombstone clear process
...
- Delete objects physically on tombstone's arrival;
- Store information about tombstones in the Graveyard;
- Clear Graveyard every epoch based on the information about TS in the
network.
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-29 16:38:52 +03:00
Pavel Karpy
daab30c391
[ #1318 ] metabase: Add iteration offset
...
Add offset element to the iterations over deleted objects (both the
Graveyard and the Garbage buckets).
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-29 16:38:52 +03:00
Pavel Karpy
8107c8d1a9
[ #1318 ] metabase: Separate buckets with TS and GC marks
...
It allows storing information about object in both ways at the same time:
1. Metabase should know if an object is covered by a tombstone (that is
not expired yet);
2. It should be possible to physically delete objects covered by a
tombstone immediately (mark with GC) but keep tombstone knowledge.
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-29 16:38:52 +03:00
Pavel Karpy
bd27837364
[ #1321 ] node: Register GC event channel before shard init
...
Morph "NewEpoch" event handling was registered in a closure over
`addNewEpochNotificationHandler` func. That may lead to the data race:
if a shard was initialized before the event registration, everything works
as planned, but if registration was made earlier, it was not able to
include GC handlers since a shard has not called `eventChanInit` yet and,
therefore, it has not registered handler yet.
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-25 10:39:37 +03:00
Evgenii Stratonikov
6472a170eb
[ #1143 ] shard: Introduce explicit Degraded mode
...
`Degraded` mode is set automatically after error counter is over the
threshold. `ReadOnly` mode can still be set by an administrator.
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-31 15:33:22 +03:00
Leonard Lyubich
9dff07200c
[ #1181 ] local storage: Process expired locks similar to tombstones
...
There is a need to process expired `LOCK` objects similar to `TOMBSTONE`
ones: we collect them on `Shard`, notify all other shards about
expiration so they could unlock the objects, and only after that mark
lockers as garbage.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-15 13:03:23 +03:00
