TrueCloudLab
/
frostfs-node
19
1
Fork 20
Code Issues 93 Pull Requests 9 Actions Releases Activity
4,609 Commits
7 Branches
14 Tags
44 MiB
Commit Graph

61 Commits (v0.38.5)

Author SHA1 Message Date
Ekaterina Lebedeva d5194ab2a6 [#949] metabase: fix shard.UpdateID()
Vulncheck / Vulncheck (pull_request) Successful in 1m20s Details
DCO action / DCO (pull_request) Successful in 1m59s Details
Build / Build Components (1.21) (pull_request) Successful in 3m25s Details
Build / Build Components (1.20) (pull_request) Successful in 4m46s Details
Tests and linters / Staticcheck (pull_request) Successful in 6m5s Details
Tests and linters / gopls check (pull_request) Successful in 6m17s Details
Tests and linters / Lint (pull_request) Successful in 7m7s Details
Tests and linters / Tests (1.20) (pull_request) Successful in 8m38s Details
Tests and linters / Tests with -race (pull_request) Successful in 8m51s Details
Tests and linters / Tests (1.21) (pull_request) Successful in 8m56s Details 
metabase.Open() now reports metabase mode metric. shard.UpdateID()
needs to read shard ID from metabase => needs to open metabase.
It caused reporting 'shard undefined' metrics. To avoid reporting
wrong metrics metabase.GetShardID() was added which also opens
metabase and does not report metrics.

Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
 2024-04-01 17:27:34 +03:00
Dmitrii Stepanov 2ad433dbcb [#1005] engine: Drop shards weights
DCO action / DCO (pull_request) Successful in 4m7s Details
Vulncheck / Vulncheck (pull_request) Successful in 4m53s Details
Build / Build Components (1.21) (pull_request) Successful in 5m46s Details
Build / Build Components (1.20) (pull_request) Successful in 6m21s Details
Tests and linters / Staticcheck (pull_request) Successful in 7m45s Details
Tests and linters / Lint (pull_request) Successful in 8m44s Details
Tests and linters / Tests (1.21) (pull_request) Successful in 13m1s Details
Tests and linters / Tests (1.20) (pull_request) Successful in 15m42s Details
Tests and linters / Tests with -race (pull_request) Successful in 16m10s Details 
Unused.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2024-02-26 17:25:05 +03:00
Dmitrii Stepanov 9ba48c582d [#917] engine: Allow to detach shards 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2024-02-06 14:49:47 +03:00
Anton Nikiforov e3573de6db [#930] gc: Stop internal activity by context 
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
 2024-01-31 08:30:34 +00:00
Dmitrii Stepanov 931a5e9aaf [#918] engine: Move shard to degraded mode if metabase open failed
DCO action / DCO (pull_request) Successful in 1m55s Details
Build / Build Components (1.20) (pull_request) Successful in 2m14s Details
Vulncheck / Vulncheck (pull_request) Successful in 2m15s Details
Build / Build Components (1.21) (pull_request) Successful in 4m6s Details
Tests and linters / Tests (1.21) (pull_request) Successful in 5m58s Details
Tests and linters / Tests (1.20) (pull_request) Successful in 6m24s Details
Tests and linters / Staticcheck (pull_request) Successful in 6m11s Details
Tests and linters / Lint (pull_request) Successful in 6m37s Details
Tests and linters / Tests with -race (pull_request) Successful in 6m35s Details 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2024-01-23 11:16:40 +03:00
Dmitrii Stepanov 4b8b4da681 [#864] engine: Drop container count metric if container removed
DCO action / DCO (pull_request) Successful in 1m30s Details
Build / Build Components (1.21) (pull_request) Successful in 3m29s Details
Build / Build Components (1.20) (pull_request) Successful in 3m53s Details
Tests and linters / Lint (pull_request) Successful in 4m31s Details
Tests and linters / Tests (1.20) (pull_request) Successful in 5m1s Details
Tests and linters / Staticcheck (pull_request) Successful in 4m51s Details
Tests and linters / Tests (1.21) (pull_request) Successful in 5m13s Details
Tests and linters / Tests with -race (pull_request) Successful in 8m34s Details
Vulncheck / Vulncheck (pull_request) Successful in 58s Details 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2024-01-10 10:45:32 +03:00
Dmitrii Stepanov d75e7e9a21 [#864] engine: Drop container size metric if container deleted 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2024-01-10 10:44:54 +03:00
Dmitrii Stepanov f1c7905263 [#661] blobovniczatree: Make Rebuild concurrent 
Different DBs can be rebuild concurrently.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-12-07 15:37:33 +03:00
Dmitrii Stepanov d30ab5f29e [#838] metabase: Count user objects 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-12-06 15:44:21 +03:00
Dmitrii Stepanov 70ab1ebd54 [#763] metrics: Add container_objects_total metric
DCO action / DCO (pull_request) Successful in 3m54s Details
Build / Build Components (1.20) (pull_request) Successful in 4m58s Details
Build / Build Components (1.21) (pull_request) Successful in 5m16s Details
Vulncheck / Vulncheck (pull_request) Successful in 9m54s Details
Tests and linters / Lint (pull_request) Successful in 10m57s Details
Tests and linters / Tests (1.21) (pull_request) Successful in 12m40s Details
Tests and linters / Staticcheck (pull_request) Successful in 12m34s Details
Tests and linters / Tests with -race (pull_request) Successful in 12m48s Details
Tests and linters / Tests (1.20) (pull_request) Successful in 13m19s Details 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-11-08 12:30:57 +03:00
Ekaterina Lebedeva 8a81af5a3b [#653] Add context parameter to Open functions
DCO action / DCO (pull_request) Successful in 1m38s Details
Build / Build Components (1.20) (pull_request) Successful in 4m22s Details
Build / Build Components (1.21) (pull_request) Successful in 4m25s Details
Vulncheck / Vulncheck (pull_request) Successful in 4m56s Details
Tests and linters / Lint (pull_request) Successful in 6m1s Details
Tests and linters / Tests (1.20) (pull_request) Successful in 7m43s Details
Tests and linters / Staticcheck (pull_request) Successful in 8m1s Details
Tests and linters / Tests (1.21) (pull_request) Successful in 8m14s Details
Tests and linters / Tests with -race (pull_request) Successful in 8m32s Details 
Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
 2023-09-07 18:03:29 +03:00
Alejandro Lopez 1a0cb0f34a [#421] Try using badger for the write-cache 
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
 2023-08-07 08:16:57 +00:00
Dmitrii Stepanov 028d4a8058 [#373] blobovnicza: Add metrics 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-06-21 15:13:26 +03:00
Dmitrii Stepanov 01a0c97760 [#453] engine: Set Disabled mode to deleted shard
ci/woodpecker/pr/pre-commit Pipeline was successful Details
ci/woodpecker/push/pre-commit Pipeline was successful Details 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-06-20 12:04:07 +03:00
Dmitrii Stepanov 4449006862 [#424] metrics: Use mode value as metric value for shard 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-06-14 18:26:19 +03:00
Dmitrii Stepanov 1b364d8cf4 [#424] metrics: Refactor engine metrics 
Use histogram vector to measure request duration.
Fix naming like in Prometheus best practice.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-06-14 14:53:32 +03:00
Evgenii Stratonikov 41ab4d070e [#423] *: Use hrw.StringHash() where possible
ci/woodpecker/push/pre-commit Pipeline was successful Details 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-06-13 07:18:25 +00:00
Anton Nikiforov 263c6fdc50 [#372] node: Add metrics for the error counter in the engine
ci/woodpecker/push/pre-commit Pipeline was successful Details 
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
 2023-06-07 13:04:47 +00:00
Dmitrii Stepanov 3220c4df9f [#376] metrics: Add GC metrics
ci/woodpecker/push/pre-commit Pipeline was successful Details 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-05-31 10:22:12 +00:00
Dmitrii Stepanov 2ce43935f9 [#312] metrics: Add writecache metrcis 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-05-24 10:18:39 +00:00
Evgenii Stratonikov 4b768fd115 [#381] *: Move to sync/atomic 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-05-23 08:18:01 +03:00
Evgenii Stratonikov 0e31c12e63 [#240] logs: Move log messages to constants 
Drop duplicate entities.
Format entities.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-04-14 05:06:09 +00:00
Evgenii Stratonikov dbc3811ff4 [#191] engine: Allow to remove redundant object copies 
RemoveDuplicates() removes all duplicate object copies stored on
multiple shards. All shards are processed and the command tries to leave
a copy on the best shard according to HRW.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-04-07 17:25:50 +00:00
Alexey Vanin 20de74a505 Rename package name 
Due to source code relocation from GitHub.

Signed-off-by: Alex Vanin <a.vanin@yadro.com>
 2023-03-07 16:38:26 +03:00
Dmitrii Stepanov 6925fb4c59 [TrueCloudLab/hrw#2] node: Use typed HRW methods 
Update HRW lib and use typed HRW methods to sort shards and nodes

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-02-28 13:36:25 +03:00
Dmitrii Stepanov c3a7039801 [TrueCloudLab/hrw#2] node: Optimize shard hash 
Compute shard hash only once

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-02-28 13:36:25 +03:00
Pavel Karpy 89a0266f5e [#1794] metrics: Track physical object capacity per shard 
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
 2023-01-26 20:06:28 +03:00
Evgenii Stratonikov 9513f163aa [#2116] metrics: Track physical object capacity in the container 
Currently we track based on `PayloadSize`, because it is already stored
in the metabase and it is easier to calculate without slowing down the
whole system.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
 2023-01-26 20:06:28 +03:00
Anton Nikiforov edb1428248 [#2022] Add metric `readonly` to get shards mode 
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
 2022-12-30 11:07:35 +03:00
Pavel Karpy 923f84722a Move to frostfs-node 
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
 2022-12-28 15:04:29 +03:00
Evgenii Stratonikov 777fd32d4f [#1818] writecache: Increase error counter on background errors 
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
 2022-11-02 14:24:02 +03:00
Evgenii Stratonikov 56de2f1363 [#1969] local_object_storage: Simplify logic error construction 
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
 2022-10-31 11:41:24 +03:00
Evgenii Stratonikov fcdbf5e509 [#1969] local_object_storage: Add a type for logical errors 
All logic errors are wrapped in `logicerr.Logical` type and do not
affect shard error counter.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
 2022-10-31 11:41:24 +03:00
Pavel Karpy 31c623636d [#1863] node: Fix shard id in the object counter metrics 
If shard ID is stored in metabase (it is not the first time boot), read it,
set it, use it (not a generated one) in the metrics writer.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-10-13 13:06:41 +03:00
Pavel Karpy 8ebe95747e [#1770] node: Do not lock on shard's `Close` call 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-10-04 10:08:55 +03:00
Pavel Karpy 887afeaddb [#1770] engine: Do not lock on shard init 
Init can take a lot of time. Because the mutex is taken, all new operations
are blocked.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-10-04 10:08:55 +03:00
Pavel Karpy fbd5bc8c38 [#1770] engine: Support configuration reload 
Currently, it only supports changing the compound of the shards.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-10-04 10:08:55 +03:00
Pavel Karpy 2d7166f8d0 [#1770] shard: Move NewEpoch event routing on SE level 
It will allow dynamic shard management. Closing a shard does not allow
removing event handlers.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-10-04 10:08:55 +03:00
Pavel Karpy edef26a4fd [#1658] engine: Update metrics interfaces 
It now supports typed object counter metrics.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-09-13 21:32:37 +04:00
Pavel Karpy 9c7b3ce799 [#1658] node: Read metrics from meta on startup 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-08-25 19:20:33 +03:00
Pavel Karpy 745f72fff0 [#1658] node: Support object counter metrics 
Increment shard's object counter on successful `Put` calls and decrement on
`Delete`.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-08-25 19:20:33 +03:00
Evgenii Stratonikov 339864b720 [#1559] local_object_storage: Move `shard.Mode` to a separate package 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-07-21 17:56:06 +03:00
Pavel Karpy 63c00e785d [#1461] shard: Fix option naming 
`WitDeletedLockCallback` => `WithDeletedLockCallback`.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-07-18 11:42:25 +03:00
Pavel Karpy fed9e6679d [#1461] node: Unlock locked object on its lock removal 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-07-18 11:42:25 +03:00
Leonard Lyubich 1c30414a6c [#1454] Upgrade NeoFS SDK Go module with new IDs 
Core changes:
 * avoid package-colliding variable naming
 * avoid using pointers to IDs where unnecessary
 * avoid using `idSDK` import alias pattern
 * use `EncodeToString` for protocol string calculation and `String` for
  printing

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-06-01 17:41:45 +03:00
Leonard Lyubich 9feb5f9405 [#1175] engine: Fix `AddShard` implementation 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-03-15 13:03:23 +03:00
Leonard Lyubich 9dff07200c [#1181] local storage: Process expired locks similar to tombstones 
There is a need to process expired `LOCK` objects similar to `TOMBSTONE`
ones: we collect them on `Shard`, notify all other shards about
expiration so they could unlock the objects, and only after that mark
lockers as garbage.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-03-15 13:03:23 +03:00
Evgenii Stratonikov 19ad349b27 [#1204] shard: Save ID in the metabase 
`AddShard` must return shard id, so we temporarily open metabase
there.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-03-04 17:13:13 +03:00
Pavel Karpy 1667ec9e6d [#1131] *: Adopt SDK changes 
`object.Address` has been moved to `object/address`
`object.ID` has been moved to `object/id`

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-02-08 09:45:38 +03:00
Evgenii Stratonikov 7fb15fa1d0 [#1118] services/control: allow to reset error counter in `SetShardMode` 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-02-03 15:14:27 +03:00