8ed7a676d5
[ #1492 ] metabase: Ensure Unmarshal() is called on a cloned slice
...
The slice returned from bucket.Get() is only valid during the tx
lifetime. Cloning it is not necessary everywhere, but better safe than
sorry.
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-11-13 07:30:25 +00:00
936ebbb8e5
[ #1423 ] metabase: Hide BucketName form upper levels
...
Tests and linters / Run gofumpt (pull_request) Successful in 2m11s
DCO action / DCO (pull_request) Successful in 2m28s
Pre-commit hooks / Pre-commit (pull_request) Successful in 4m17s
Vulncheck / Vulncheck (pull_request) Successful in 4m25s
Tests and linters / Lint (pull_request) Successful in 4m37s
Tests and linters / gopls check (pull_request) Successful in 4m34s
Tests and linters / Tests with -race (pull_request) Successful in 4m37s
Build / Build Components (pull_request) Successful in 4m49s
Tests and linters / Staticcheck (pull_request) Successful in 6m1s
Tests and linters / Tests (pull_request) Successful in 7m12s
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-10-08 18:41:16 +03:00
2f710d8f94
[ #1414 ] metabase: Check parameter for CountAliveObjectsInBucket
...
DCO action / DCO (pull_request) Successful in 1m33s
Tests and linters / Run gofumpt (pull_request) Successful in 1m28s
Pre-commit hooks / Pre-commit (pull_request) Successful in 2m10s
Vulncheck / Vulncheck (pull_request) Successful in 2m6s
Build / Build Components (pull_request) Successful in 2m32s
Tests and linters / gopls check (pull_request) Successful in 2m44s
Tests and linters / Staticcheck (pull_request) Successful in 3m6s
Tests and linters / Lint (pull_request) Successful in 3m35s
Tests and linters / Tests (pull_request) Successful in 4m21s
Tests and linters / Tests with -race (pull_request) Successful in 5m44s
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-10-04 17:11:15 +03:00
d0ed29b3c7
[ #1350 ] node: Add ability to evacuate objects from REP 1 only
...
Tests and linters / Run gofumpt (pull_request) Successful in 50s
DCO action / DCO (pull_request) Successful in 1m41s
Pre-commit hooks / Pre-commit (pull_request) Successful in 1m53s
Vulncheck / Vulncheck (pull_request) Successful in 2m10s
Build / Build Components (pull_request) Successful in 2m43s
Tests and linters / gopls check (pull_request) Successful in 3m0s
Tests and linters / Staticcheck (pull_request) Successful in 3m5s
Tests and linters / Lint (pull_request) Successful in 3m52s
Tests and linters / Tests (pull_request) Successful in 4m40s
Tests and linters / Tests with -race (pull_request) Successful in 6m17s
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-09-27 15:41:17 +03:00
34e6a309c6
[ #1356 ] engine: Evacuate object from shards concurrently
...
Tests and linters / Run gofumpt (pull_request) Successful in 1m34s
DCO action / DCO (pull_request) Successful in 1m45s
Pre-commit hooks / Pre-commit (pull_request) Successful in 2m43s
Vulncheck / Vulncheck (pull_request) Successful in 2m35s
Tests and linters / Staticcheck (pull_request) Successful in 2m44s
Build / Build Components (pull_request) Successful in 3m19s
Tests and linters / gopls check (pull_request) Successful in 3m26s
Tests and linters / Lint (pull_request) Successful in 3m37s
Tests and linters / Tests with -race (pull_request) Successful in 4m44s
Tests and linters / Tests (pull_request) Successful in 5m13s
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-09-24 12:50:19 +03:00
56eeb630b6
[ #1217 ] Fix grammar mistakes and misspelling
...
DCO action / DCO (pull_request) Successful in 13m23s
Tests and linters / gopls check (pull_request) Successful in 16m29s
Tests and linters / Lint (pull_request) Successful in 17m2s
Build / Build Components (1.21) (pull_request) Successful in 16m26s
Build / Build Components (1.22) (pull_request) Successful in 16m21s
Vulncheck / Vulncheck (pull_request) Successful in 16m40s
Tests and linters / Staticcheck (pull_request) Successful in 19m9s
Pre-commit hooks / Pre-commit (pull_request) Successful in 21m37s
Tests and linters / Tests with -race (pull_request) Successful in 22m33s
Tests and linters / Tests (1.22) (pull_request) Successful in 3m46s
Tests and linters / Tests (1.21) (pull_request) Successful in 3m51s
Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2024-07-01 19:14:25 +03:00
d45d086acd
[ #1129 ] policer: Add EC chunk replication
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-05-16 16:28:48 +03:00
681b2c5fd4
[ #825 ] policer: Do not drop required linking objects
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-12 11:04:03 +00:00
79088baa06
[ #772 ] node: Apply gofumpt
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-10-31 17:03:03 +03:00
033eaf77e1
[ #496 ] node: Fix linter importas
...
Build / Build Components (1.20) (pull_request) Successful in 3m52s
Build / Build Components (1.19) (pull_request) Successful in 4m1s
ci/woodpecker/pr/pre-commit Pipeline was successful
Tests and linters / Tests with -race (pull_request) Successful in 5m36s
Tests and linters / Tests (1.20) (pull_request) Successful in 5m55s
Tests and linters / Lint (pull_request) Successful in 14m40s
Tests and linters / Tests (1.19) (pull_request) Successful in 15m29s
ci/woodpecker/push/pre-commit Pipeline was successful
Standardize the alias of the
import frostfs-sdk-go/object as objectSDK.
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2023-07-06 15:36:41 +03:00
059e9e88a2
[ #373 ] metabase: Add metrics
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-21 15:13:26 +03:00
fe01781811
[ #446 ] los: Wrap SSD errors in a separate type
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-16 14:53:31 +03:00
a9c4ba62c3
[ #248 ] metabase: Remove storage group bucket
...
Backwards compatible change, so no version increase.
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-28 17:00:43 +03:00
20de74a505
Rename package name
...
Due to source code relocation from GitHub.
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2023-03-07 16:38:26 +03:00
Pavel Karpy
923f84722a
Move to frostfs-node
...
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2022-12-28 15:04:29 +03:00
Pavel Karpy
fdeea1dfac
[ #2057 ] meta: Fix concurrent mode changes
...
Includes:
1. mode change read lock operation in every exported method that r/w the
underlying database;
2. returning `ErrDegradedMode` logical error if any exported method is
called in degraded (without a metabase) mode.
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-19 11:01:04 +03:00
Pavel Karpy
634792077e
[ #1502 ] node: Store lock object on every container node
...
Includes extending listing methods in the Storage Engine with object types.
It allows tuning replication/policer algorithms: container nodes do
not remove `LOCK` objects as redundant and try to fulfill `LOCK` placement
on the ohter container nodes.
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-19 11:01:04 +03:00
Evgenii Stratonikov
56de2f1363
[ #1969 ] local_object_storage: Simplify logic error construction
...
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-31 11:41:24 +03:00
Evgenii Stratonikov
fcdbf5e509
[ #1969 ] local_object_storage: Add a type for logical errors
...
All logic errors are wrapped in `logicerr.Logical` type and do not
affect shard error counter.
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-31 11:41:24 +03:00
Evgenii Stratonikov
ae1dab29bc
[ #1482 ] metabase: Encode database keys in binary
...
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-15 10:28:26 +03:00
Evgenii Stratonikov
1691364653
[ #1559 ] local_object_storage: Fix tests and some data races
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 17:56:06 +03:00
Evgenii Stratonikov
34d319fed2
[ #1559 ] metabase: Use Set prefix for parameter setting
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 17:56:06 +03:00
Evgenii Stratonikov
f58234aa2f
[ #1559 ] metabase: Remove public functions
...
Reduce public interface of this package. Later each result will contain
an additional status, so it makes more sense to use the same functions
and result processing everywhere.
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 17:56:06 +03:00
Evgenii Stratonikov
81f925d5a0
[ #1516 ] metabase: Optimize ListWithCursor for long listings
...
Cache buckets outside of the main loop and allocate memory for the
resulting offset only once.
```
name old time/op new time/op delta
ListWithCursor/1_item-8 6.45µs ±14% 5.79µs ±11% -10.24% (p=0.002 n=10+10)
ListWithCursor/10_items-8 20.9µs ±17% 17.3µs ± 9% -17.27% (p=0.000 n=10+10)
ListWithCursor/100_items-8 153µs ±12% 131µs ± 9% -14.63% (p=0.000 n=10+10)
name old alloc/op new alloc/op delta
ListWithCursor/1_item-8 2.31kB ± 0% 1.91kB ± 0% -17.46% (p=0.000 n=10+10)
ListWithCursor/10_items-8 6.94kB ± 0% 5.50kB ± 0% -20.78% (p=0.000 n=8+8)
ListWithCursor/100_items-8 53.3kB ± 0% 41.5kB ± 0% -22.18% (p=0.000 n=10+10)
name old allocs/op new allocs/op delta
ListWithCursor/1_item-8 40.0 ± 0% 34.0 ± 0% -15.00% (p=0.000 n=10+10)
ListWithCursor/10_items-8 121 ± 0% 100 ± 0% -17.36% (p=0.000 n=10+10)
ListWithCursor/100_items-8 930 ± 0% 758 ± 0% -18.49% (p=0.000 n=10+10)
```
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-15 20:49:41 +03:00
Evgenii Stratonikov
a93373fe71
[ #1516 ] metabase: Cache graveyard buckets in ListWithCursor
...
```
name old time/op new time/op delta
ListWithCursor/1_item-8 6.40µs ±13% 6.45µs ±14% ~ (p=0.739 n=10+10)
ListWithCursor/10_items-8 30.9µs ±21% 20.9µs ±17% -32.49% (p=0.000 n=10+10)
ListWithCursor/100_items-8 274µs ±27% 153µs ±12% -44.09% (p=0.000 n=10+10)
name old alloc/op new alloc/op delta
ListWithCursor/1_item-8 2.26kB ± 0% 2.31kB ± 0% +2.46% (p=0.000 n=10+10)
ListWithCursor/10_items-8 10.8kB ± 0% 6.9kB ± 0% -36.07% (p=0.000 n=8+8)
ListWithCursor/100_items-8 96.8kB ± 0% 53.3kB ± 0% -44.98% (p=0.000 n=10+10)
name old allocs/op new allocs/op delta
ListWithCursor/1_item-8 39.0 ± 0% 40.0 ± 0% +2.56% (p=0.000 n=10+10)
ListWithCursor/10_items-8 192 ± 0% 121 ± 0% -36.98% (p=0.000 n=10+10)
ListWithCursor/100_items-8 1.72k ± 0% 0.93k ± 0% -45.93% (p=0.000 n=10+10)
```
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
name old time/op new time/op delta
ListWithCursor/1_item-8 5.23µs ±19% 5.26µs ±15% ~ (p=0.853 n=10+10)
ListWithCursor/10_items-8 27.2µs ±15% 18.0µs ±19% -33.80% (p=0.000 n=10+10)
ListWithCursor/100_items-8 250µs ±13% 139µs ±15% -44.27% (p=0.000 n=10+10)
name old alloc/op new alloc/op delta
ListWithCursor/1_item-8 1.99kB ± 0% 2.04kB ± 0% +2.82% (p=0.000 n=8+8)
ListWithCursor/10_items-8 10.3kB ± 0% 6.4kB ± 0% -37.83% (p=0.000 n=8+10)
ListWithCursor/100_items-8 93.9kB ± 0% 50.4kB ± 0% -46.37% (p=0.000 n=10+10)
name old allocs/op new allocs/op delta
ListWithCursor/1_item-8 35.0 ± 0% 36.0 ± 0% +2.86% (p=0.000 n=10+10)
ListWithCursor/10_items-8 184 ± 0% 113 ± 0% -38.59% (p=0.000 n=10+10)
ListWithCursor/100_items-8 1.67k ± 0% 0.88k ± 0% -47.29% (p=0.000 n=10+10)
```
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-15 20:49:41 +03:00
Evgenii Stratonikov
af4db8a73b
[ #1516 ] metabase: Cache address key and do not decode address twice
...
```
name old time/op new time/op delta
ListWithCursor/1_item-8 10.6µs ± 1% 6.4µs ±13% -39.62% (p=0.000 n=7+10)
ListWithCursor/10_items-8 75.3µs ± 2% 30.9µs ±21% -58.97% (p=0.000 n=10+10)
ListWithCursor/100_items-8 726µs ± 2% 274µs ±27% -62.28% (p=0.000 n=10+10)
name old alloc/op new alloc/op delta
ListWithCursor/1_item-8 3.19kB ± 0% 2.26kB ± 0% -29.21% (p=0.000 n=10+10)
ListWithCursor/10_items-8 20.7kB ± 0% 10.8kB ± 0% -47.68% (p=0.000 n=10+8)
ListWithCursor/100_items-8 196kB ± 0% 97kB ± 0% -50.65% (p=0.000 n=7+10)
name old allocs/op new allocs/op delta
ListWithCursor/1_item-8 55.0 ± 0% 39.0 ± 0% -29.09% (p=0.000 n=10+10)
ListWithCursor/10_items-8 346 ± 0% 192 ± 0% -44.51% (p=0.000 n=10+10)
ListWithCursor/100_items-8 3.25k ± 0% 1.72k ± 0% -47.13% (p=0.000 n=9+10)
```
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-15 20:49:41 +03:00
Pavel Karpy
9b2932609b
[ #1460 ] meta: Do not use pointers as the results
...
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-06 18:03:12 +03:00
Pavel Karpy
e265ce2d52
[ #1418 ] meta: Do not use pointers as parameters
...
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-03 07:35:17 +03:00
Leonard Lyubich
1c30414a6c
[ #1454 ] Upgrade NeoFS SDK Go module with new IDs
...
Core changes:
* avoid package-colliding variable naming
* avoid using pointers to IDs where unnecessary
* avoid using `idSDK` import alias pattern
* use `EncodeToString` for protocol string calculation and `String` for
printing
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-06-01 17:41:45 +03:00
Leonard Lyubich
f15e6e888f
[ #1377 ] oid, cid: Upgrade SDK package
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-05-16 15:33:22 +03:00
Leonard Lyubich
9508633a7e
[ #1175 ] metabase: Work with LOCK objects
...
After introduction of LOCK objects (of type `TypeLock`) complicated
extended its behavior:
* create `lockers` container bucket (LCB) during PUT;
* remove object from LCB during DELETE;
* look up object in LCB during EXISTS;
* get object from LCB during GET;
* list objects from LCB during LIST with cursor;
* select objects from LCB during SELECT with '*'.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-15 13:03:23 +03:00
Pavel Karpy
1667ec9e6d
[ #1131 ] *: Adopt SDK changes
...
`object.Address` has been moved to `object/address`
`object.ID` has been moved to `object/id`
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-02-08 09:45:38 +03:00
Alex Vanin
1f911830a5
[ #948 ] engine: Define EndOfListing error in component packages
...
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-11-19 16:57:19 +03:00
Alex Vanin
c80c83b0b8
[ #948 ] metabase: Simplify cursor structure
...
Use bbolt bucket keys to seek offset in root bucket
and container buckets.
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-11-19 16:57:19 +03:00
Alex Vanin
468caa83d9
[ #948 ] Adopt sdk-go client
...
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-11-19 16:57:19 +03:00
Alex Vanin
08bdd0d561
[ #948 ] engine: Fix comments of object listing with cursor
...
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-11-19 16:57:19 +03:00
Alex Vanin
aa9ce8a853
[ #948 ] engine: Define cursor for object listing as a type
...
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-11-19 16:57:19 +03:00
Alex Vanin
0f6d8f6eea
[ #948 ] metabase: Use seek for faster cursor listing
...
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-11-19 16:57:19 +03:00
Alex Vanin
8d471c7e36
[ #948 ] core/object: Move listing objects error to core
...
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-11-19 16:57:19 +03:00
Alex Vanin
1a829a521f
[ #948 ] metabase: Add ListWithCursor method
...
ListWithCursor allows listing physically stored objects
from metabase with small chunks. Cursor tracks last
processed object, therefore new chunks are returned
on each request.
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-11-19 16:57:19 +03:00
