58668e6bc3
storage: Remove "could not/can't/failed to" from error messages
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-12-18 09:38:14 +03:00
1f6cf57e30
[ #1548 ] metabase: Check if EC parent is removed or expired
...
Tests and linters / Run gofumpt (push) Successful in 1m26s
Tests and linters / Staticcheck (push) Successful in 3m14s
Vulncheck / Vulncheck (push) Successful in 3m46s
Build / Build Components (push) Successful in 4m16s
Pre-commit hooks / Pre-commit (push) Successful in 4m27s
Tests and linters / Lint (push) Successful in 4m30s
Tests and linters / Tests (push) Successful in 5m22s
Tests and linters / Tests with -race (push) Successful in 5m35s
Tests and linters / gopls check (push) Successful in 5m46s
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-12-11 07:26:33 +00:00
6db46257c0
[ #1437 ] node: Use ctx for logging
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-11-13 10:36:07 +03:00
8ed7a676d5
[ #1492 ] metabase: Ensure Unmarshal() is called on a cloned slice
...
The slice returned from bucket.Get() is only valid during the tx
lifetime. Cloning it is not necessary everywhere, but better safe than
sorry.
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-11-13 07:30:25 +00:00
9b13a18aac
[ #1479 ] go.mod: Bump frostfs-sdk-go version
...
* Update version within go.mod;
* Fix deprecated frostfs-api-go/v2 package and use frostfs-sdk-go/api
instead.
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-11-08 10:43:19 +03:00
f6582081a4
[ #1431 ] obj_storage/metabase: Delete unused variable
...
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2024-10-15 08:53:59 +00:00
be744ae3e6
[ #1412 ] metabase: Index attributes for indexed containers
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-10-08 11:41:49 +03:00
c34b8acedd
[ #1312 ] Drop handling of system attributes with NeoFS prefix
...
DCO action / DCO (pull_request) Successful in 1m2s
Tests and linters / Run gofumpt (pull_request) Successful in 1m14s
Vulncheck / Vulncheck (pull_request) Successful in 1m46s
Pre-commit hooks / Pre-commit (pull_request) Successful in 2m16s
Build / Build Components (pull_request) Successful in 2m29s
Tests and linters / Staticcheck (pull_request) Successful in 2m42s
Tests and linters / gopls check (pull_request) Successful in 3m5s
Tests and linters / Lint (pull_request) Successful in 3m32s
Tests and linters / Tests (pull_request) Successful in 4m14s
Tests and linters / Tests with -race (pull_request) Successful in 5m8s
Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-09-20 11:37:18 +00:00
7d0d781db1
[ #1323 ] metabase: Drop user attribute index
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-08-22 08:21:40 +00:00
0f08a2efba
[ #1323 ] metabase: Resolve funlen linter
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-08-22 08:21:40 +00:00
7bf20c9f1f
[ #1323 ] metabase: Add expiration epoch buckets
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-08-22 08:21:40 +00:00
2542d4f5df
[ #1323 ] metabase: Drop payload checksum index
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-08-22 08:21:40 +00:00
15dae8685e
[ #1323 ] metabase: Drop ownerID index
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-08-22 08:21:40 +00:00
27caa8a72f
[ #1256 ] metabase: Put split parent object ID for EC chunks
...
DCO action / DCO (pull_request) Successful in 6m13s
Vulncheck / Vulncheck (pull_request) Successful in 6m44s
Tests and linters / gopls check (pull_request) Successful in 9m52s
Build / Build Components (1.22) (pull_request) Successful in 10m1s
Build / Build Components (1.21) (pull_request) Successful in 10m33s
Tests and linters / Staticcheck (pull_request) Successful in 11m5s
Pre-commit hooks / Pre-commit (pull_request) Successful in 14m47s
Tests and linters / Lint (pull_request) Successful in 16m21s
Tests and linters / Tests (1.22) (pull_request) Successful in 16m22s
Tests and linters / Tests with -race (pull_request) Successful in 16m32s
Tests and linters / Tests (1.21) (pull_request) Successful in 16m47s
It is required to save split parent ID too, not only split ID.
Otherwise inhume operation works incorrect: shard with last part may be skipped
and parent object will be available.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-07-17 18:08:07 +03:00
3f1961157e
[ #1163 ] metabase: Handle multiple splitInfos for EC
...
DCO action / DCO (pull_request) Successful in 3m5s
Pre-commit hooks / Pre-commit (pull_request) Successful in 4m41s
Vulncheck / Vulncheck (pull_request) Successful in 4m10s
Build / Build Components (1.21) (pull_request) Successful in 5m49s
Build / Build Components (1.22) (pull_request) Successful in 5m46s
Tests and linters / Staticcheck (pull_request) Successful in 6m37s
Tests and linters / Lint (pull_request) Successful in 7m12s
Tests and linters / gopls check (pull_request) Successful in 7m16s
Tests and linters / Tests (1.21) (pull_request) Successful in 10m11s
Tests and linters / Tests (1.22) (pull_request) Successful in 10m20s
Tests and linters / Tests with -race (pull_request) Successful in 10m21s
For REP updating split info is handled explicitly by a high-level PUT logic.
For EC it is trickier, because the address of an object we put is only
distantly related to a split info.
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-06-06 16:26:29 +03:00
2e074d3846
[ #1163 ] metabase: Properly save EC parent split ID
...
Search by SplitID should return all parts of a complex object.
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-06-05 12:40:16 +03:00
cc2449beaf
[ #1158 ] metabase: Fix EC storage schema
...
Vulncheck / Vulncheck (pull_request) Successful in 3m5s
Build / Build Components (1.21) (pull_request) Successful in 4m0s
Build / Build Components (1.22) (pull_request) Successful in 3m56s
Tests and linters / Staticcheck (pull_request) Successful in 4m54s
Tests and linters / gopls check (pull_request) Successful in 4m55s
Tests and linters / Lint (pull_request) Successful in 5m54s
Pre-commit hooks / Pre-commit (pull_request) Successful in 6m29s
DCO action / DCO (pull_request) Successful in 3m34s
Tests and linters / Tests (1.22) (pull_request) Successful in 7m28s
Tests and linters / Tests (1.21) (pull_request) Successful in 7m50s
Tests and linters / Tests with -race (pull_request) Successful in 9m25s
Do not store EC info twice.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-06-04 17:24:40 +03:00
5aacb8fc86
[ #1144 ] metabase: Save parent attributes for ec-chunks
...
Vulncheck / Vulncheck (pull_request) Successful in 4m15s
DCO action / DCO (pull_request) Successful in 4m4s
Build / Build Components (1.22) (pull_request) Successful in 5m17s
Build / Build Components (1.21) (pull_request) Successful in 5m22s
Tests and linters / Lint (pull_request) Successful in 7m6s
Tests and linters / Staticcheck (pull_request) Successful in 7m25s
Tests and linters / gopls check (pull_request) Successful in 7m19s
Pre-commit hooks / Pre-commit (pull_request) Successful in 9m39s
Tests and linters / Tests (1.22) (pull_request) Successful in 10m36s
Tests and linters / Tests (1.21) (pull_request) Successful in 10m50s
Tests and linters / Tests with -race (pull_request) Successful in 10m51s
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-05-31 19:55:32 +03:00
f8e33f8e3a
[ #1144 ] metabase: Proprely choose root OID for EC-splitted objects
...
* If EC-parent is a part of Split itself, then save to root bucket
its parent;
* If EC-parent is not a part of Split itself, then save to root bucket
OID of this EC-parent.
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-05-31 19:53:32 +03:00
f0edebea18
[ #1144 ] metabase: Support ec parent filter for Search
...
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-05-31 19:53:32 +03:00
6130650bb6
[ #1147 ] node: Implement Lock\Delete requests for EC object
...
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-05-30 08:13:04 +00:00
89a80e9a0f
[ #1141 ] metabase: Fix putUniqueIndexItem
...
* `GetECHeader` is not correct way to determine if an object's got
EC-header: `ECHeader` must be used for that.
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-05-20 13:24:08 +03:00
112a7c690f
[ #1103 ] node: Implement Get\Head requests for EC object
...
DCO action / DCO (pull_request) Successful in 1m44s
Vulncheck / Vulncheck (pull_request) Successful in 3m3s
Build / Build Components (1.21) (pull_request) Successful in 4m0s
Build / Build Components (1.22) (pull_request) Successful in 3m57s
Tests and linters / Staticcheck (pull_request) Successful in 4m46s
Tests and linters / gopls check (pull_request) Successful in 4m48s
Tests and linters / Lint (pull_request) Successful in 5m45s
Tests and linters / Tests (1.21) (pull_request) Successful in 8m57s
Tests and linters / Tests with -race (pull_request) Successful in 9m10s
Tests and linters / Tests (1.22) (pull_request) Successful in 9m20s
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-04-24 18:15:53 +03:00
918613546f
[ #1008 ] metabase: Do not update storageID on put
...
DCO action / DCO (pull_request) Successful in 4m56s
Build / Build Components (1.21) (pull_request) Successful in 7m52s
Build / Build Components (1.20) (pull_request) Successful in 8m11s
Vulncheck / Vulncheck (pull_request) Successful in 8m9s
Tests and linters / Staticcheck (pull_request) Successful in 9m58s
Tests and linters / Lint (pull_request) Successful in 10m42s
Tests and linters / Tests (1.20) (pull_request) Successful in 12m14s
Tests and linters / Tests with -race (pull_request) Successful in 12m17s
Tests and linters / Tests (1.21) (pull_request) Successful in 12m34s
There may be a race condition between put an object and
flushing the writecache:
1. Put object to the writecache
2. Writecache flushes object to the blobstore and sets blobstore's
storageID
3. Put object to the metabase, set writecache's storageID
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-02-28 11:01:50 +03:00
adf7ebab5b
[ #996 ] metabase: Speed up bucket creation
...
Most of the time it exits, e.g. when it is per-container and use on each
object PUT. Bbolt implementation first tries to create bucket and then
returns it if it exists. Create operation uses cursor and thus is not
very lightweight, we can avoid it.
```
goos: linux
goarch: amd64
pkg: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/metabase
cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz
│ old │ new │
│ sec/op │ sec/op vs base │
Put/parallel-8 174.4µ ± 3% 163.3µ ± 3% -6.39% (p=0.000 n=10)
Put/sequential-8 263.3µ ± 2% 259.0µ ± 1% -1.64% (p=0.000 n=10)
geomean 214.3µ 205.6µ -4.05%
│ old │ new │
│ B/op │ B/op vs base │
Put/parallel-8 275.3Ki ± 3% 281.1Ki ± 4% ~ (p=0.063 n=10)
Put/sequential-8 413.0Ki ± 2% 426.6Ki ± 2% +3.29% (p=0.003 n=10)
geomean 337.2Ki 346.3Ki +2.70%
│ old │ new │
│ allocs/op │ allocs/op vs base │
Put/parallel-8 678.0 ± 1% 524.5 ± 2% -22.64% (p=0.000 n=10)
Put/sequential-8 1.329k ± 0% 1.183k ± 0% -10.91% (p=0.000 n=10)
geomean 949.1 787.9 -16.98%
```
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-02-20 15:42:58 +00:00
57171907e3
[ #904 ] metabase: Return if object was actuall inserted
...
This requires to count metrics properly.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-01-17 19:04:39 +03:00
d30ab5f29e
[ #838 ] metabase: Count user objects
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-06 15:44:21 +03:00
9c98fa6152
[ #763 ] metabase: Add container objects counter
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-08 12:30:57 +03:00
79088baa06
[ #772 ] node: Apply gofumpt
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-10-31 17:03:03 +03:00
35dc64bd7b
[ #547 ] metabase: Fix datarace in tests
...
Quite an old one bf9e938a3b
2023-07-31 15:35:15 +00:00
059e9e88a2
[ #373 ] metabase: Add metrics
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-21 15:13:26 +03:00
fe01781811
[ #446 ] los: Wrap SSD errors in a separate type
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-16 14:53:31 +03:00
74578052f9
[ #412 ] node: Replace tracing package
...
Use observability module.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-01 13:23:11 +00:00
a9c4ba62c3
[ #248 ] metabase: Remove storage group bucket
...
Backwards compatible change, so no version increase.
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-28 17:00:43 +03:00
d62c6e4ce6
[ #242 ] node: Add tracing spans
...
Add tracing spans for PUT requests.
Add tracing spans for DELETE requests.
Add tracing spans for SELECT requests.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-14 10:25:53 +00:00
9e2df4b7c7
[ #203 ] node: Fix double imports
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-06 16:33:36 +03:00
0739c36a3b
[ #188 ] metabase: Refactor object put to metabase
...
Resolve funlen linter for db.put method
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-04 14:50:43 +03:00
97c36ed3ec
[ #148 ] linter: Add funlen linter
...
Long functions are hard to understand and source of errors
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-21 09:54:41 +03:00
20de74a505
Rename package name
...
Due to source code relocation from GitHub.
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2023-03-07 16:38:26 +03:00
Pavel Karpy
923f84722a
Move to frostfs-node
...
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2022-12-28 15:04:29 +03:00
Pavel Karpy
b1025bdb42
[ #2057 ] meta: Fail write operations in R/O mode
...
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-19 11:01:04 +03:00
Pavel Karpy
fdeea1dfac
[ #2057 ] meta: Fix concurrent mode changes
...
Includes:
1. mode change read lock operation in every exported method that r/w the
underlying database;
2. returning `ErrDegradedMode` logical error if any exported method is
called in degraded (without a metabase) mode.
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-19 11:01:04 +03:00
Evgenii Stratonikov
ae1dab29bc
[ #1482 ] metabase: Encode database keys in binary
...
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-15 10:28:26 +03:00
Pavel Karpy
ad47e2a985
[ #1658 ] meta: Add logic counter
...
- Meta now supports (and requires) inc/dec labeled counters
- The new logic counter is incremented on `Put` operations and is
decremented on `Inhume` and `Delete` operations that are performed on
_stored_ objects only
- Allow force counters sync. "Force" mode should be used on metabase resync
and should not be used on a regular meta start
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-13 21:32:37 +04:00
Pavel Karpy
745f72fff0
[ #1658 ] node: Support object counter metrics
...
Increment shard's object counter on successful `Put` calls and decrement on
`Delete`.
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-25 19:20:33 +03:00
Pavel Karpy
626766db08
[ #1658 ] meta: Add object counter
...
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-25 19:20:33 +03:00
Evgenii Stratonikov
73f8bb3e5f
[ #1523 ] shard: Store generic storage ID in metabase
...
Allow to extend blobstor with more storage sub-systems. Currently
objects stored in the FSTree have empty byte slice descriptor and object
from blobovnicza tree have the same id as earlier. Each such change in
the identifier formation should be accompanied with metabase version
increase.
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-08-22 13:14:19 +03:00
Pavel Karpy
156ba85326
[ #1634 ] node: Do not return expired objects
...
If an object has not been marked for removal by the GC in the current epoch
yet but has already expired, respond with `ErrObjectNotFound` api status.
Also, optimize shard iteration: a node must stop any iteration if the object
is found but gonna be removed soon.
All the checks are performed by the Metabase.
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-04 16:31:49 +03:00
Evgenii Stratonikov
1691364653
[ #1559 ] local_object_storage: Fix tests and some data races
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 17:56:06 +03:00
Evgenii Stratonikov
34d319fed2
[ #1559 ] metabase: Use Set prefix for parameter setting
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 17:56:06 +03:00
