TrueCloudLab/frostfs-node
19
1
Fork 27
Code Issues 91 Pull requests 7 Projects Releases 1 Activity Actions
4332 commits 8 branches 83 tags 53 MiB
Commit graph

91 commits

Author SHA1 Message Date
Dmitrii Stepanov
ad0697adc4 [#661] blobovnicza: Compute size with record size 
To get more accurate size of blobovnicza use record
size (lenght of key + lenght of data).

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-12-07 15:37:33 +03:00
Dmitrii Stepanov
e54dc3dc7c [#698] blobovnicza: Store counter values 
Blobovnicza initialization take a long time because of bucket
Stat() call. So now blobovnicza stores counters in META bucket.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-12-07 15:37:33 +03:00
Dmitrii Stepanov
b2769ca3de [#661] blobovniczatree: Make Rebuild failover safe 
Now move info stores in blobovnicza, so in case of failover
rebuild completes previous operation first.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-12-07 15:37:33 +03:00
Dmitrii Stepanov
484eb59893 [#661] blobovniczatree: Use .db extension for db files 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-12-07 15:37:32 +03:00
Dmitrii Stepanov
76ff26039c [#96] node: Drop neo-go's slices package
All checks were successful
DCO action / DCO (pull_request) Successful in 2m13s
Details
Build / Build Components (1.20) (pull_request) Successful in 5m48s
Details
Build / Build Components (1.21) (pull_request) Successful in 5m47s
Details
Vulncheck / Vulncheck (pull_request) Successful in 5m47s
Details
Tests and linters / Tests (1.21) (pull_request) Successful in 7m33s
Details
Tests and linters / Staticcheck (pull_request) Successful in 7m37s
Details
Tests and linters / Lint (pull_request) Successful in 8m8s
Details
Tests and linters / Tests (1.20) (pull_request) Successful in 10m38s
Details
Tests and linters / Tests with -race (pull_request) Successful in 11m22s
Details 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-11-17 13:24:04 +03:00
Dmitrii Stepanov
79088baa06 [#772] node: Apply gofumpt 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-10-31 17:03:03 +03:00
Alexander Chuprov
c1e4130020 [#146] node: Add trace_id to logs
All checks were successful
Vulncheck / Vulncheck (pull_request) Successful in 3m7s
Details
DCO action / DCO (pull_request) Successful in 3m36s
Details
Build / Build Components (1.21) (pull_request) Successful in 3m29s
Details
Build / Build Components (1.20) (pull_request) Successful in 3m37s
Details
Tests and linters / Staticcheck (pull_request) Successful in 4m39s
Details
Tests and linters / Lint (pull_request) Successful in 5m2s
Details
Tests and linters / Tests (1.21) (pull_request) Successful in 6m27s
Details
Tests and linters / Tests with -race (pull_request) Successful in 6m29s
Details
Tests and linters / Tests (1.20) (pull_request) Successful in 9m19s
Details 
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
 2023-09-27 11:05:27 +03:00
Dmitrii Stepanov
c672f59ab8 [#536] blobovnicza: Drop cache 
Each blobovnicza instance is opened
while is in use.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-09-01 13:51:26 +03:00
Evgenii Stratonikov
96e690883f [#638] Unify test loggers 
In some places we have debug=false, in others debug=true.
Let's be consistent.

Semantic patch:
```
@@
@@
-test.NewLogger(..., false)
+test.NewLogger(..., true)
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-08-23 11:21:05 +00:00
Dmitrii Stepanov
f2811f8585 [#602] metrics: Add blobovnicza items counter
All checks were successful
Vulncheck / Vulncheck (pull_request) Successful in 1m29s
Details
Build / Build Components (1.21) (pull_request) Successful in 3m23s
Details
DCO action / DCO (pull_request) Successful in 3m50s
Details
Tests and linters / Tests (1.21) (pull_request) Successful in 4m27s
Details
Tests and linters / Lint (pull_request) Successful in 4m48s
Details
Tests and linters / Tests (1.20) (pull_request) Successful in 5m6s
Details
Tests and linters / Staticcheck (pull_request) Successful in 5m8s
Details
Tests and linters / Tests with -race (pull_request) Successful in 5m38s
Details
Build / Build Components (1.20) (pull_request) Successful in 7m46s
Details 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-08-18 13:01:27 +03:00
Dmitrii Stepanov
c4e1d8eb07 [#602] node: Fix blobovnicza typos 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-08-18 11:14:10 +03:00
Dmitrii Stepanov
10e63537b2 [#602] metrics: Rename blobovnicza size metric 
`Size` is not size, but open db size.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-08-17 19:17:35 +00:00
Dmitrii Stepanov
809e97626b [#602] blobovnicza: Fix size counter 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-08-17 19:17:35 +00:00
Dmitrii Stepanov
2e49d7ea7e [#602] blobovnicza: Init before using 
Fix blobovnicza size: after restart size metric resets.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-08-17 19:17:35 +00:00
Evgenii Stratonikov
d641cba2fc [#587] Do not use math/rand.Read
Some checks failed
Build / Build Components (1.20) (pull_request) Failing after 3s
Details
Build / Build Components (1.21) (pull_request) Failing after 3s
Details
Tests and linters / Lint (pull_request) Failing after 3s
Details
Tests and linters / Tests (1.20) (pull_request) Failing after 3s
Details
Tests and linters / Tests (1.21) (pull_request) Failing after 3s
Details
Tests and linters / Tests with -race (pull_request) Failing after 5s
Details
Tests and linters / Staticcheck (pull_request) Failing after 8s
Details
Vulncheck / Vulncheck (pull_request) Failing after 2s
Details 
Fix staticcheck warnings after go1.20 update.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-08-09 16:02:44 +03:00
Alejandro Lopez
5b7e4a51b7 [#481] Update frostfs-sdk-go and error pointer receivers 
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
 2023-08-09 10:26:53 +00:00
Dmitrii Stepanov
af608da952 [#373] metrics: Add blobovnizca metrics 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-06-21 15:13:26 +03:00
Dmitrii Stepanov
8318d90ad0 [#373] blobovniczatree: Add metrics 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-06-21 15:13:26 +03:00
Dmitrii Stepanov
a8526d45e9 [#373] blobovnizca: Add missed/fix tracing spans 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-06-21 15:13:26 +03:00
Dmitrii Stepanov
028d4a8058 [#373] blobovnicza: Add metrics 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-06-21 15:13:26 +03:00
Dmitrii Stepanov
74578052f9 [#412] node: Replace tracing package 
Use observability module.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-06-01 13:23:11 +00:00
Anton Nikiforov
8dcd06c587 [#394] node: Use Context in Blobovniczas.Iterate()
All checks were successful
ci/woodpecker/push/pre-commit Pipeline was successful
Details 
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
 2023-05-31 10:09:18 +00:00
Evgenii Stratonikov
4b768fd115 [#381] *: Move to sync/atomic 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-05-23 08:18:01 +03:00
Evgenii Stratonikov
1f4061c0e2 [#285] blobonicza: Optimize upperPowerOfTwo()
All checks were successful
ci/woodpecker/pr/pre-commit Pipeline was successful
Details
ci/woodpecker/push/pre-commit Pipeline was successful
Details 
The real reason is this:
```
pkg/local_object_storage/blobovnicza/sizes.go:36:69
    revive empty-block: this block is empty, you can remove it
```

Didn't want to make this function longer or to add `nolint`, thus this
change. To justify:
```
UpperBound/size=1-8          0.4924n ± 1%   0.2472n ± 2%  -49.80% (p=0.000 n=10)
UpperBound/size=1023-8       0.4936n ± 3%   0.2442n ± 1%  -50.52% (p=0.000 n=10)
UpperBound/size=66560-8      0.8201n ± 2%   0.2436n ± 1%  -70.29% (p=0.000 n=10)
UpperBound/size=41943040-8   6.6900n ± 5%   0.2432n ± 0%  -96.36% (p=0.000 n=10)
geomean                       1.075n        0.2446n       -77.24%
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-04-26 17:35:11 +03:00
Dmitrii Stepanov
d62c6e4ce6 [#242] node: Add tracing spans 
Add tracing spans for PUT requests.
Add tracing spans for DELETE requests.
Add tracing spans for SELECT requests.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-14 10:25:53 +00:00
Evgenii Stratonikov
0e31c12e63 [#240] logs: Move log messages to constants 
Drop duplicate entities.
Format entities.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-04-14 05:06:09 +00:00
Dmitrii Stepanov
0920d848d0 [#135] get-object: Add tracing spans 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-12 06:52:00 +00:00
Airat Arifullin
221203beeb [#180] node: Refactor panics in unit test
All checks were successful
ci/woodpecker/push/pre-commit Pipeline was successful
Details 
* Replace panics in unit tests by require.NoError and t.Fatalf

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
 2023-03-29 12:39:07 +03:00
Alex Vanin
20de74a505 Rename package name 
Due to source code relocation from GitHub.

Signed-off-by: Alex Vanin <a.vanin@yadro.com>
 2023-03-07 16:38:26 +03:00
Stanislav Bogatyrev
cb016d53a6 [#1] Fix comments and error messages 
Signed-off-by: Stanislav Bogatyrev <s.bogatyrev@yadro.com>
 2023-02-06 17:41:14 +03:00
Pavel Karpy
cf1a91a758 [#2206] blobovnicza: Use Latin letters in the code 
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-01-25 15:31:47 +03:00
Pavel Karpy
923f84722a Move to frostfs-node 
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
 2022-12-28 15:04:29 +03:00
Evgenii Stratonikov
6ad2b5d5b8 [#2068] blobovnicza: Add Exists method 
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
 2022-12-02 11:52:05 +03:00
Evgenii Stratonikov
d65604ad30 [#1985] blobstor: Allow to report multiple errors to caller 
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
 2022-11-19 11:01:04 +03:00
Pavel Karpy
f037022a7a [#1770] logger: Refactor Logger component 
Make it store its internal `zap.Logger`'s level. Also, make all the
components to accept internal `logger.Logger` instead of `zap.Logger`; it
will simplify future refactor.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-10-12 18:11:05 +03:00
Leonard Lyubich
8e8265d5ea [#1707] blobovnicza/get: Iterate over all buckets regardless of config 
Re-configuration of Blobovnicza's object size limit must not affect
already stored objects. In previous implementation `Blobovnicza` didn't
see objects stored in buckets which became too big after size limit
re-configuration. For example, lets consider 1st configuration with 64KB
size limit for stored objects. Lets assume that we stored object of 64KB
size, and re-configured `Blobovnicza` with 32KB limit. After reboot
object should be still available, but actually it isn't. This is caused
by `Get` operation algorithm which iterates over configured size ranges
only, and doesn't process any other existing size bucket. By the way,
increasing of the object size limit didn't lead to the problem even in
previous implementation.

Make `Blobovnicza.Get` method to iterate over all size buckets
regardless of current configuration. This covers the described scenario.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-09-08 12:24:28 +04:00
Evgenii Stratonikov
b2d4cc556e [#1686] blobovnicza: Remove IterateObjects 
It is used only once, makes sense to inline.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
 2022-08-30 10:02:43 +03:00
Evgenii Stratonikov
73f8bb3e5f [#1523] shard: Store generic storage ID in metabase 
Allow to extend blobstor with more storage sub-systems. Currently
objects stored in the FSTree have empty byte slice descriptor and object
from blobovnicza tree have the same id as earlier. Each such change in
the identifier formation should be accompanied with metabase version
increase.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-08-22 13:14:19 +03:00
Evgenii Stratonikov
1e786233bf [#1559] local_object_storage: Provide readOnly flag to Open 
We should be able to reopen storage in readonly in runtime.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-07-21 17:56:06 +03:00
Evgenii Stratonikov
7df50297cd [#1520] shard: Ignore errors on metabase refill 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-06-21 17:25:28 +03:00
Pavel Karpy
010253a97a [#1460] blobovnicza: Do not use pointers as the results 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-06-06 18:03:12 +03:00
Pavel Karpy
b0c7b7851a [#1418] blobstor: Do not use pointers as parameters 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-06-06 18:03:12 +03:00
Evgenii Stratonikov
b602fff01f [#1463] blobovnicza: Remove pointers from struct members 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-06-03 13:42:54 +03:00
Evgenii Stratonikov
d9f0ac8909 [#1463] blobovnicza: Determine db size via os.Stat 
Currently we use `(*bbolt.Bucket).Stats().KeyN` for estimating database
size. However, it iterates over all pages in bucket and thus heavily
depends on the bucket size. This commit replaces initial size estimation
with a single `os.Stat` call.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-06-03 13:42:54 +03:00
Pavel Karpy
021aa97965 [#1418] blobovnicza: Do not use pointers as parameters 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-06-03 07:35:17 +03:00
Pavel Karpy
281befec67 [#1418] blobstor: Do not use pointers as parameters 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-06-03 07:35:17 +03:00
Leonard Lyubich
1c30414a6c [#1454] Upgrade NeoFS SDK Go module with new IDs 
Core changes:
 * avoid package-colliding variable naming
 * avoid using pointers to IDs where unnecessary
 * avoid using `idSDK` import alias pattern
 * use `EncodeToString` for protocol string calculation and `String` for
  printing

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-06-01 17:41:45 +03:00
Leonard Lyubich
f15e6e888f [#1377] oid, cid: Upgrade SDK package 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-05-16 15:33:22 +03:00
Elizaveta Chichindaeva
cc7a723d77 [#1320] English Check 
Signed-off-by: Elizaveta Chichindaeva <elizaveta@nspcc.ru>
 2022-05-11 10:40:02 +03:00
Evgenii Stratonikov
a4769d8624 *: use require.ErrorIs where possible 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-04-29 16:47:57 +03:00