TrueCloudLab/frostfs-node
19
1
Fork 27
Code Issues 92 Pull requests 9 Projects Releases 1 Activity Actions
4931 commits 8 branches 83 tags 54 MiB
Commit graph

26 commits

Author SHA1 Message Date
Dmitrii Stepanov
e39378b1c3 [#1367] writecache: Add background flushing objects limiter 
To limit memory usage by background flush.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2024-09-12 15:06:33 +03:00
Dmitrii Stepanov
b142b6f48e [#1367] fstree: Add size to file counter 
FSTree file counter used by writecache. As writecache has now only one
storage, so it is required to use real object size to get writecache
size more accurate than `count * max_object_size`.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2024-09-12 15:06:33 +03:00
Dmitrii Stepanov
5f6c7cbdb1 [#1367] writecache: Drop bbolt DB 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2024-09-12 14:22:29 +03:00
Dmitrii Stepanov
68029d756e [#1302] writecache: Allow to specify custom page size
All checks were successful
DCO action / DCO (pull_request) Successful in 1m57s
Details
Vulncheck / Vulncheck (pull_request) Successful in 2m6s
Details
Tests and linters / Run gofumpt (pull_request) Successful in 2m9s
Details
Build / Build Components (1.22) (pull_request) Successful in 2m37s
Details
Pre-commit hooks / Pre-commit (pull_request) Successful in 2m49s
Details
Tests and linters / Tests (1.21) (pull_request) Successful in 3m1s
Details
Tests and linters / Tests (1.22) (pull_request) Successful in 3m4s
Details
Tests and linters / Staticcheck (pull_request) Successful in 3m0s
Details
Tests and linters / Tests with -race (pull_request) Successful in 3m9s
Details
Tests and linters / Lint (pull_request) Successful in 3m30s
Details
Tests and linters / gopls check (pull_request) Successful in 3m40s
Details
Build / Build Components (1.21) (pull_request) Successful in 1m12s
Details 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2024-08-09 12:12:07 +03:00
Dmitrii Stepanov
08b1f18bca [#1296] writecache: Add count limit 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2024-08-09 06:30:32 +00:00
Dmitrii Stepanov
8180a0664f [#887] node: Drop badger writecache implementation
All checks were successful
Vulncheck / Vulncheck (pull_request) Successful in 6m1s
Details
Build / Build Components (1.21) (pull_request) Successful in 7m37s
Details
Build / Build Components (1.20) (pull_request) Successful in 7m52s
Details
Tests and linters / Staticcheck (pull_request) Successful in 8m56s
Details
Tests and linters / Lint (pull_request) Successful in 9m26s
Details
Tests and linters / Tests (1.21) (pull_request) Successful in 15m5s
Details
Tests and linters / Tests with -race (pull_request) Successful in 15m7s
Details
DCO action / DCO (pull_request) Successful in 1m1s
Details
Tests and linters / Tests (1.20) (pull_request) Successful in 4m1s
Details 
Badger implementation isn't tested and works not well,
but requires human resources to maintain.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-12-22 13:00:54 +03:00
Alejandro Lopez
1a0cb0f34a [#421] Try using badger for the write-cache 
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
 2023-08-07 08:16:57 +00:00
Dmitrii Stepanov
d212d908b5 [#312] wc: Add metrics 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-05-24 10:18:39 +00:00
Dmitrii Stepanov
d62c6e4ce6 [#242] node: Add tracing spans 
Add tracing spans for PUT requests.
Add tracing spans for DELETE requests.
Add tracing spans for SELECT requests.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-14 10:25:53 +00:00
Dmitrii Stepanov
0920d848d0 [#135] get-object: Add tracing spans 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-12 06:52:00 +00:00
Alejandro Lopez
341fe1688f [#139] test: Add test storage implementation
All checks were successful
ci/woodpecker/push/pre-commit Pipeline was successful
Details 
This aims to reduce the usage of chmod hackery to induce or simulate
OS-related failures.

Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
 2023-03-29 14:28:49 +00:00
Alex Vanin
20de74a505 Rename package name 
Due to source code relocation from GitHub.

Signed-off-by: Alex Vanin <a.vanin@yadro.com>
 2023-03-07 16:38:26 +03:00
Pavel Karpy
923f84722a Move to frostfs-node 
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
 2022-12-28 15:04:29 +03:00
Evgenii Stratonikov
857d2dc3f5 [#2068] writecache: Optimize initial flush existence checking 
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
 2022-12-02 11:52:05 +03:00
Evgenii Stratonikov
777fd32d4f [#1818] writecache: Increase error counter on background errors 
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
 2022-11-02 14:24:02 +03:00
Evgenii Stratonikov
bffb0f894c [#1818] writecache: Update storage ID during flush 
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
 2022-11-02 14:24:02 +03:00
Evgenii Stratonikov
98a152256b [#1992] writecache: Allow to open in NOSYNC mode 
Applicable only to FSTree as we cannot handle corrupted databases
properly yet.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
 2022-11-01 09:42:26 +03:00
Pavel Karpy
f037022a7a [#1770] logger: Refactor Logger component 
Make it store its internal `zap.Logger`'s level. Also, make all the
components to accept internal `logger.Logger` instead of `zap.Logger`; it
will simplify future refactor.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-10-12 18:11:05 +03:00
Evgenii Stratonikov
55148404ee [#1806] writecache: Use interfaces for blobstor and metabase 
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
 2022-09-28 09:28:01 +03:00
Evgenii Stratonikov
177e8e01b1 [#1745] neofs-node: Remove memcache_capacity from the configuration 
It is unused since ddaed283e9 .

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
 2022-09-02 11:59:53 +03:00
Evgenii Stratonikov
bda084f331 [#1745] writecache: Simplify object counters 
Remove unused option and additional pointers to db/fstree.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
 2022-09-02 11:59:53 +03:00
Evgenii Stratonikov
972ca83e23 [#1524] writecache: Add some bolt parameters to the configuration 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-06-20 17:04:35 +03:00
Leonard Lyubich
a1696a81b6 [#776] writecache: Limit size of used disk space 
There is a need to limit disk space used by write-cache. It is almost
impossible to calculate the value exactly. It is proposed to estimate the
size of the cache by the number of objects stored in it.

Track amounts of objects saved in DB and FSTree separately. To do this,
`ObjectCounters` interface is defined. It is generalized to a store of
numbers that can be made persistent (new option `WithObjectCounters`). By
default DB number is calculated as key number in default bucket, and FS
number is set same to DB since it is currently hard to read the actual value
from `FSTree` instance. Each PUT/DELETE operation to DB or FS
increases/decreases corresponding counter. Before each PUT op an overflow
check is performed with the following formula for evaluating the occupied
space: `NumDB * MaxDBSize + NumFS * MaxFSSize`. If next PUT can cause
write-cache overflow, object is written to the main storage.

By default maximum write-cache size is set to 1GB.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2021-09-15 18:07:36 +03:00
Leonard Lyubich
3258d9c616 [#790] writecache: Log each writing operation 
Call `storagelog.Write` in all places after the successful writing op.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2021-09-13 15:02:13 +03:00
Pavel Karpy
c54f524df9 [#773] writecache: Delete unused dbSize param 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2021-08-27 15:05:53 +03:00
Evgenii Stratonikov
59de521fd1 [#472] blobstor: implement write-cache 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2021-04-29 17:36:36 +03:00