TrueCloudLab/frostfs-node
19
1
Fork 27
Code Issues 91 Pull requests 6 Projects Releases 1 Activity Actions
5058 commits 8 branches 81 tags 52 MiB
Commit graph

189 commits

Author SHA1 Message Date
Alejandro Lopez
345a1a69a2 [#635] Use internal key type when deleting from badger wc
All checks were successful
DCO action / DCO (pull_request) Successful in 2m48s
Details
Build / Build Components (1.21) (pull_request) Successful in 4m59s
Details
Build / Build Components (1.20) (pull_request) Successful in 5m5s
Details
Vulncheck / Vulncheck (pull_request) Successful in 4m55s
Details
Tests and linters / Staticcheck (pull_request) Successful in 6m51s
Details
Tests and linters / Tests (1.21) (pull_request) Successful in 7m21s
Details
Tests and linters / Tests (1.20) (pull_request) Successful in 7m53s
Details
Tests and linters / Lint (pull_request) Successful in 8m35s
Details
Tests and linters / Tests with -race (pull_request) Successful in 9m57s
Details 
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
 2023-08-22 10:53:19 +03:00
Evgenii Stratonikov
f7042c5a6f [#609] Replace zaptest.NewLogger() with zap.L() 
Semantic patch:
```
@@
@@
-import "go.uber.org/zap/zaptest"
+import "go.uber.org/zap"

-zaptest.NewLogger(t)
+zap.L()
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-08-17 16:51:10 +00:00
Dmitrii Stepanov
2efe9cc1be [#585] writecache: Fix DB counter
Some checks failed
DCO action / DCO (pull_request) Successful in 2m14s
Details
Vulncheck / Vulncheck (pull_request) Successful in 2m44s
Details
Build / Build Components (1.21) (pull_request) Successful in 2m54s
Details
Build / Build Components (1.20) (pull_request) Successful in 3m5s
Details
Tests and linters / Staticcheck (pull_request) Successful in 3m43s
Details
Tests and linters / Lint (pull_request) Successful in 5m12s
Details
Tests and linters / Tests (1.21) (pull_request) Successful in 7m44s
Details
Tests and linters / Tests (1.20) (pull_request) Successful in 21m5s
Details
Tests and linters / Tests with -race (pull_request) Failing after 26m26s
Details 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-08-16 14:47:44 +03:00
Dmitrii Stepanov
58c8722c81 [#585] fstree: Add optional file counter 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-08-16 14:47:44 +03:00
Dmitrii Stepanov
eec97d177e [#585] writecache: Count items periodically 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-08-16 14:47:44 +03:00
Alejandro Lopez
88b6755c5e [#598] Fix use-after-close bug in badger writecache 
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
 2023-08-15 07:04:06 +00:00
Alejandro Lopez
376f03a445 [#598] Hold mode mutex when setting mode 
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
 2023-08-14 07:12:27 +00:00
Alexander Chuprov
21800e9fcc [#162] core: Move literals to constants 
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
 2023-08-11 15:48:42 +03:00
Alejandro Lopez
8f994163ee [#586] Fix writecache benchmarks and refactor hacky NeedsCompression 
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
 2023-08-10 08:05:18 +00:00
Alejandro Lopez
023b90342c [#584] Disable compression in badger writecache
Some checks failed
Build / Build Components (1.20) (pull_request) Failing after 3s
Details
Build / Build Components (1.21) (pull_request) Failing after 2s
Details
Tests and linters / Lint (pull_request) Failing after 4s
Details
Tests and linters / Tests (1.20) (pull_request) Failing after 3s
Details
Tests and linters / Tests (1.21) (pull_request) Failing after 3s
Details
Tests and linters / Tests with -race (pull_request) Failing after 3s
Details
Tests and linters / Staticcheck (pull_request) Failing after 3s
Details
Vulncheck / Vulncheck (pull_request) Failing after 3s
Details 
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
 2023-08-10 10:14:06 +03:00
Alejandro Lopez
5b7e4a51b7 [#481] Update frostfs-sdk-go and error pointer receivers 
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
 2023-08-09 10:26:53 +00:00
Alejandro Lopez
ae322e9f73 [#576] Set SyncWrites for badger writecache by default
Some checks failed
Build / Build Components (1.19) (pull_request) Failing after 3s
Details
Build / Build Components (1.20) (pull_request) Failing after 2s
Details
Tests and linters / Tests (1.19) (pull_request) Failing after 3s
Details
Tests and linters / Tests with -race (pull_request) Failing after 3s
Details
Tests and linters / Lint (pull_request) Failing after 12s
Details
Tests and linters / Staticcheck (pull_request) Failing after 3s
Details
Vulncheck / Vulncheck (pull_request) Successful in 5m46s
Details
Tests and linters / Tests (1.20) (pull_request) Successful in 11m37s
Details 
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
 2023-08-09 11:28:26 +03:00
Alejandro Lopez
7da4306e38 [#575] writecache: Fix log level for badger writecache 
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
 2023-08-08 08:16:36 +00:00
Dmitrii Stepanov
0e697266c3 [#563] writecache: Fix metrics and bolt delete 
Estimate cache size after delete objects to update metric.
Update counters on small object deletion.
Do not count bbolt DB file as FSTree object.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-08-07 12:53:28 +00:00
Alejandro Lopez
1a0cb0f34a [#421] Try using badger for the write-cache 
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
 2023-08-07 08:16:57 +00:00
Evgenii Stratonikov
ec8b4fdc48 [#541] writecache/test: Close writecache on exit 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-07-29 10:57:25 +00:00
Evgenii Stratonikov
ad5f527bd3 [#541] writecache/test: Remove initWC() 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-07-29 10:57:25 +00:00
Alejandro Lopez
11027945d8 [#479] writecache: Fix writecache fstree flush premature ctx cancel
All checks were successful
Build / Build Components (1.20) (pull_request) Successful in 3m49s
Details
Build / Build Components (1.19) (pull_request) Successful in 4m2s
Details
ci/woodpecker/pr/pre-commit Pipeline was successful
Details
Tests and linters / Tests (1.20) (pull_request) Successful in 6m13s
Details
Tests and linters / Lint (pull_request) Successful in 11m25s
Details
Tests and linters / Tests with -race (pull_request) Successful in 9m4s
Details
Tests and linters / Staticcheck (pull_request) Successful in 6m0s
Details
Tests and linters / Tests (1.19) (pull_request) Successful in 25m11s
Details
ci/woodpecker/push/pre-commit Pipeline was successful
Details 
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
 2023-07-14 10:25:52 +03:00
Alexander Chuprov
033eaf77e1 [#496] node: Fix linter importas
All checks were successful
Build / Build Components (1.20) (pull_request) Successful in 3m52s
Details
Build / Build Components (1.19) (pull_request) Successful in 4m1s
Details
ci/woodpecker/pr/pre-commit Pipeline was successful
Details
Tests and linters / Tests with -race (pull_request) Successful in 5m36s
Details
Tests and linters / Tests (1.20) (pull_request) Successful in 5m55s
Details
Tests and linters / Lint (pull_request) Successful in 14m40s
Details
Tests and linters / Tests (1.19) (pull_request) Successful in 15m29s
Details
ci/woodpecker/push/pre-commit Pipeline was successful
Details 
Standardize the alias of the
import frostfs-sdk-go/object as objectSDK.

Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
 2023-07-06 15:36:41 +03:00
Evgenii Stratonikov
fe01781811 [#446] los: Wrap SSD errors in a separate type 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-06-16 14:53:31 +03:00
Dmitrii Stepanov
85deb12f4d [#424] writecache: Drop metrics when close 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-06-14 14:52:40 +03:00
Dmitrii Stepanov
74578052f9 [#412] node: Replace tracing package 
Use observability module.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-06-01 13:23:11 +00:00
Anton Nikiforov
802168c0c6 [#364] node: Stop flushing big object when termination signal received
All checks were successful
ci/woodpecker/pr/pre-commit Pipeline was successful
Details
ci/woodpecker/push/pre-commit Pipeline was successful
Details 
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
 2023-05-26 16:46:58 +03:00
Dmitrii Stepanov
2ce43935f9 [#312] metrics: Add writecache metrcis 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-05-24 10:18:39 +00:00
Dmitrii Stepanov
d212d908b5 [#312] wc: Add metrics 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-05-24 10:18:39 +00:00
Dmitrii Stepanov
4503a61997 [#312] wc: Delete unused Iterate method 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-05-24 10:18:39 +00:00
Evgenii Stratonikov
4b768fd115 [#381] *: Move to sync/atomic 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-05-23 08:18:01 +03:00
Evgenii Stratonikov
35c9b6b26d [#314] writecache: remove objects right after they are flushed
All checks were successful
ci/woodpecker/push/pre-commit Pipeline was successful
Details 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-05-19 09:37:05 +00:00
Pavel Karpy
bf79d06f03 [#314] writecache: Do not lose small objects on disk errors 
Do return error if an object could not been stored on WC's disk.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
 2023-05-19 09:37:05 +00:00
Pavel Karpy
9e56592be3 [#314] writecache: Simplify background workers naming 
Also, drop not used arg.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
 2023-05-19 09:37:05 +00:00
Pavel Karpy
e9461686b8 [#274] wc: Resolve possible deadlock 
If operation with WC are _fast enough_ (e.g. `Init` failed and `Close` is
called immediately) there is a race and a deadlock that do not allow finish
(and start, in fact) an initialization routine because of taken `modeMtx`
and also do not allow finish `Close` call because of awaiting initialization
finish. So do stop initialization _before_ any mutex is taken.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
 2023-04-20 19:22:54 +03:00
Pavel Karpy
6b6f33ed71 [#274] wc: Make wait groups work more explicit 
Do not run routine that calls `wg.Done()` inside, it is hard to read.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
 2023-04-20 19:22:54 +03:00
Pavel Karpy
20cd080323 [#255] write-cache: Fix init race condition
All checks were successful
ci/woodpecker/push/pre-commit Pipeline was successful
Details 
Do not use WC's internals in the initialization routines without mode
protection. WC should be able to change its mode even if the initialization
is not finished yet.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
 2023-04-19 09:29:07 +00:00
Dmitrii Stepanov
d62c6e4ce6 [#242] node: Add tracing spans 
Add tracing spans for PUT requests.
Add tracing spans for DELETE requests.
Add tracing spans for SELECT requests.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-14 10:25:53 +00:00
Evgenii Stratonikov
0e31c12e63 [#240] logs: Move log messages to constants 
Drop duplicate entities.
Format entities.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-04-14 05:06:09 +00:00
Dmitrii Stepanov
0920d848d0 [#135] get-object: Add tracing spans 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-12 06:52:00 +00:00
Alejandro Lopez
341fe1688f [#139] test: Add test storage implementation
All checks were successful
ci/woodpecker/push/pre-commit Pipeline was successful
Details 
This aims to reduce the usage of chmod hackery to induce or simulate
OS-related failures.

Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
 2023-03-29 14:28:49 +00:00
Pavel Karpy
f1f3c80dbf [#32] node: Init write-cache asynchronously 
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
 2023-03-09 11:07:33 +00:00
Alex Vanin
20de74a505 Rename package name 
Due to source code relocation from GitHub.

Signed-off-by: Alex Vanin <a.vanin@yadro.com>
 2023-03-07 16:38:26 +03:00
Evgenii Stratonikov
d1d123d180 [#2234] writecache: Fix possible panic in initFlushMarks 
In case we have many small objects in the write-cache, `indices` should
not be reused between iterations.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-02-20 13:53:27 +03:00
Evgenii Stratonikov
e0309e398c [#2239] writecache: Fix possible deadlock 
LRU `Peek`/`Contains` take LRU mutex _inside_ of a `View` transaction.
`View` transaction itself takes `mmapLock` [1], which is lifted after tx
finishes (in `tx.Commit()` -> `tx.close()` -> `tx.db.removeTx`)

When we evict items from LRU cache mutex order is different:
first we take LRU mutex and then execute `Batch` which _does_ take
`mmapLock` in case we need to remap. Thus the deadlock.

[1] 8f4a7e1f92/db.go (L708)

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-02-20 13:53:27 +03:00
Evgenii Stratonikov
4155c1bdff [#5] writecache: Use generic LRU cache 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2022-12-31 23:04:06 +03:00
Pavel Karpy
1608fd1c07 [#2167] write-cache: Add "write-cache" to its logs 
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
 2022-12-30 11:07:35 +03:00
Pavel Karpy
923f84722a Move to frostfs-node 
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
 2022-12-28 15:04:29 +03:00
Evgenii Stratonikov
42554a9298 [#2068] writecache: Remove deleted objects from the writecache 
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
 2022-12-02 11:52:05 +03:00
Evgenii Stratonikov
4a49ea0855 [#2068] writecache: Allow to open FSTree in read-only mode 
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
 2022-12-02 11:52:05 +03:00
Evgenii Stratonikov
857d2dc3f5 [#2068] writecache: Optimize initial flush existence checking 
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
 2022-12-02 11:52:05 +03:00
Pavel Karpy
ed4351aab0 [#2074] write-cache: Do not flush same object twice 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-11-19 11:01:04 +03:00
Pavel Karpy
dd225906a0 [#2074] write-cache: Remove unused variables 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-11-19 11:01:04 +03:00
Evgenii Stratonikov
b0e94b6a6b [#1906] writecache: Do not require read-only mode in Flush 
It was needed before we started to flush during transition to
`degraded` mode. Now it is confusing.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
 2022-11-19 11:01:04 +03:00
Evgenii Stratonikov
777fd32d4f [#1818] writecache: Increase error counter on background errors 
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
 2022-11-02 14:24:02 +03:00
Evgenii Stratonikov
bffb0f894c [#1818] writecache: Update storage ID during flush 
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
 2022-11-02 14:24:02 +03:00
Evgenii Stratonikov
b64b14eb54 [#1818] writecache: Reuse FSTree flushing code between flushes 
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
 2022-11-02 14:24:02 +03:00
Evgenii Stratonikov
a56927e3d4 [#1818] writecache: Remove unused variable 
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
 2022-11-02 14:24:02 +03:00
Evgenii Stratonikov
98a152256b [#1992] writecache: Allow to open in NOSYNC mode 
Applicable only to FSTree as we cannot handle corrupted databases
properly yet.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
 2022-11-01 09:42:26 +03:00
Evgenii Stratonikov
34501685b7 [#1969] local_object_storage: Move ErrObjectIsExpired to another package 
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
 2022-10-31 11:41:24 +03:00
Evgenii Stratonikov
56de2f1363 [#1969] local_object_storage: Simplify logic error construction 
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
 2022-10-31 11:41:24 +03:00
Evgenii Stratonikov
fcdbf5e509 [#1969] local_object_storage: Add a type for logical errors 
All logic errors are wrapped in `logicerr.Logical` type and do not
affect shard error counter.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
 2022-10-31 11:41:24 +03:00
Leonard Lyubich
e8c5f03c30 [#1905] shard: Don't log read-only errors of write-cache 
There is no need to log `writecache.ErrReadOnly` errors in `Delete`
method of the `Shard`.

Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
 2022-10-28 18:30:45 +03:00
Pavel Karpy
f037022a7a [#1770] logger: Refactor Logger component 
Make it store its internal `zap.Logger`'s level. Also, make all the
components to accept internal `logger.Logger` instead of `zap.Logger`; it
will simplify future refactor.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-10-12 18:11:05 +03:00
Evgenii Stratonikov
8b3b16fe62 [#1825] writecache: Flush cache when moving to the DEGRADED mode 
Degraded mode allows us to operate without an SSD,
thus writecache should be unavailable in this mode.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
 2022-10-04 12:13:09 +03:00
Evgenii Stratonikov
0a411908ee [#1806] writecache: Allow to ignore read errors during flush 
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
 2022-09-28 09:28:01 +03:00
Evgenii Stratonikov
0b4c867ef1 [#1806] writecache: Allow to start flush manually 
Allow user to initiate flushing objects from a writecache.
We need this in 2 cases:
1. During writecache storage schema update, it should be flushed with
   the old version of node and started clean with a new one.
2. During SSD replacement, to avoid data loss.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
 2022-09-28 09:28:01 +03:00
Evgenii Stratonikov
55148404ee [#1806] writecache: Use interfaces for blobstor and metabase 
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
 2022-09-28 09:28:01 +03:00
Evgenii Stratonikov
e6cf0b3d42 [#1745] writecache: Set flush parameters based on max size 
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
 2022-09-02 11:59:53 +03:00
Evgenii Stratonikov
c04126f35e [#1745] writecache: Remove IsErrNotFound 
We specify the error in the doc-comment, and it is the same for all our
components.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
 2022-09-02 11:59:53 +03:00
Evgenii Stratonikov
9e41e85295 [#1745] writecache: Actualize docs 
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
 2022-09-02 11:59:53 +03:00
Evgenii Stratonikov
e9c6ee2623 [#1745] writecache: Rename constants for default values 
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
 2022-09-02 11:59:53 +03:00
Evgenii Stratonikov
177e8e01b1 [#1745] neofs-node: Remove memcache_capacity from the configuration 
It is unused since ddaed283e9 .

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
 2022-09-02 11:59:53 +03:00
Evgenii Stratonikov
bda084f331 [#1745] writecache: Simplify object counters 
Remove unused option and additional pointers to db/fstree.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
 2022-09-02 11:59:53 +03:00
Evgenii Stratonikov
20abdaeed4 [#1745] writecache: Set flush mark only on success 
Set flush mark in the inside the flush worker because writing to the blobstor
can fail. Because each evicted object must be deleted, it is reasonable
to do this in the evict callback.

The evict callback is protected by LRU mutex and thus potentially interferes
with `Get` and `Iterate` methods. This problem will be addressed in the
future.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
 2022-09-02 11:59:53 +03:00
Evgenii Stratonikov
b8b9d25f9d [#1686] local_object_storage: Add generic tests 
Use them for writecache as a simple example.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
 2022-08-30 10:02:43 +03:00
Pavel Karpy
30341f2192 [#1687] *: Perform go fmt using go v1.19 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-08-22 18:59:57 +03:00
Evgenii Stratonikov
26b4a258e0 [#1523] neofs-node: Refactor configuration 
1. Move compression parameters to the `shard` section.
2. Allow to use multiple sub-storage components in the blobstor.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-08-22 13:14:19 +03:00
Evgenii Stratonikov
266458fe5c [#1523] blobstor: Unify fstree and blobovnicza interfaces 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-08-22 13:14:19 +03:00
Evgenii Stratonikov
5aa3defc67 [#1523] writecache: Remove cloneBytes 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-08-22 13:14:19 +03:00
Evgenii Stratonikov
ddaed283e9 [#1523] writecache: Simplify logic 
1. Remove in-memory cache. It doesn't persist objects and if we want
   more speed, `NoSync` option can be used for the bolt DB.
2. Put to the metabase in a synchronous fashion. This considerably
   simplifies overall logic and plays nicely with the metabase bolt DB
   batch settings.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-08-22 13:14:19 +03:00
Evgenii Stratonikov
4176b2a1bc [#1523] local_object_storage: Unify parameters for the Iterate operation 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-08-22 13:14:19 +03:00
Evgenii Stratonikov
73f8bb3e5f [#1523] shard: Store generic storage ID in metabase 
Allow to extend blobstor with more storage sub-systems. Currently
objects stored in the FSTree have empty byte slice descriptor and object
from blobovnicza tree have the same id as earlier. Each such change in
the identifier formation should be accompanied with metabase version
increase.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-08-22 13:14:19 +03:00
Evgenii Stratonikov
9eb018672c [#1523] blobstor: Unify parameters for Put operation 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-08-22 13:14:19 +03:00
Evgenii Stratonikov
ca15083a50 [#1523] local_object_storage: Unify parameters for the Exists operation 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-08-22 13:14:19 +03:00
Evgenii Stratonikov
d9a2f280c9 [#1523] local_object_storage: Unify parameters for the Get operation 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-08-22 13:14:19 +03:00
Evgenii Stratonikov
4e361e2ab4 [#1647] writecache: Open DB in sync mode 
Close #1647.

Initially, `Sync: false` was provided because we can already lose
objects cached in memory. However, future changes in writecache will
remove inmemory cache and speed up it via other means.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
 2022-08-16 15:27:18 +04:00
Pavel Karpy
da2975a2f9 [#1664] write-cache: Fix panic on Delete operation 
If an object is found in the Write-cache and is placed at the end of
the in-memory cache, the memory counter update operation tries to
dereference the index that is out of the sliced array. Moreover, even if
panic does not appear, the counter is updated with the wrong value.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-08-09 20:18:26 +03:00
Pavel Karpy
156ba85326 [#1634] node: Do not return expired objects 
If an object has not been marked for removal by the GC in the current epoch
yet but has already expired, respond with `ErrObjectNotFound` api status.
Also, optimize shard iteration: a node must stop any iteration if the object
 is found but gonna be removed soon.
All the checks are performed by the Metabase.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-08-04 16:31:49 +03:00
Evgenii Stratonikov
1691364653 [#1559] local_object_storage: Fix tests and some data races 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-07-21 17:56:06 +03:00
Evgenii Stratonikov
3df62769c0 [#1559] local_object_storage: Allow to set mode for all components 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-07-21 17:56:06 +03:00
Evgenii Stratonikov
1e786233bf [#1559] local_object_storage: Provide readOnly flag to Open 
We should be able to reopen storage in readonly in runtime.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-07-21 17:56:06 +03:00
Evgenii Stratonikov
339864b720 [#1559] local_object_storage: Move shard.Mode to a separate package 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-07-21 17:56:06 +03:00
Evgenii Stratonikov
34d319fed2 [#1559] metabase: Use Set prefix for parameter setting 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-07-21 17:56:06 +03:00
Evgenii Stratonikov
f58234aa2f [#1559] metabase: Remove public functions 
Reduce public interface of this package. Later each result will contain
an additional status, so it makes more sense to use the same functions
and result processing everywhere.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-07-21 17:56:06 +03:00
Evgenii Stratonikov
11c9df6f00 [#1611] shard: Print shard ID in component logs 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-07-19 15:57:53 +03:00
Evgenii Stratonikov
972ca83e23 [#1524] writecache: Add some bolt parameters to the configuration 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-06-20 17:04:35 +03:00
Evgenii Stratonikov
38772e1a2e writecache: provide timeout to bbolt.Open 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-06-09 14:19:47 +03:00
Evgenii Stratonikov
feef9a98f7 [#1462] writecache: remove useless logs 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-06-03 14:30:00 +03:00
Evgenii Stratonikov
43e776dfb1 [#1462] writecache: Persist objects in batches 
`Update` becomes a botleneck under a heavy load.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-06-03 14:30:00 +03:00
Evgenii Stratonikov
6ad87e7959 [#1462] writecache: Fill flush marks during startup 
Some of the objects are already flushed, don't do it twice.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-06-03 14:30:00 +03:00
Evgenii Stratonikov
f2a7503964 [#1462] fstree: Allow to fetch file content lazily 
If we should process address based on some condition, there is no need
to read file content in memory.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-06-03 14:30:00 +03:00
Evgenii Stratonikov
54d4503701 [#1462] writecache: Rename flushWorkersCount 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-06-03 14:30:00 +03:00
Evgenii Stratonikov
0148209168 [#1462] writecache: Use Timer instead of Ticker in flush 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-06-03 14:30:00 +03:00