Commit Graph

1040 Commits (5c252c9193f638eacce54b580fa7a6493e1651c7)

Author SHA1 Message Date
Anton Nikiforov 3195142d67 [#959] writecache: Avoid manipulation with cache in `DEGRADED` mode
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-03-11 18:35:41 +00:00
Dmitrii Stepanov d433b49265 [#973] node: Resolve perfsprint linter
DCO action / DCO (pull_request) Successful in 2m40s Details
Vulncheck / Vulncheck (pull_request) Successful in 3m41s Details
Build / Build Components (1.20) (pull_request) Successful in 4m27s Details
Build / Build Components (1.21) (pull_request) Successful in 5m6s Details
Tests and linters / Staticcheck (pull_request) Successful in 6m16s Details
Tests and linters / gopls check (pull_request) Successful in 6m23s Details
Tests and linters / Lint (pull_request) Successful in 6m48s Details
Tests and linters / Tests (1.20) (pull_request) Successful in 9m4s Details
Tests and linters / Tests with -race (pull_request) Successful in 9m9s Details
Tests and linters / Tests (1.21) (pull_request) Successful in 9m23s Details
`fmt.Errorf can be replaced with errors.New` and `fmt.Sprintf can be replaced with string addition`

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-03-11 17:55:50 +03:00
Dmitrii Stepanov 66a26b7775 [#973] node: Resolve revive: unused-parameter linter
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-03-11 17:11:49 +03:00
Dmitrii Stepanov 0882840bf5 [#634] shard: Add writecache inhume tests
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-03-06 13:12:34 +03:00
Anton Nikiforov ae5bb87e70 Revert "[#866] Use TTL for blobovnicza tree cache"
DCO action / DCO (pull_request) Successful in 2m7s Details
Vulncheck / Vulncheck (pull_request) Successful in 3m7s Details
Tests and linters / Staticcheck (pull_request) Successful in 5m12s Details
Tests and linters / Lint (pull_request) Successful in 6m23s Details
Tests and linters / Tests with -race (pull_request) Successful in 7m11s Details
Tests and linters / gopls check (pull_request) Successful in 7m27s Details
Tests and linters / Tests (1.20) (pull_request) Successful in 8m5s Details
Tests and linters / Tests (1.21) (pull_request) Successful in 8m22s Details
Build / Build Components (1.21) (pull_request) Successful in 1m41s Details
Build / Build Components (1.20) (pull_request) Successful in 1m56s Details
This reverts commit d9cbb16bd3.

Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-03-01 19:29:33 +03:00
Dmitrii Stepanov d6534fd755 [#1016] frostfs-node: Fix gopls issues
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-03-01 12:13:43 +03:00
Dmitrii Stepanov 918613546f [#1008] metabase: Do not update storageID on put
DCO action / DCO (pull_request) Successful in 4m56s Details
Build / Build Components (1.21) (pull_request) Successful in 7m52s Details
Build / Build Components (1.20) (pull_request) Successful in 8m11s Details
Vulncheck / Vulncheck (pull_request) Successful in 8m9s Details
Tests and linters / Staticcheck (pull_request) Successful in 9m58s Details
Tests and linters / Lint (pull_request) Successful in 10m42s Details
Tests and linters / Tests (1.20) (pull_request) Successful in 12m14s Details
Tests and linters / Tests with -race (pull_request) Successful in 12m17s Details
Tests and linters / Tests (1.21) (pull_request) Successful in 12m34s Details
There may be a race condition between put an object and
flushing the writecache:
1. Put object to the writecache
2. Writecache flushes object to the blobstore and sets blobstore's
storageID
3. Put object to the metabase, set writecache's storageID

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-02-28 11:01:50 +03:00
Dmitrii Stepanov 2ad433dbcb [#1005] engine: Drop shards weights
DCO action / DCO (pull_request) Successful in 4m7s Details
Vulncheck / Vulncheck (pull_request) Successful in 4m53s Details
Build / Build Components (1.21) (pull_request) Successful in 5m46s Details
Build / Build Components (1.20) (pull_request) Successful in 6m21s Details
Tests and linters / Staticcheck (pull_request) Successful in 7m45s Details
Tests and linters / Lint (pull_request) Successful in 8m44s Details
Tests and linters / Tests (1.21) (pull_request) Successful in 13m1s Details
Tests and linters / Tests (1.20) (pull_request) Successful in 15m42s Details
Tests and linters / Tests with -race (pull_request) Successful in 16m10s Details
Unused.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-02-26 17:25:05 +03:00
Dmitrii Stepanov 7470c383dd [#997] metabase: Drop toMoveIt bucket
DCO action / DCO (pull_request) Successful in 2m9s Details
Vulncheck / Vulncheck (pull_request) Successful in 3m3s Details
Build / Build Components (1.20) (pull_request) Successful in 3m56s Details
Build / Build Components (1.21) (pull_request) Successful in 3m54s Details
Tests and linters / Staticcheck (pull_request) Successful in 5m26s Details
Tests and linters / Lint (pull_request) Successful in 6m19s Details
Tests and linters / Tests (1.20) (pull_request) Successful in 8m6s Details
Tests and linters / Tests with -race (pull_request) Successful in 8m20s Details
Tests and linters / Tests (1.21) (pull_request) Successful in 8m39s Details
It is not used.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-02-21 10:06:05 +03:00
Evgenii Stratonikov adf7ebab5b [#996] metabase: Speed up bucket creation
Most of the time it exits, e.g. when it is per-container and use on each
object PUT. Bbolt implementation first tries to create bucket and then
returns it if it exists. Create operation uses cursor and thus is not
very lightweight, we can avoid it.

```
goos: linux
goarch: amd64
pkg: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/metabase
cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz
                 │     old     │                new                 │
                 │   sec/op    │   sec/op     vs base               │
Put/parallel-8     174.4µ ± 3%   163.3µ ± 3%  -6.39% (p=0.000 n=10)
Put/sequential-8   263.3µ ± 2%   259.0µ ± 1%  -1.64% (p=0.000 n=10)
geomean            214.3µ        205.6µ       -4.05%

                 │     old      │                 new                 │
                 │     B/op     │     B/op      vs base               │
Put/parallel-8     275.3Ki ± 3%   281.1Ki ± 4%       ~ (p=0.063 n=10)
Put/sequential-8   413.0Ki ± 2%   426.6Ki ± 2%  +3.29% (p=0.003 n=10)
geomean            337.2Ki        346.3Ki       +2.70%

                 │     old     │                 new                 │
                 │  allocs/op  │  allocs/op   vs base                │
Put/parallel-8      678.0 ± 1%    524.5 ± 2%  -22.64% (p=0.000 n=10)
Put/sequential-8   1.329k ± 0%   1.183k ± 0%  -10.91% (p=0.000 n=10)
geomean             949.1         787.9       -16.98%
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-02-20 15:42:58 +00:00
Evgenii Stratonikov 9f68305c2e Revert "[#972] pilorama: Remove removeDuplicatesInPlace()"
This reverts commit 45fd4e4ff1.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-02-19 15:36:01 +00:00
Evgenii Stratonikov dad56d2e98 Revert "[#972] Use min/max builtins"
This reverts commit 89784b2e0a.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-02-19 15:36:01 +00:00
Evgenii Stratonikov 7627d08914 Revert "[#972] Use require.ElementsMatch() where possible"
This reverts commit 6d9707ff1f.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-02-19 15:36:01 +00:00
Evgenii Stratonikov 946f2ec2bf Revert "[#972] Drop x/exp/slices dependency"
This reverts commit f3e50772fd.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-02-19 15:36:01 +00:00
Evgenii Stratonikov f3e50772fd [#972] Drop x/exp/slices dependency
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-02-19 13:13:09 +00:00
Evgenii Stratonikov 6d9707ff1f [#972] Use require.ElementsMatch() where possible
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-02-19 13:13:09 +00:00
Evgenii Stratonikov 89784b2e0a [#972] Use min/max builtins
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-02-19 13:13:09 +00:00
Evgenii Stratonikov 45fd4e4ff1 [#972] pilorama: Remove removeDuplicatesInPlace()
Also, check that slices.CompareFunc() indeed passes all the tests before
removal.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-02-19 13:13:09 +00:00
Anton Nikiforov 05b5f5ca85 [#959] writecache: Fix panic on `Get` when it is not initialized
Vulncheck / Vulncheck (pull_request) Successful in 2m41s Details
DCO action / DCO (pull_request) Successful in 2m43s Details
Tests and linters / Staticcheck (pull_request) Successful in 3m55s Details
Build / Build Components (1.21) (pull_request) Successful in 3m45s Details
Build / Build Components (1.20) (pull_request) Successful in 4m19s Details
Tests and linters / Lint (pull_request) Successful in 5m30s Details
Tests and linters / Tests (1.20) (pull_request) Successful in 7m57s Details
Tests and linters / Tests (1.21) (pull_request) Successful in 8m22s Details
Tests and linters / Tests with -race (pull_request) Successful in 8m39s Details
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-02-14 16:10:33 +03:00
Anton Nikiforov 2429508ac5 [#959] shard: Skip rebuild in `DEGRADED` mode
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-02-14 15:39:28 +03:00
Anton Nikiforov 0bd030507e [#948] metrics: Set actual value for `shard_id` after restart
DCO action / DCO (pull_request) Successful in 2m26s Details
Vulncheck / Vulncheck (pull_request) Successful in 3m56s Details
Tests and linters / Staticcheck (pull_request) Successful in 4m18s Details
Build / Build Components (1.21) (pull_request) Successful in 4m11s Details
Build / Build Components (1.20) (pull_request) Successful in 4m19s Details
Tests and linters / Lint (pull_request) Successful in 6m3s Details
Tests and linters / Tests (1.20) (pull_request) Successful in 8m7s Details
Tests and linters / Tests (1.21) (pull_request) Successful in 8m26s Details
Tests and linters / Tests with -race (pull_request) Successful in 9m20s Details
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-02-13 09:43:21 +03:00
Anton Nikiforov 6a5769d1da [#948] Fix `gofumpt` issue
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-02-13 09:40:46 +03:00
Evgenii Stratonikov b36a453238 [#970] fstree: Add build tag to enable generic version on linux
Unless tested, generic version can start gaining bugs. With a separate
build tag we can have the best of both worlds:
1. Use optimized implementation for linux by default.
2. Run tests or benchmarks for both. Note that they are not actually
   run automatically now, but this is at leas possible.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-02-09 16:12:11 +00:00
Evgenii Stratonikov abd502215f [#970] fstree: Move file locking to the generic writer
It is not a part of FSTree itself, but rather a way to solve concurrent
counter update on non-linux implementations. New linux implementations
is pretty simple: link fails when the file exists, unlink fails when the
file doesn't exist.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-02-09 16:12:11 +00:00
Evgenii Stratonikov fb74524ac7 [#970] fstree: Move delete implementation to a separate file
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-02-09 16:12:11 +00:00
Evgenii Stratonikov 7f692409cf [#970] fstree: Handle unsupported O_TMPFILE
Metabase test relied on this behaviour, so fix the test too.

Cherry-picking was hard and did too many conflicts,
here is an original PR:
https://github.com/nspcc-dev/neofs-node/pull/2624

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-02-09 16:12:11 +00:00
Roman Khimov fc31b9c947 [#970] fstree: Add linux-specific file writer using O_TMPFILE
O_TMPFILE is implemented for all modern FSes and it's much easier and safer to
use. If application crashes in the middle of writing this file would be gone
and won't leave any garbage.

Notice that this implementation makes a different choice wrt EEXIST handling,
generic one always overwrites, while this one keeps the old data.

There is no real performance difference.

SSD (slow&old), XFS, Core i7-8565U:

Sync
```
name                                  old time/op    new time/op    delta
Put/size=1024,thread=1/fstree-8         1.74ms ± 3%    0.06ms ± 7%  -96.31%  (p=0.000 n=10+10)
Put/size=1024,thread=20/fstree-8        10.0ms ±41%     1.1ms ±18%  -88.95%  (p=0.000 n=9+10)
Put/size=1024,thread=100/fstree-8       32.3ms ±60%     6.5ms ±14%  -79.97%  (p=0.000 n=10+10)
Put/size=1048576,thread=1/fstree-8      17.8ms ±90%     3.4ms ±70%  -81.08%  (p=0.000 n=10+10)
Put/size=1048576,thread=20/fstree-8     103ms ±174%    112ms ±158%     ~     (p=0.971 n=10+10)
Put/size=1048576,thread=100/fstree-8     949ms ±78%    583ms ±132%     ~     (p=0.089 n=10+10)

name                                  old alloc/op   new alloc/op   delta
Put/size=1024,thread=1/fstree-8         3.17kB ± 1%    1.96kB ± 0%  -38.09%  (p=0.000 n=10+10)
Put/size=1024,thread=20/fstree-8        59.6kB ± 1%    39.2kB ± 1%  -34.30%  (p=0.000 n=8+10)
Put/size=1024,thread=100/fstree-8        299kB ± 0%     198kB ± 0%  -33.90%  (p=0.000 n=7+9)
Put/size=1048576,thread=1/fstree-8      3.38kB ± 1%    2.36kB ± 1%  -30.22%  (p=0.000 n=10+10)
Put/size=1048576,thread=20/fstree-8     65.7kB ± 4%    47.7kB ± 6%  -27.27%  (p=0.000 n=10+10)
Put/size=1048576,thread=100/fstree-8     351kB ± 8%     245kB ± 8%  -30.22%  (p=0.000 n=10+10)

name                                  old allocs/op  new allocs/op  delta
Put/size=1024,thread=1/fstree-8           30.3 ± 2%      21.0 ± 0%  -30.69%  (p=0.000 n=10+10)
Put/size=1024,thread=20/fstree-8           554 ± 1%       413 ± 0%  -25.35%  (p=0.000 n=8+10)
Put/size=1024,thread=100/fstree-8        2.77k ± 0%     2.07k ± 0%  -25.27%  (p=0.000 n=7+10)
Put/size=1048576,thread=1/fstree-8        32.0 ± 0%      25.0 ± 0%  -21.88%  (p=0.000 n=9+8)
Put/size=1048576,thread=20/fstree-8        609 ± 5%       494 ± 6%  -18.93%  (p=0.000 n=10+10)
Put/size=1048576,thread=100/fstree-8     3.25k ± 9%     2.50k ± 8%  -23.21%  (p=0.000 n=10+10)
```

No sync
```
name                                  old time/op    new time/op    delta
Put/size=1024,thread=1/fstree-8         71.3µs ±10%    59.8µs ±10%  -16.21%  (p=0.000 n=10+10)
Put/size=1024,thread=20/fstree-8        1.43ms ± 6%    1.22ms ±13%  -14.53%  (p=0.000 n=10+10)
Put/size=1024,thread=100/fstree-8       8.12ms ± 3%    6.36ms ± 2%  -21.67%  (p=0.000 n=8+9)
Put/size=1048576,thread=1/fstree-8      1.88ms ±70%    1.61ms ±78%     ~     (p=0.393 n=10+10)
Put/size=1048576,thread=20/fstree-8     32.7ms ±28%   34.2ms ±112%     ~     (p=0.968 n=9+10)
Put/size=1048576,thread=100/fstree-8     262ms ±56%     226ms ±34%     ~     (p=0.447 n=10+9)

name                                  old alloc/op   new alloc/op   delta
Put/size=1024,thread=1/fstree-8         2.89kB ± 0%    1.96kB ± 0%  -32.28%  (p=0.000 n=10+10)
Put/size=1024,thread=20/fstree-8        58.2kB ± 0%    39.5kB ± 0%  -32.09%  (p=0.000 n=8+8)
Put/size=1024,thread=100/fstree-8        291kB ± 0%     198kB ± 0%  -32.19%  (p=0.000 n=9+9)
Put/size=1048576,thread=1/fstree-8      3.05kB ± 1%    2.13kB ± 1%  -30.16%  (p=0.000 n=10+9)
Put/size=1048576,thread=20/fstree-8     62.6kB ± 0%    44.3kB ± 0%  -29.23%  (p=0.000 n=9+9)
Put/size=1048576,thread=100/fstree-8     302kB ± 0%     210kB ± 1%  -30.39%  (p=0.000 n=9+9)

name                                  old allocs/op  new allocs/op  delta
Put/size=1024,thread=1/fstree-8           27.0 ± 0%      21.0 ± 0%  -22.22%  (p=0.000 n=10+10)
Put/size=1024,thread=20/fstree-8           539 ± 0%       415 ± 0%  -22.98%  (p=0.000 n=10+10)
Put/size=1024,thread=100/fstree-8        2.69k ± 0%     2.07k ± 0%  -23.09%  (p=0.000 n=9+9)
Put/size=1048576,thread=1/fstree-8        28.0 ± 0%      22.3 ± 3%  -20.36%  (p=0.000 n=8+10)
Put/size=1048576,thread=20/fstree-8        577 ± 0%       458 ± 0%  -20.72%  (p=0.000 n=9+9)
Put/size=1048576,thread=100/fstree-8     2.76k ± 0%     2.15k ± 0%  -22.05%  (p=0.000 n=9+8)
```

HDD (LVM), ext4, Ryzen 5 1600:

Sync
```
                                      │ fs.sync-generic │            fs.sync-linux            │
                                      │     sec/op      │    sec/op     vs base               │
Put/size=1024,thread=1/fstree-12           34.70m ± 19%   33.59m ± 16%       ~ (p=0.529 n=10)
Put/size=1024,thread=20/fstree-12          188.8m ±  8%   189.2m ± 16%       ~ (p=0.739 n=10)
Put/size=1024,thread=100/fstree-12         264.8m ± 22%   273.6m ± 28%       ~ (p=0.353 n=10)
Put/size=1048576,thread=1/fstree-12        54.90m ± 14%   47.08m ± 18%       ~ (p=0.063 n=10)
Put/size=1048576,thread=20/fstree-12       244.1m ± 14%   220.4m ± 22%       ~ (p=0.579 n=10)
Put/size=1048576,thread=100/fstree-12      847.2m ±  5%   893.6m ±  3%  +5.48% (p=0.000 n=10)
geomean                                    164.3m         158.9m        -3.29%

                                      │ fs.sync-generic │            fs.sync-linux             │
                                      │      B/op       │     B/op      vs base                │
Put/size=1024,thread=1/fstree-12           3.375Ki ± 1%   2.471Ki ± 1%  -26.80% (p=0.000 n=10)
Put/size=1024,thread=20/fstree-12          66.62Ki ± 6%   49.21Ki ± 6%  -26.15% (p=0.000 n=10)
Put/size=1024,thread=100/fstree-12         319.2Ki ± 1%   230.9Ki ± 2%  -27.64% (p=0.000 n=10)
Put/size=1048576,thread=1/fstree-12        3.457Ki ± 1%   2.559Ki ± 1%  -25.97% (p=0.000 n=10)
Put/size=1048576,thread=20/fstree-12       66.91Ki ± 1%   49.16Ki ± 1%  -26.52% (p=0.000 n=10)
Put/size=1048576,thread=100/fstree-12      338.8Ki ± 2%   252.3Ki ± 3%  -25.54% (p=0.000 n=10)
geomean                                    42.17Ki        31.02Ki       -26.44%

                                      │ fs.sync-generic │            fs.sync-linux            │
                                      │    allocs/op    │  allocs/op   vs base                │
Put/size=1024,thread=1/fstree-12             33.00 ± 0%    27.00 ± 0%  -18.18% (p=0.000 n=10)
Put/size=1024,thread=20/fstree-12            639.5 ± 1%    519.0 ± 2%  -18.84% (p=0.000 n=10)
Put/size=1024,thread=100/fstree-12          3.059k ± 1%   2.478k ± 2%  -18.99% (p=0.000 n=10)
Put/size=1048576,thread=1/fstree-12          33.50 ± 1%    28.00 ± 4%  -16.42% (p=0.000 n=10)
Put/size=1048576,thread=20/fstree-12         638.5 ± 1%    520.0 ± 1%  -18.56% (p=0.000 n=10)
Put/size=1048576,thread=100/fstree-12       3.209k ± 2%   2.655k ± 2%  -17.28% (p=0.000 n=10)
geomean                                      405.3         332.1       -18.05%
```

No sync
```
                                      │ fs.nosync-generic │             fs.nosync-linux              │
                                      │      sec/op       │    sec/op     vs base                    │
Put/size=1024,thread=1/fstree-12           148.2µ ± 20%     136.6µ ± 19%   -7.89% (p=0.029 n=10)
Put/size=1024,thread=20/fstree-12          1.140m ± 26%     1.364m ± 16%        ~ (p=0.143 n=10)
Put/size=1024,thread=100/fstree-12         11.93m ± 68%     26.89m ± 62%        ~ (p=0.123 n=10)
Put/size=1048576,thread=1/fstree-12        1.302m ±  3%     1.287m ±  5%        ~ (p=0.481 n=10)
Put/size=1048576,thread=20/fstree-12       77.52m ±  8%     74.07m ±  7%        ~ (p=0.278 n=10+9)
Put/size=1048576,thread=100/fstree-12      226.1m ±   ∞ ¹
geomean                                    5.986m           3.434m        +18.60%                  ²
¹ need >= 6 samples for confidence interval at level 0.95
² benchmark set differs from baseline; geomeans may not be comparable

                                      │ fs.nosync-generic │             fs.nosync-linux              │
                                      │       B/op        │     B/op      vs base                    │
Put/size=1024,thread=1/fstree-12           2.879Ki ± 0%     1.972Ki ± 0%  -31.51% (p=0.000 n=10)
Put/size=1024,thread=20/fstree-12          55.94Ki ± 1%     37.90Ki ± 1%  -32.25% (p=0.000 n=10)
Put/size=1024,thread=100/fstree-12         272.6Ki ± 0%     182.1Ki ± 9%  -33.21% (p=0.000 n=10)
Put/size=1048576,thread=1/fstree-12        3.158Ki ± 0%     2.259Ki ± 0%  -28.46% (p=0.000 n=10)
Put/size=1048576,thread=20/fstree-12       58.87Ki ± 0%     41.03Ki ± 0%  -30.30% (p=0.000 n=10+9)
Put/size=1048576,thread=100/fstree-12      299.8Ki ±  ∞ ¹
geomean                                    36.71Ki          16.60Ki       -31.17%                  ²
¹ need >= 6 samples for confidence interval at level 0.95
² benchmark set differs from baseline; geomeans may not be comparable

                                      │ fs.nosync-generic │            fs.nosync-linux            │
                                      │     allocs/op     │  allocs/op   vs base                  │
Put/size=1024,thread=1/fstree-12             28.00 ± 0%      22.00 ± 0%  -21.43% (p=0.000 n=10)
Put/size=1024,thread=20/fstree-12            530.0 ± 0%      407.5 ± 1%  -23.11% (p=0.000 n=10)
Put/size=1024,thread=100/fstree-12          2.567k ± 0%     1.956k ± 9%  -23.77% (p=0.000 n=10)
Put/size=1048576,thread=1/fstree-12          30.00 ± 0%      24.00 ± 0%  -20.00% (p=0.000 n=10)
Put/size=1048576,thread=20/fstree-12         553.5 ± 0%      434.0 ± 0%  -21.59% (n=10+9)
Put/size=1048576,thread=100/fstree-12       2.803k ±  ∞ ¹
geomean                                      347.9           178.8       -21.99%                ²
¹ need >= 6 samples for confidence interval at level 0.95
² benchmark set differs from baseline; geomeans may not be comparable
```

Signed-off-by: Roman Khimov <roman@nspcc.ru>
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-02-09 16:12:11 +00:00
Evgenii Stratonikov ff488b53a1 [#970] fstree: Move write functions to a separate file
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-02-09 16:12:11 +00:00
Evgenii Stratonikov 9a622a750d [#970] fstree: Move temporary path handling in a separate function
Allow to easier test different implementations.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-02-09 16:12:11 +00:00
Anton Nikiforov d19ade23c8 [#959] node: Set mode to shard's components when open it
Avoid opening database for `metabase` and `cache` in `Degraded` mode.

Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-02-09 14:04:01 +00:00
Dmitrii Stepanov db67c21d55 [#947] engine: Evacuate trees to remote nodes
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-02-09 11:33:15 +03:00
Dmitrii Stepanov 728150d1d2 [#947] engine: Evacuate trees to local shards
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-02-09 11:33:15 +03:00
Dmitrii Stepanov 15d853ea22 [#947] controlSvc: Return tree evacuation stat
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-02-09 11:20:39 +03:00
Dmitrii Stepanov b3f3505ada [#947] cli: Allow to specify evacuation scope
It may be required to evacuate only objects or only tree or all, so
now it spossible to specify.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-02-09 11:20:38 +03:00
Dmitrii Stepanov a6eb66bf9c [#947] evacuate: Refactor evacuate parameters
Drop methods to make it easier to extend.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-02-09 11:20:38 +03:00
Dmitrii Stepanov 8e2a0611f4 [#947] tree: Add method to list all trees
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-02-09 11:20:38 +03:00
Dmitrii Stepanov cfc5ce7853 [#964] metabase: Drop GC marks if object not found
GC inhumes expired locks and tombstones on all the shards.
So it could be GC mark without object.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-02-08 07:54:39 +00:00
Dmitrii Stepanov 9ba48c582d [#917] engine: Allow to detach shards
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-02-06 14:49:47 +03:00
Anton Nikiforov d0eadf7ea2 [#799] engine: Skip put when object removed from shard
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-02-01 17:49:22 +00:00
Anton Nikiforov e3573de6db [#930] gc: Stop internal activity by context
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-01-31 08:30:34 +00:00
Dmitrii Stepanov 675eec91f3 [#938] shard: Update only changed counters
DCO action / DCO (pull_request) Successful in 5m40s Details
Vulncheck / Vulncheck (pull_request) Successful in 5m33s Details
Tests and linters / Staticcheck (pull_request) Successful in 7m12s Details
Build / Build Components (1.21) (pull_request) Successful in 7m25s Details
Build / Build Components (1.20) (pull_request) Successful in 7m29s Details
Tests and linters / Lint (pull_request) Successful in 8m26s Details
Tests and linters / Tests (1.20) (pull_request) Successful in 9m48s Details
Tests and linters / Tests with -race (pull_request) Successful in 9m56s Details
Tests and linters / Tests (1.21) (pull_request) Successful in 10m10s Details
If metric value hasn't changed, but we update metric, then
non existed metric will apear with zero value.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-01-30 12:37:48 +03:00
Dmitrii Stepanov c681354afd [#938] engine: Fix container count removal
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-01-30 12:37:48 +03:00
Dmitrii Stepanov 5ed330e436 [#927] metabase: Delete GC marks
DCO action / DCO (pull_request) Successful in 6m34s Details
Vulncheck / Vulncheck (pull_request) Successful in 7m19s Details
Build / Build Components (1.20) (pull_request) Successful in 7m48s Details
Build / Build Components (1.21) (pull_request) Successful in 9m39s Details
Tests and linters / Staticcheck (pull_request) Successful in 10m36s Details
Tests and linters / Lint (pull_request) Successful in 11m26s Details
Tests and linters / Tests (1.20) (pull_request) Successful in 12m57s Details
Tests and linters / Tests (1.21) (pull_request) Successful in 13m8s Details
Tests and linters / Tests with -race (pull_request) Successful in 13m7s Details
`key` is changed inside `db.get`, so encode address again after get.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-01-24 18:51:16 +03:00
Dmitrii Stepanov 931a5e9aaf [#918] engine: Move shard to degraded mode if metabase open failed
DCO action / DCO (pull_request) Successful in 1m55s Details
Build / Build Components (1.20) (pull_request) Successful in 2m14s Details
Vulncheck / Vulncheck (pull_request) Successful in 2m15s Details
Build / Build Components (1.21) (pull_request) Successful in 4m6s Details
Tests and linters / Tests (1.21) (pull_request) Successful in 5m58s Details
Tests and linters / Tests (1.20) (pull_request) Successful in 6m24s Details
Tests and linters / Staticcheck (pull_request) Successful in 6m11s Details
Tests and linters / Lint (pull_request) Successful in 6m37s Details
Tests and linters / Tests with -race (pull_request) Successful in 6m35s Details
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-01-23 11:16:40 +03:00
Dmitrii Stepanov f526f49995 [#874] engine: Check object existance concurrently
DCO action / DCO (pull_request) Successful in 1m36s Details
Vulncheck / Vulncheck (pull_request) Successful in 2m33s Details
Build / Build Components (1.21) (pull_request) Successful in 3m9s Details
Build / Build Components (1.20) (pull_request) Successful in 4m49s Details
Tests and linters / Tests (1.21) (pull_request) Successful in 6m5s Details
Tests and linters / Staticcheck (pull_request) Successful in 6m6s Details
Tests and linters / Tests (1.20) (pull_request) Successful in 6m39s Details
Tests and linters / Lint (pull_request) Successful in 7m0s Details
Tests and linters / Tests with -race (pull_request) Successful in 6m46s Details
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-01-23 09:28:29 +03:00
Dmitrii Stepanov f5160b27fc [#920] tests: Fix data races
DCO action / DCO (pull_request) Successful in 2m2s Details
Build / Build Components (1.21) (pull_request) Successful in 3m35s Details
Vulncheck / Vulncheck (pull_request) Successful in 3m3s Details
Tests and linters / Staticcheck (pull_request) Successful in 5m23s Details
Tests and linters / Tests (1.20) (pull_request) Successful in 7m33s Details
Tests and linters / Tests (1.21) (pull_request) Successful in 7m41s Details
Tests and linters / Tests with -race (pull_request) Successful in 7m47s Details
Tests and linters / Lint (pull_request) Successful in 8m12s Details
Build / Build Components (1.20) (pull_request) Successful in 1m32s Details
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-01-19 14:06:05 +03:00
Dmitrii Stepanov 63d3ed1ad8 [#904] tests: Close test engine after test
DCO action / DCO (pull_request) Successful in 2m7s Details
Vulncheck / Vulncheck (pull_request) Successful in 2m37s Details
Build / Build Components (1.20) (pull_request) Successful in 4m0s Details
Build / Build Components (1.21) (pull_request) Successful in 3m56s Details
Tests and linters / Tests (1.21) (pull_request) Successful in 6m25s Details
Tests and linters / Staticcheck (pull_request) Successful in 6m22s Details
Tests and linters / Tests (1.20) (pull_request) Successful in 6m43s Details
Tests and linters / Tests with -race (pull_request) Successful in 6m47s Details
Tests and linters / Lint (pull_request) Successful in 7m5s Details
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-01-17 19:04:39 +03:00
Dmitrii Stepanov 57171907e3 [#904] metabase: Return if object was actuall inserted
This requires to count metrics properly.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-01-17 19:04:39 +03:00
Dmitrii Stepanov c1a80235db [#904] metabase: Log Inhume operation
It will be very useful for troubleshooting.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-01-17 18:42:52 +03:00
Dmitrii Stepanov a2ab373a0a [#895] metabase: Do not delete GC mark for virtual objects
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-01-11 12:32:09 +00:00