dstepanov-yadro/frostfs-node
1
0
Fork 0
forked from TrueCloudLab/frostfs-node
Code Issues Pull requests Projects Releases Packages Wiki Activity
4499 commits 260 branches 26 tags 23 MiB
Commit graph

320 commits

Author SHA1 Message Date
Dmitrii Stepanov
6bafdab004 [#988] go.mod: Update api-go and sdk-go versions 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2024-02-15 17:20:59 +03:00
Dmitrii Stepanov
962e5a9c19 [#736] logger: Add journald support 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2024-02-12 06:17:22 +00:00
Roman Khimov
fc31b9c947 [#970] fstree: Add linux-specific file writer using O_TMPFILE 
O_TMPFILE is implemented for all modern FSes and it's much easier and safer to
use. If application crashes in the middle of writing this file would be gone
and won't leave any garbage.

Notice that this implementation makes a different choice wrt EEXIST handling,
generic one always overwrites, while this one keeps the old data.

There is no real performance difference.

SSD (slow&old), XFS, Core i7-8565U:

Sync
```
name                                  old time/op    new time/op    delta
Put/size=1024,thread=1/fstree-8         1.74ms ± 3%    0.06ms ± 7%  -96.31%  (p=0.000 n=10+10)
Put/size=1024,thread=20/fstree-8        10.0ms ±41%     1.1ms ±18%  -88.95%  (p=0.000 n=9+10)
Put/size=1024,thread=100/fstree-8       32.3ms ±60%     6.5ms ±14%  -79.97%  (p=0.000 n=10+10)
Put/size=1048576,thread=1/fstree-8      17.8ms ±90%     3.4ms ±70%  -81.08%  (p=0.000 n=10+10)
Put/size=1048576,thread=20/fstree-8     103ms ±174%    112ms ±158%     ~     (p=0.971 n=10+10)
Put/size=1048576,thread=100/fstree-8     949ms ±78%    583ms ±132%     ~     (p=0.089 n=10+10)

name                                  old alloc/op   new alloc/op   delta
Put/size=1024,thread=1/fstree-8         3.17kB ± 1%    1.96kB ± 0%  -38.09%  (p=0.000 n=10+10)
Put/size=1024,thread=20/fstree-8        59.6kB ± 1%    39.2kB ± 1%  -34.30%  (p=0.000 n=8+10)
Put/size=1024,thread=100/fstree-8        299kB ± 0%     198kB ± 0%  -33.90%  (p=0.000 n=7+9)
Put/size=1048576,thread=1/fstree-8      3.38kB ± 1%    2.36kB ± 1%  -30.22%  (p=0.000 n=10+10)
Put/size=1048576,thread=20/fstree-8     65.7kB ± 4%    47.7kB ± 6%  -27.27%  (p=0.000 n=10+10)
Put/size=1048576,thread=100/fstree-8     351kB ± 8%     245kB ± 8%  -30.22%  (p=0.000 n=10+10)

name                                  old allocs/op  new allocs/op  delta
Put/size=1024,thread=1/fstree-8           30.3 ± 2%      21.0 ± 0%  -30.69%  (p=0.000 n=10+10)
Put/size=1024,thread=20/fstree-8           554 ± 1%       413 ± 0%  -25.35%  (p=0.000 n=8+10)
Put/size=1024,thread=100/fstree-8        2.77k ± 0%     2.07k ± 0%  -25.27%  (p=0.000 n=7+10)
Put/size=1048576,thread=1/fstree-8        32.0 ± 0%      25.0 ± 0%  -21.88%  (p=0.000 n=9+8)
Put/size=1048576,thread=20/fstree-8        609 ± 5%       494 ± 6%  -18.93%  (p=0.000 n=10+10)
Put/size=1048576,thread=100/fstree-8     3.25k ± 9%     2.50k ± 8%  -23.21%  (p=0.000 n=10+10)
```

No sync
```
name                                  old time/op    new time/op    delta
Put/size=1024,thread=1/fstree-8         71.3µs ±10%    59.8µs ±10%  -16.21%  (p=0.000 n=10+10)
Put/size=1024,thread=20/fstree-8        1.43ms ± 6%    1.22ms ±13%  -14.53%  (p=0.000 n=10+10)
Put/size=1024,thread=100/fstree-8       8.12ms ± 3%    6.36ms ± 2%  -21.67%  (p=0.000 n=8+9)
Put/size=1048576,thread=1/fstree-8      1.88ms ±70%    1.61ms ±78%     ~     (p=0.393 n=10+10)
Put/size=1048576,thread=20/fstree-8     32.7ms ±28%   34.2ms ±112%     ~     (p=0.968 n=9+10)
Put/size=1048576,thread=100/fstree-8     262ms ±56%     226ms ±34%     ~     (p=0.447 n=10+9)

name                                  old alloc/op   new alloc/op   delta
Put/size=1024,thread=1/fstree-8         2.89kB ± 0%    1.96kB ± 0%  -32.28%  (p=0.000 n=10+10)
Put/size=1024,thread=20/fstree-8        58.2kB ± 0%    39.5kB ± 0%  -32.09%  (p=0.000 n=8+8)
Put/size=1024,thread=100/fstree-8        291kB ± 0%     198kB ± 0%  -32.19%  (p=0.000 n=9+9)
Put/size=1048576,thread=1/fstree-8      3.05kB ± 1%    2.13kB ± 1%  -30.16%  (p=0.000 n=10+9)
Put/size=1048576,thread=20/fstree-8     62.6kB ± 0%    44.3kB ± 0%  -29.23%  (p=0.000 n=9+9)
Put/size=1048576,thread=100/fstree-8     302kB ± 0%     210kB ± 1%  -30.39%  (p=0.000 n=9+9)

name                                  old allocs/op  new allocs/op  delta
Put/size=1024,thread=1/fstree-8           27.0 ± 0%      21.0 ± 0%  -22.22%  (p=0.000 n=10+10)
Put/size=1024,thread=20/fstree-8           539 ± 0%       415 ± 0%  -22.98%  (p=0.000 n=10+10)
Put/size=1024,thread=100/fstree-8        2.69k ± 0%     2.07k ± 0%  -23.09%  (p=0.000 n=9+9)
Put/size=1048576,thread=1/fstree-8        28.0 ± 0%      22.3 ± 3%  -20.36%  (p=0.000 n=8+10)
Put/size=1048576,thread=20/fstree-8        577 ± 0%       458 ± 0%  -20.72%  (p=0.000 n=9+9)
Put/size=1048576,thread=100/fstree-8     2.76k ± 0%     2.15k ± 0%  -22.05%  (p=0.000 n=9+8)
```

HDD (LVM), ext4, Ryzen 5 1600:

Sync
```
                                      │ fs.sync-generic │            fs.sync-linux            │
                                      │     sec/op      │    sec/op     vs base               │
Put/size=1024,thread=1/fstree-12           34.70m ± 19%   33.59m ± 16%       ~ (p=0.529 n=10)
Put/size=1024,thread=20/fstree-12          188.8m ±  8%   189.2m ± 16%       ~ (p=0.739 n=10)
Put/size=1024,thread=100/fstree-12         264.8m ± 22%   273.6m ± 28%       ~ (p=0.353 n=10)
Put/size=1048576,thread=1/fstree-12        54.90m ± 14%   47.08m ± 18%       ~ (p=0.063 n=10)
Put/size=1048576,thread=20/fstree-12       244.1m ± 14%   220.4m ± 22%       ~ (p=0.579 n=10)
Put/size=1048576,thread=100/fstree-12      847.2m ±  5%   893.6m ±  3%  +5.48% (p=0.000 n=10)
geomean                                    164.3m         158.9m        -3.29%

                                      │ fs.sync-generic │            fs.sync-linux             │
                                      │      B/op       │     B/op      vs base                │
Put/size=1024,thread=1/fstree-12           3.375Ki ± 1%   2.471Ki ± 1%  -26.80% (p=0.000 n=10)
Put/size=1024,thread=20/fstree-12          66.62Ki ± 6%   49.21Ki ± 6%  -26.15% (p=0.000 n=10)
Put/size=1024,thread=100/fstree-12         319.2Ki ± 1%   230.9Ki ± 2%  -27.64% (p=0.000 n=10)
Put/size=1048576,thread=1/fstree-12        3.457Ki ± 1%   2.559Ki ± 1%  -25.97% (p=0.000 n=10)
Put/size=1048576,thread=20/fstree-12       66.91Ki ± 1%   49.16Ki ± 1%  -26.52% (p=0.000 n=10)
Put/size=1048576,thread=100/fstree-12      338.8Ki ± 2%   252.3Ki ± 3%  -25.54% (p=0.000 n=10)
geomean                                    42.17Ki        31.02Ki       -26.44%

                                      │ fs.sync-generic │            fs.sync-linux            │
                                      │    allocs/op    │  allocs/op   vs base                │
Put/size=1024,thread=1/fstree-12             33.00 ± 0%    27.00 ± 0%  -18.18% (p=0.000 n=10)
Put/size=1024,thread=20/fstree-12            639.5 ± 1%    519.0 ± 2%  -18.84% (p=0.000 n=10)
Put/size=1024,thread=100/fstree-12          3.059k ± 1%   2.478k ± 2%  -18.99% (p=0.000 n=10)
Put/size=1048576,thread=1/fstree-12          33.50 ± 1%    28.00 ± 4%  -16.42% (p=0.000 n=10)
Put/size=1048576,thread=20/fstree-12         638.5 ± 1%    520.0 ± 1%  -18.56% (p=0.000 n=10)
Put/size=1048576,thread=100/fstree-12       3.209k ± 2%   2.655k ± 2%  -17.28% (p=0.000 n=10)
geomean                                      405.3         332.1       -18.05%
```

No sync
```
                                      │ fs.nosync-generic │             fs.nosync-linux              │
                                      │      sec/op       │    sec/op     vs base                    │
Put/size=1024,thread=1/fstree-12           148.2µ ± 20%     136.6µ ± 19%   -7.89% (p=0.029 n=10)
Put/size=1024,thread=20/fstree-12          1.140m ± 26%     1.364m ± 16%        ~ (p=0.143 n=10)
Put/size=1024,thread=100/fstree-12         11.93m ± 68%     26.89m ± 62%        ~ (p=0.123 n=10)
Put/size=1048576,thread=1/fstree-12        1.302m ±  3%     1.287m ±  5%        ~ (p=0.481 n=10)
Put/size=1048576,thread=20/fstree-12       77.52m ±  8%     74.07m ±  7%        ~ (p=0.278 n=10+9)
Put/size=1048576,thread=100/fstree-12      226.1m ±   ∞ ¹
geomean                                    5.986m           3.434m        +18.60%                  ²
¹ need >= 6 samples for confidence interval at level 0.95
² benchmark set differs from baseline; geomeans may not be comparable

                                      │ fs.nosync-generic │             fs.nosync-linux              │
                                      │       B/op        │     B/op      vs base                    │
Put/size=1024,thread=1/fstree-12           2.879Ki ± 0%     1.972Ki ± 0%  -31.51% (p=0.000 n=10)
Put/size=1024,thread=20/fstree-12          55.94Ki ± 1%     37.90Ki ± 1%  -32.25% (p=0.000 n=10)
Put/size=1024,thread=100/fstree-12         272.6Ki ± 0%     182.1Ki ± 9%  -33.21% (p=0.000 n=10)
Put/size=1048576,thread=1/fstree-12        3.158Ki ± 0%     2.259Ki ± 0%  -28.46% (p=0.000 n=10)
Put/size=1048576,thread=20/fstree-12       58.87Ki ± 0%     41.03Ki ± 0%  -30.30% (p=0.000 n=10+9)
Put/size=1048576,thread=100/fstree-12      299.8Ki ±  ∞ ¹
geomean                                    36.71Ki          16.60Ki       -31.17%                  ²
¹ need >= 6 samples for confidence interval at level 0.95
² benchmark set differs from baseline; geomeans may not be comparable

                                      │ fs.nosync-generic │            fs.nosync-linux            │
                                      │     allocs/op     │  allocs/op   vs base                  │
Put/size=1024,thread=1/fstree-12             28.00 ± 0%      22.00 ± 0%  -21.43% (p=0.000 n=10)
Put/size=1024,thread=20/fstree-12            530.0 ± 0%      407.5 ± 1%  -23.11% (p=0.000 n=10)
Put/size=1024,thread=100/fstree-12          2.567k ± 0%     1.956k ± 9%  -23.77% (p=0.000 n=10)
Put/size=1048576,thread=1/fstree-12          30.00 ± 0%      24.00 ± 0%  -20.00% (p=0.000 n=10)
Put/size=1048576,thread=20/fstree-12         553.5 ± 0%      434.0 ± 0%  -21.59% (n=10+9)
Put/size=1048576,thread=100/fstree-12       2.803k ±  ∞ ¹
geomean                                      347.9           178.8       -21.99%                ²
¹ need >= 6 samples for confidence interval at level 0.95
² benchmark set differs from baseline; geomeans may not be comparable
```

Signed-off-by: Roman Khimov <roman@nspcc.ru>
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2024-02-09 16:12:11 +00:00
Olga Konstantinova
80b581d499 [#466] adm: Allow to download contracts from Gitea 
Signed-off-by: Olga Konstantinova <kola43843@gmail.com>
 2024-02-08 21:07:49 +00:00
Anton Nikiforov
2cb04379a4 [#922] go.mod: Update APE 
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
 2024-02-02 12:09:51 +00:00
Airat Arifullin
5be2af881a [#934] container: Make container APE middleware read namespaces 
* Those methods that can access already existing containers and thus
  can get container properties should read namespace from Zone
  property. If Zone is not set, take a namespace for root.
* Otherwise, define namespaces by owner ID via frostfs-id contract.
* Improve unit-tests, consider more cases.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
 2024-02-01 17:38:24 +00:00
Evgenii Stratonikov
cc2da73b20 [#929] go.mod: Update dependencies 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2024-01-25 07:24:11 +00:00
Denis Kirillov
0e3d144695 [#908] adm: Support frostfsid basic operations 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2024-01-19 11:04:03 +00:00
Denis Kirillov
e43609c616 [#885] go.mod: Update APE 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2024-01-12 15:38:37 +00:00
Evgenii Stratonikov
530249e3bd [#893] go.mod: Update neo-go 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-12-29 15:03:47 +00:00
Dmitrii Stepanov
8180a0664f [#887] node: Drop badger writecache implementation 
Badger implementation isn't tested and works not well,
but requires human resources to maintain.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-12-22 13:00:54 +03:00
Anton Nikiforov
5b672fb392 [#876] adm: Add morph ape get/set-admin commands 
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
 2023-12-21 18:21:37 +03:00
Airat Arifullin
7d7cf05575 [#851] ape: Initialize and use policy contract interface 
* Replace inmemory policy contract mock by initialized
  policy contract interface.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
 2023-12-18 11:06:35 +00:00
Ekaterina Lebedeva
eca7ac9f0d [#552] Add sdnotify package 
To avoid using third-party dependencies.

Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
 2023-12-13 17:49:26 +03:00
Ekaterina Lebedeva
9b2dce5763 [#552] Add status notification to systemd 
Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
 2023-12-13 15:02:39 +03:00
Dmitrii Stepanov
d5c9dd3c83 [#852] ape: Use first match for eACL->APE converter 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-12-11 16:55:32 +03:00
aarifullin
e361e017f3 [#842] control: Pass target instead resource name 
* Update policy-engine package version in go.mod, go.sum.
* Refactor CheckIfRequestPermitted: pass container target
  instead container ID.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
 2023-12-07 14:21:55 +00:00
Evgenii Stratonikov
44806aa9f1 [#837] go.mod: Update dependencies 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-12-06 11:07:10 +00:00
Evgenii Stratonikov
5048236441 [#808] go.mod: Update dependencies 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-11-27 09:58:19 +00:00
Dmitrii Stepanov
c99157f0b2 [#821] go.mod: Update SDK-Go version 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-11-23 10:21:06 +03:00
aarifullin
02454df14a [#809] client: Refactor PrmInit, PrmDial usage 
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
 2023-11-17 13:37:03 +00:00
aarifullin
4d5be5ccb5 [#811] ape: Update policy-engine module version and rebase 
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
 2023-11-16 11:31:37 +03:00
Dmitrii Stepanov
fd9128d051 [#800] node: eACL -> APE converter 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-11-15 11:55:55 +03:00
Alexander Chuprov
364f835b7e [#740] logs: Add Loki 
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
 2023-11-14 19:01:05 +00:00
Airat Arifullin
5ec73fe8a0 [#770] node: Introduce ape chain source 
* Provide methods to access rule chains with access
  policy engine (APE) chain source
* Initialize apeChainSource within object service
  initialization
* Share apeChainSource with control service
* Implement dummy apeChainSource instance based on
  in-memory implementation

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
 2023-11-08 13:34:03 +00:00
aarifullin
74c91eeef5 [#777] client: Refactor PrmContainerList, PrmObjectSearch usage 
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
 2023-11-06 06:50:11 +00:00
Dmitrii Stepanov
7b1eda5107 [#531] go.mod: Update api-go version 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-11-02 17:32:41 +03:00
Evgenii Stratonikov
7c8591f83b [#779] go.mod: Update frostfs-contract 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-11-02 09:23:41 +00:00
Evgenii Stratonikov
261d281154 [#762] go.mod: Update sdk-go 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-10-31 11:22:29 +00:00
Evgenii Stratonikov
4f62fded01 [#752] go.mod: Update dependencies 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-10-25 16:06:44 +03:00
Evgenii Stratonikov
2dbf5c612a [#752] go.mod: Update neo-go to v0.103.0 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-10-25 16:06:44 +03:00
aarifullin
12b7cf2533 [#747] client: Refactor PrmObjectPutSingle usage 
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
 2023-10-20 11:55:40 +00:00
Dmitrii Stepanov
a0a35ffbec [#702] node: Update SDK version 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-09-29 18:41:48 +03:00
Evgenii Stratonikov
10570fc035 [#690] go.mod: Update contract and api-go 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-09-14 14:50:30 +03:00
Evgenii Stratonikov
f435ab1b26 [#682] go.mod: Update sdk-go 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-09-12 10:42:34 +03:00
Airat Arifullin
aa9f8dce3d [#677] client: Refactor PrmAnnounceSpace/EndpointInfo/NetworkInfo usage 
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
 2023-09-08 09:42:28 +03:00
Airat Arifullin
d9de9e2bbb [#675] client: Refactor PrmObjectDelete usage 
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
 2023-09-06 08:05:47 +00:00
Anton Nikiforov
4dff9555f1 [#568] writecache: Improve flushing scheme for badger 
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
 2023-08-30 17:22:28 +00:00
Airat Arifullin
806cc13d9f [#658] client: Refactor PrmObjectGet/Head/Range usage 
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
 2023-08-30 17:13:23 +00:00
Airat Arifullin
238b8f10a0 [#630] cli: Fix SDK SetEACLPrm usage for PrmContainerSetEACL 
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
 2023-08-22 14:25:39 +00:00
Airat Arifullin
84ea075587 [#625] cli: Fix SDK EACLPrm usage for PrmContainerEACL 
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
 2023-08-21 10:36:47 +03:00
Airat Arifullin
20af34ecdb [#591] cli: fix SDK PrmContainerDelete usage for DeleteContainerPrm 
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
 2023-08-10 16:40:35 +00:00
Alejandro Lopez
5b7e4a51b7 [#481] Update frostfs-sdk-go and error pointer receivers 
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
 2023-08-09 10:26:53 +00:00
Evgenii Stratonikov
de3d1eb99c [#581] Bump required go version to go1.20 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-08-09 10:16:48 +00:00
Alejandro Lopez
1a0cb0f34a [#421] Try using badger for the write-cache 
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
 2023-08-07 08:16:57 +00:00
Evgenii Stratonikov
435a581b5e [#557] go.mod: Update sdk-go and api-go 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-08-02 13:37:40 +00:00
Airat Arifullin
1b7b54ba89 [#540] cli: refactor client parameters usage 
Signed-off-by: Airat Arifullin a.arifullin@yadro.com
 2023-08-02 09:44:51 +00:00
Evgenii Stratonikov
b8052c794e [#547] go.mod: Update api-go, sdk-go 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-07-31 15:35:15 +00:00
Airat Arifullin
94df541426 [#530] go.mod: Update frostfs-sdk-go and frostfs-api-go versions 
Signed-off-by: Airat Arifullin a.arifullin@yadro.com
 2023-07-19 16:15:21 +03:00
Alejandro Lopez
4680087711 [#527] Add support for select-filter expressions in policy playground 
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
 2023-07-18 13:15:58 +00:00