4503 commits

This branch

This branch

All branches

Author	SHA1	Message	Date
Evgenii Stratonikov	45fd4e4ff1	[#972] pilorama: Remove removeDuplicatesInPlace() ... Also, check that slices.CompareFunc() indeed passes all the tests before removal. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2024-02-19 13:13:09 +00:00
Evgenii Stratonikov	e39a714c25	[#972] go.mod: Bump go version to go1.21 ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2024-02-19 13:13:09 +00:00
Evgenii Stratonikov	15fc5bac26	[#972] keyer: Use UncompressedBytes() for marshaling ... elliptic.Marshal() becomes deprecated in go1.21 Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2024-02-19 13:13:09 +00:00
Dmitrii Stepanov	2680192ba0	[#988] objectSvc: Fix SetMarshalData for PutSingle ... After api-go update it is required to pass marshal data to `SetMarshalData`. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-02-15 17:21:08 +03:00
Dmitrii Stepanov	6bafdab004	[#988] go.mod: Update api-go and sdk-go versions ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-02-15 17:20:59 +03:00
Anton Nikiforov	05b5f5ca85	[#959] writecache: Fix panic on Get when it is not initialized ... Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>	2024-02-14 16:10:33 +03:00
Anton Nikiforov	2429508ac5	[#959] shard: Skip rebuild in DEGRADED mode ... Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>	2024-02-14 15:39:28 +03:00
Anton Nikiforov	5cbf57081f	[#980] adm: Introduce flag chain-name for APE managing commands ... Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>	2024-02-14 09:41:56 +03:00
Anton Nikiforov	35370283ba	[#948] adm: Move TestNextPollInterval to package helper ... Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>	2024-02-13 10:08:49 +03:00
Anton Nikiforov	802192cfef	[#932] adm: Rename util to helper ... To avoid conflicts with `util` packages in other imports. Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>	2024-02-13 09:59:27 +03:00
Anton Nikiforov	e2cee4cf09	[#932] adm: Move const to package constants ... Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>	2024-02-13 09:59:27 +03:00
Anton Nikiforov	814c411f4a	[#932] adm: Move flags to package commonflags ... Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>	2024-02-13 09:59:27 +03:00
Anton Nikiforov	7b0e3f5010	[#932] adm: Remove unnecessary import alias usage ... Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>	2024-02-13 09:59:27 +03:00
Anton Nikiforov	63c34ea707	[#932] adm: Move command init to package initialize ... Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>	2024-02-13 09:59:27 +03:00
Anton Nikiforov	86b2515744	[#932] adm: Move generate.go to package generate ... Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>	2024-02-13 09:59:27 +03:00
Anton Nikiforov	b8cf0a6b88	[#932] adm: Move deploy to package contract ... Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>	2024-02-13 09:59:27 +03:00
Anton Nikiforov	76343f19e5	[#932] adm: Move update-contracts to package contract ... Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>	2024-02-13 09:59:27 +03:00
Anton Nikiforov	e2557b2f0b	[#932] adm: Move dump-hashes to package contract ... Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>	2024-02-13 09:59:27 +03:00
Anton Nikiforov	9b65f1595a	[#932] adm: Move force-new-epoch to package netmap ... Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>	2024-02-13 09:59:27 +03:00
Anton Nikiforov	ce42547980	[#932] adm: Move remove-nodes to package node ... Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>	2024-02-13 09:59:27 +03:00
Anton Nikiforov	9690bd02aa	[#932] adm: Move netmap-candidates to package netmap ... Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>	2024-02-13 09:59:27 +03:00
Anton Nikiforov	36fd6c663c	[#932] adm: Move dump/restore/list-containers to package container ... Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>	2024-02-13 09:59:27 +03:00
Anton Nikiforov	be15eab82a	[#932] adm: Move dump/set-config to package config ... Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>	2024-02-13 09:59:27 +03:00
Anton Nikiforov	f7a8f51c66	[#932] adm: Move command deposit-notary to package notary ... Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>	2024-02-13 09:59:27 +03:00
Anton Nikiforov	beb9d80e34	[#932] adm: Move command dump-balances to package balance ... Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>	2024-02-13 09:59:27 +03:00
Anton Nikiforov	8148c9dc19	[#932] adm: Move command morph proxy to package proxy ... Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>	2024-02-13 09:59:27 +03:00
Anton Nikiforov	218bd72f9a	[#932] adm: Move command frostfsid to package frostfsid ... Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>	2024-02-13 09:59:27 +03:00
Anton Nikiforov	a92188e5f9	[#932] adm: Reduce methods visibility in util package ... Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>	2024-02-13 09:59:27 +03:00
Anton Nikiforov	f6ff3de0ae	[#932] adm: Move set-/dump-policy to policy package ... Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>	2024-02-13 09:59:27 +03:00
Anton Nikiforov	77694a2f3b	[#932] adm: Move InitializeContext to util package ... Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>	2024-02-13 09:59:27 +03:00
Anton Nikiforov	b68f7be0b6	[#932] adm: Prepare to move InitializeContext to util package ... Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>	2024-02-13 09:59:27 +03:00
Anton Nikiforov	bee3741f4e	[#932] adm: Move cmd morph ape to ape package ... Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>	2024-02-13 09:59:27 +03:00
Anton Nikiforov	ba00fc4971	[#932] adm: Move contract name constants to util package ... Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>	2024-02-13 09:59:27 +03:00
Anton Nikiforov	fdeb99c52f	[#932] adm: Move DomainOf to util package ... Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>	2024-02-13 09:59:27 +03:00
Anton Nikiforov	34fcab3498	[#932] adm: Move alphabet-wallets flag to util package ... Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>	2024-02-13 09:59:27 +03:00
Anton Nikiforov	7954c7f8af	[#932] adm: Move NNS methods to util package ... Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>	2024-02-13 09:59:27 +03:00
Anton Nikiforov	cda3a3d834	[#932] adm: Refactor command morph ... Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>	2024-02-13 09:59:27 +03:00
Anton Nikiforov	0bd030507e	[#948] metrics: Set actual value for shard_id after restart ... Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>	2024-02-13 09:43:21 +03:00
Anton Nikiforov	6a5769d1da	[#948] Fix gofumpt issue ... Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>	2024-02-13 09:40:46 +03:00
Dmitrii Stepanov	3a41858a0f	[#975] ir: Add default logger destination ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-02-12 13:30:32 +03:00
Dmitrii Stepanov	962e5a9c19	[#736] logger: Add journald support ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-02-12 06:17:22 +00:00
Evgenii Stratonikov	b36a453238	[#970] fstree: Add build tag to enable generic version on linux ... Unless tested, generic version can start gaining bugs. With a separate build tag we can have the best of both worlds: 1. Use optimized implementation for linux by default. 2. Run tests or benchmarks for both. Note that they are not actually run automatically now, but this is at leas possible. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2024-02-09 16:12:11 +00:00
Evgenii Stratonikov	abd502215f	[#970] fstree: Move file locking to the generic writer ... It is not a part of FSTree itself, but rather a way to solve concurrent counter update on non-linux implementations. New linux implementations is pretty simple: link fails when the file exists, unlink fails when the file doesn't exist. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2024-02-09 16:12:11 +00:00
Evgenii Stratonikov	fb74524ac7	[#970] fstree: Move delete implementation to a separate file ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2024-02-09 16:12:11 +00:00
Evgenii Stratonikov	7f692409cf	[#970] fstree: Handle unsupported O_TMPFILE ... Metabase test relied on this behaviour, so fix the test too. Cherry-picking was hard and did too many conflicts, here is an original PR: https://github.com/nspcc-dev/neofs-node/pull/2624 Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2024-02-09 16:12:11 +00:00
Roman Khimov	fc31b9c947	[#970] fstree: Add linux-specific file writer using O_TMPFILE ... O_TMPFILE is implemented for all modern FSes and it's much easier and safer to use. If application crashes in the middle of writing this file would be gone and won't leave any garbage. Notice that this implementation makes a different choice wrt EEXIST handling, generic one always overwrites, while this one keeps the old data. There is no real performance difference. SSD (slow&old), XFS, Core i7-8565U: Sync ``` name old time/op new time/op delta Put/size=1024,thread=1/fstree-8 1.74ms ± 3% 0.06ms ± 7% -96.31% (p=0.000 n=10+10) Put/size=1024,thread=20/fstree-8 10.0ms ±41% 1.1ms ±18% -88.95% (p=0.000 n=9+10) Put/size=1024,thread=100/fstree-8 32.3ms ±60% 6.5ms ±14% -79.97% (p=0.000 n=10+10) Put/size=1048576,thread=1/fstree-8 17.8ms ±90% 3.4ms ±70% -81.08% (p=0.000 n=10+10) Put/size=1048576,thread=20/fstree-8 103ms ±174% 112ms ±158% ~ (p=0.971 n=10+10) Put/size=1048576,thread=100/fstree-8 949ms ±78% 583ms ±132% ~ (p=0.089 n=10+10) name old alloc/op new alloc/op delta Put/size=1024,thread=1/fstree-8 3.17kB ± 1% 1.96kB ± 0% -38.09% (p=0.000 n=10+10) Put/size=1024,thread=20/fstree-8 59.6kB ± 1% 39.2kB ± 1% -34.30% (p=0.000 n=8+10) Put/size=1024,thread=100/fstree-8 299kB ± 0% 198kB ± 0% -33.90% (p=0.000 n=7+9) Put/size=1048576,thread=1/fstree-8 3.38kB ± 1% 2.36kB ± 1% -30.22% (p=0.000 n=10+10) Put/size=1048576,thread=20/fstree-8 65.7kB ± 4% 47.7kB ± 6% -27.27% (p=0.000 n=10+10) Put/size=1048576,thread=100/fstree-8 351kB ± 8% 245kB ± 8% -30.22% (p=0.000 n=10+10) name old allocs/op new allocs/op delta Put/size=1024,thread=1/fstree-8 30.3 ± 2% 21.0 ± 0% -30.69% (p=0.000 n=10+10) Put/size=1024,thread=20/fstree-8 554 ± 1% 413 ± 0% -25.35% (p=0.000 n=8+10) Put/size=1024,thread=100/fstree-8 2.77k ± 0% 2.07k ± 0% -25.27% (p=0.000 n=7+10) Put/size=1048576,thread=1/fstree-8 32.0 ± 0% 25.0 ± 0% -21.88% (p=0.000 n=9+8) Put/size=1048576,thread=20/fstree-8 609 ± 5% 494 ± 6% -18.93% (p=0.000 n=10+10) Put/size=1048576,thread=100/fstree-8 3.25k ± 9% 2.50k ± 8% -23.21% (p=0.000 n=10+10) ``` No sync ``` name old time/op new time/op delta Put/size=1024,thread=1/fstree-8 71.3µs ±10% 59.8µs ±10% -16.21% (p=0.000 n=10+10) Put/size=1024,thread=20/fstree-8 1.43ms ± 6% 1.22ms ±13% -14.53% (p=0.000 n=10+10) Put/size=1024,thread=100/fstree-8 8.12ms ± 3% 6.36ms ± 2% -21.67% (p=0.000 n=8+9) Put/size=1048576,thread=1/fstree-8 1.88ms ±70% 1.61ms ±78% ~ (p=0.393 n=10+10) Put/size=1048576,thread=20/fstree-8 32.7ms ±28% 34.2ms ±112% ~ (p=0.968 n=9+10) Put/size=1048576,thread=100/fstree-8 262ms ±56% 226ms ±34% ~ (p=0.447 n=10+9) name old alloc/op new alloc/op delta Put/size=1024,thread=1/fstree-8 2.89kB ± 0% 1.96kB ± 0% -32.28% (p=0.000 n=10+10) Put/size=1024,thread=20/fstree-8 58.2kB ± 0% 39.5kB ± 0% -32.09% (p=0.000 n=8+8) Put/size=1024,thread=100/fstree-8 291kB ± 0% 198kB ± 0% -32.19% (p=0.000 n=9+9) Put/size=1048576,thread=1/fstree-8 3.05kB ± 1% 2.13kB ± 1% -30.16% (p=0.000 n=10+9) Put/size=1048576,thread=20/fstree-8 62.6kB ± 0% 44.3kB ± 0% -29.23% (p=0.000 n=9+9) Put/size=1048576,thread=100/fstree-8 302kB ± 0% 210kB ± 1% -30.39% (p=0.000 n=9+9) name old allocs/op new allocs/op delta Put/size=1024,thread=1/fstree-8 27.0 ± 0% 21.0 ± 0% -22.22% (p=0.000 n=10+10) Put/size=1024,thread=20/fstree-8 539 ± 0% 415 ± 0% -22.98% (p=0.000 n=10+10) Put/size=1024,thread=100/fstree-8 2.69k ± 0% 2.07k ± 0% -23.09% (p=0.000 n=9+9) Put/size=1048576,thread=1/fstree-8 28.0 ± 0% 22.3 ± 3% -20.36% (p=0.000 n=8+10) Put/size=1048576,thread=20/fstree-8 577 ± 0% 458 ± 0% -20.72% (p=0.000 n=9+9) Put/size=1048576,thread=100/fstree-8 2.76k ± 0% 2.15k ± 0% -22.05% (p=0.000 n=9+8) ``` HDD (LVM), ext4, Ryzen 5 1600: Sync ``` │ fs.sync-generic │ fs.sync-linux │ │ sec/op │ sec/op vs base │ Put/size=1024,thread=1/fstree-12 34.70m ± 19% 33.59m ± 16% ~ (p=0.529 n=10) Put/size=1024,thread=20/fstree-12 188.8m ± 8% 189.2m ± 16% ~ (p=0.739 n=10) Put/size=1024,thread=100/fstree-12 264.8m ± 22% 273.6m ± 28% ~ (p=0.353 n=10) Put/size=1048576,thread=1/fstree-12 54.90m ± 14% 47.08m ± 18% ~ (p=0.063 n=10) Put/size=1048576,thread=20/fstree-12 244.1m ± 14% 220.4m ± 22% ~ (p=0.579 n=10) Put/size=1048576,thread=100/fstree-12 847.2m ± 5% 893.6m ± 3% +5.48% (p=0.000 n=10) geomean 164.3m 158.9m -3.29% │ fs.sync-generic │ fs.sync-linux │ │ B/op │ B/op vs base │ Put/size=1024,thread=1/fstree-12 3.375Ki ± 1% 2.471Ki ± 1% -26.80% (p=0.000 n=10) Put/size=1024,thread=20/fstree-12 66.62Ki ± 6% 49.21Ki ± 6% -26.15% (p=0.000 n=10) Put/size=1024,thread=100/fstree-12 319.2Ki ± 1% 230.9Ki ± 2% -27.64% (p=0.000 n=10) Put/size=1048576,thread=1/fstree-12 3.457Ki ± 1% 2.559Ki ± 1% -25.97% (p=0.000 n=10) Put/size=1048576,thread=20/fstree-12 66.91Ki ± 1% 49.16Ki ± 1% -26.52% (p=0.000 n=10) Put/size=1048576,thread=100/fstree-12 338.8Ki ± 2% 252.3Ki ± 3% -25.54% (p=0.000 n=10) geomean 42.17Ki 31.02Ki -26.44% │ fs.sync-generic │ fs.sync-linux │ │ allocs/op │ allocs/op vs base │ Put/size=1024,thread=1/fstree-12 33.00 ± 0% 27.00 ± 0% -18.18% (p=0.000 n=10) Put/size=1024,thread=20/fstree-12 639.5 ± 1% 519.0 ± 2% -18.84% (p=0.000 n=10) Put/size=1024,thread=100/fstree-12 3.059k ± 1% 2.478k ± 2% -18.99% (p=0.000 n=10) Put/size=1048576,thread=1/fstree-12 33.50 ± 1% 28.00 ± 4% -16.42% (p=0.000 n=10) Put/size=1048576,thread=20/fstree-12 638.5 ± 1% 520.0 ± 1% -18.56% (p=0.000 n=10) Put/size=1048576,thread=100/fstree-12 3.209k ± 2% 2.655k ± 2% -17.28% (p=0.000 n=10) geomean 405.3 332.1 -18.05% ``` No sync ``` │ fs.nosync-generic │ fs.nosync-linux │ │ sec/op │ sec/op vs base │ Put/size=1024,thread=1/fstree-12 148.2µ ± 20% 136.6µ ± 19% -7.89% (p=0.029 n=10) Put/size=1024,thread=20/fstree-12 1.140m ± 26% 1.364m ± 16% ~ (p=0.143 n=10) Put/size=1024,thread=100/fstree-12 11.93m ± 68% 26.89m ± 62% ~ (p=0.123 n=10) Put/size=1048576,thread=1/fstree-12 1.302m ± 3% 1.287m ± 5% ~ (p=0.481 n=10) Put/size=1048576,thread=20/fstree-12 77.52m ± 8% 74.07m ± 7% ~ (p=0.278 n=10+9) Put/size=1048576,thread=100/fstree-12 226.1m ± ∞ ¹ geomean 5.986m 3.434m +18.60% ² ¹ need >= 6 samples for confidence interval at level 0.95 ² benchmark set differs from baseline; geomeans may not be comparable │ fs.nosync-generic │ fs.nosync-linux │ │ B/op │ B/op vs base │ Put/size=1024,thread=1/fstree-12 2.879Ki ± 0% 1.972Ki ± 0% -31.51% (p=0.000 n=10) Put/size=1024,thread=20/fstree-12 55.94Ki ± 1% 37.90Ki ± 1% -32.25% (p=0.000 n=10) Put/size=1024,thread=100/fstree-12 272.6Ki ± 0% 182.1Ki ± 9% -33.21% (p=0.000 n=10) Put/size=1048576,thread=1/fstree-12 3.158Ki ± 0% 2.259Ki ± 0% -28.46% (p=0.000 n=10) Put/size=1048576,thread=20/fstree-12 58.87Ki ± 0% 41.03Ki ± 0% -30.30% (p=0.000 n=10+9) Put/size=1048576,thread=100/fstree-12 299.8Ki ± ∞ ¹ geomean 36.71Ki 16.60Ki -31.17% ² ¹ need >= 6 samples for confidence interval at level 0.95 ² benchmark set differs from baseline; geomeans may not be comparable │ fs.nosync-generic │ fs.nosync-linux │ │ allocs/op │ allocs/op vs base │ Put/size=1024,thread=1/fstree-12 28.00 ± 0% 22.00 ± 0% -21.43% (p=0.000 n=10) Put/size=1024,thread=20/fstree-12 530.0 ± 0% 407.5 ± 1% -23.11% (p=0.000 n=10) Put/size=1024,thread=100/fstree-12 2.567k ± 0% 1.956k ± 9% -23.77% (p=0.000 n=10) Put/size=1048576,thread=1/fstree-12 30.00 ± 0% 24.00 ± 0% -20.00% (p=0.000 n=10) Put/size=1048576,thread=20/fstree-12 553.5 ± 0% 434.0 ± 0% -21.59% (n=10+9) Put/size=1048576,thread=100/fstree-12 2.803k ± ∞ ¹ geomean 347.9 178.8 -21.99% ² ¹ need >= 6 samples for confidence interval at level 0.95 ² benchmark set differs from baseline; geomeans may not be comparable ``` Signed-off-by: Roman Khimov <roman@nspcc.ru> Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2024-02-09 16:12:11 +00:00
Evgenii Stratonikov	ff488b53a1	[#970] fstree: Move write functions to a separate file ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2024-02-09 16:12:11 +00:00
Evgenii Stratonikov	9a622a750d	[#970] fstree: Move temporary path handling in a separate function ... Allow to easier test different implementations. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2024-02-09 16:12:11 +00:00
Anton Nikiforov	d19ade23c8	[#959] node: Set mode to shard's components when open it ... Avoid opening database for `metabase` and `cache` in `Degraded` mode. Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>	2024-02-09 14:04:01 +00:00
Dmitrii Stepanov	60527abb65	[#947] docs: Extend evacuation docs ... Add pilorama evacuation description and examples. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-02-09 11:33:15 +03:00