Ekaterina Lebedeva
8a81af5a3b
[ #653 ] Add context parameter to Open functions
...
DCO action / DCO (pull_request) Successful in 1m38s
Details
Build / Build Components (1.20) (pull_request) Successful in 4m22s
Details
Build / Build Components (1.21) (pull_request) Successful in 4m25s
Details
Vulncheck / Vulncheck (pull_request) Successful in 4m56s
Details
Tests and linters / Lint (pull_request) Successful in 6m1s
Details
Tests and linters / Tests (1.20) (pull_request) Successful in 7m43s
Details
Tests and linters / Staticcheck (pull_request) Successful in 8m1s
Details
Tests and linters / Tests (1.21) (pull_request) Successful in 8m14s
Details
Tests and linters / Tests with -race (pull_request) Successful in 8m32s
Details
Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2023-09-07 18:03:29 +03:00
Alexander Chuprov
4d9a6c07fb
[ #618 ] core: Replace fmt.Sprintf with strconv.FormatUint
...
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2023-08-21 07:14:50 +00:00
Evgenii Stratonikov
a8de37c8a2
[ #607 ] *: Remove redundant if on error returns
...
Semantic patch:
```
@@
@@
-if err != nil { return err }
-return nil
+return err
```
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-17 16:50:02 +00:00
Evgenii Stratonikov
5335e7089e
[ #615 ] pilorama: Speedup TestForest_ApplyRandom()
...
Some of our pilorama tests fail on CI.
The reasons are not obvious, but one possible improvement
is using `WithNoSync` option for these. It should have much effect,
because we are writing on the tmpfs, but doesn't hurt anyway.
If I replace `t.TempDir()` with a local directory, test execution time
goes down from 5s (sync) to 0.4s (nosync), which is the same time as
with `t.TempDir()`. Maybe we have some strange CI configuration.
```
panic: test timed out after 10m0s
running tests:
TestForest_ApplyRandom (8m22s)
TestForest_ApplyRandom/bbolt (8m21s)
...
goroutine 170 [syscall]:
syscall.Syscall(0xc000100000?, 0xc00047b758?, 0x6aff9a?, 0xc00041c1b0?)
/opt/hostedtoolcache/go/1.20.7/x64/src/syscall/syscall_linux.go:69 +0x27
syscall.Fdatasync(0x9e35c0?)
/opt/hostedtoolcache/go/1.20.7/x64/src/syscall/zsyscall_linux_amd64.go:418 +0x2a
go.etcd.io/bbolt.fdatasync(0xc000189000?)
```
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-16 19:09:08 +00:00
Alejandro Lopez
ae8be495c8
[#xx] Avoid manual management of files in tests
...
Vulncheck / Vulncheck (pull_request) Successful in 2m36s
Details
Build / Build Components (1.21) (pull_request) Successful in 3m33s
Details
DCO action / DCO (pull_request) Successful in 3m33s
Details
Tests and linters / Staticcheck (pull_request) Successful in 4m15s
Details
Tests and linters / Lint (pull_request) Successful in 5m31s
Details
Build / Build Components (1.20) (pull_request) Successful in 7m2s
Details
Tests and linters / Tests with -race (pull_request) Failing after 8m51s
Details
Tests and linters / Tests (1.20) (pull_request) Successful in 9m55s
Details
Tests and linters / Tests (1.21) (pull_request) Successful in 10m6s
Details
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-08-14 14:01:39 +03:00
Evgenii Stratonikov
d641cba2fc
[ #587 ] Do not use math/rand.Read
...
Build / Build Components (1.20) (pull_request) Failing after 3s
Details
Build / Build Components (1.21) (pull_request) Failing after 3s
Details
Tests and linters / Lint (pull_request) Failing after 3s
Details
Tests and linters / Tests (1.20) (pull_request) Failing after 3s
Details
Tests and linters / Tests (1.21) (pull_request) Failing after 3s
Details
Tests and linters / Tests with -race (pull_request) Failing after 5s
Details
Tests and linters / Staticcheck (pull_request) Failing after 8s
Details
Vulncheck / Vulncheck (pull_request) Failing after 2s
Details
Fix staticcheck warnings after go1.20 update.
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-09 16:02:44 +03:00
Dmitrii Stepanov
af82c2865e
[ #335 ] treesvc: Fix inmemory unit tests and nil meta items
...
Bolt forest saves empty slice of items. Now inmemory forest
does it the same way.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-20 10:14:10 +03:00
Dmitrii Stepanov
b4e72a2dfd
[ #335 ] treesvc: Sort nodes by Filename in GetSubTree
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-20 10:14:10 +03:00
Evgenii Stratonikov
4bf345225c
[ #447 ] pilorama: Use named constant for the key size
...
ci/woodpecker/push/pre-commit Pipeline was successful
Details
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-26 07:42:05 +00:00
Evgenii Stratonikov
b4ce0b0412
[ #447 ] pilorama: Do not undo log for create ops
...
```
goos: linux
goarch: amd64
cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz
│ old │ new │
│ sec/op │ sec/op vs base │
Create-8 36.48µ ± 11% 30.34µ ± 14% -16.84% (p=0.000 n=10)
│ old │ new │
│ B/op │ B/op vs base │
Create-8 43.01Ki ± 4% 37.78Ki ± 5% -12.15% (p=0.000 n=10)
│ old │ new │
│ allocs/op │ allocs/op vs base │
Create-8 166.0 ± 3% 146.0 ± 3% -12.05% (p=0.000 n=10)
```
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-26 07:42:05 +00:00
Evgenii Stratonikov
dd3874eff1
[ #447 ] pilorama: Add benchmark for create ops
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-26 07:42:05 +00:00
Dmitrii Stepanov
d8ecc69d00
[ #373 ] local storage: Pass parent ID
...
This is required to add shard ID as metric label.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-21 15:13:26 +03:00
Dmitrii Stepanov
d5aaec1107
[ #373 ] pilorama: Add metrics
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-21 15:13:26 +03:00
Evgenii Stratonikov
fe01781811
[ #446 ] los: Wrap SSD errors in a separate type
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-16 14:53:31 +03:00
Dmitrii Stepanov
2541d319de
[ #266 ] pilorama: Allow to get current tree height
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-13 10:00:45 +00:00
Dmitrii Stepanov
74578052f9
[ #412 ] node: Replace tracing package
...
Use observability module.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-01 13:23:11 +00:00
Evgenii Stratonikov
f2e5dead7e
[ #398 ] pilorama: Disallow applying same operations
...
ci/woodpecker/push/pre-commit Pipeline was successful
Details
1. In redo() we save the old state.
2. If we do redo() for the same operation twice, the old state will be
overritten with the new one.
3. This in turn affects undo() and subsequent isAncestor() check.
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-26 12:14:29 +00:00
Alejandro Lopez
973af12854
[ #327 ] tests: replace os.MkdirTemp with t.TempDir
...
ci/woodpecker/push/pre-commit Pipeline was successful
Details
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-05-05 11:24:51 +00:00
Evgenii Stratonikov
58f1ba4b51
[ #288 ] pilorama: Add missing operation in log
...
ci/woodpecker/push/pre-commit Pipeline was successful
Details
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-28 07:56:20 +00:00
Evgenii Stratonikov
daa26f6e9b
[ #288 ] pilorama/test: Check operation order for TreeGetByPath()
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-28 07:56:20 +00:00
Evgenii Stratonikov
291f9e809a
[ #288 ] pilorama: Remove getMeta() wrapper
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-28 07:56:20 +00:00
Evgenii Stratonikov
0045f1bcd4
[ #288 ] pilorama: Use more descriptive names for memory tree
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-28 07:56:20 +00:00
Evgenii Stratonikov
f856ad7480
[ #288 ] pilorama: Remove `childMap` from memory forest
...
Memory forest is here to check the correctness of boltdb optimized
implementation. Let's keep it simple.
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-28 07:56:20 +00:00
Dmitrii Stepanov
6121b541b5
[ #242 ] treesvc: Add tracing spans
...
ci/woodpecker/push/pre-commit Pipeline was successful
Details
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-14 10:25:53 +00:00
Dmitrii Stepanov
f32f61df87
[ #203 ] pilorama: Refactor tests
...
Do not pass 0 as channel capacity.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-06 16:33:36 +03:00
Dmitrii Stepanov
bab11492ad
[ #203 ] node: Resolve never used errors
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-06 16:33:36 +03:00
Dmitrii Stepanov
9e2df4b7c7
[ #203 ] node: Fix double imports
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-06 16:33:36 +03:00
Alejandro Lopez
341fe1688f
[ #139 ] test: Add test storage implementation
...
ci/woodpecker/push/pre-commit Pipeline was successful
Details
This aims to reduce the usage of chmod hackery to induce or simulate
OS-related failures.
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-03-29 14:28:49 +00:00
Evgenii Stratonikov
47e8c5bf23
[ #156 ] pilorama: Remove CIDDescriptor from TreeApply()
...
Initially it was there to check whether an update is being initiated by
a proper node. It is now obsolete for 2 reasons:
1. Background synchronization fetches all operations from a single node.
2. There are a lot more problems with trust in the tree service, it is
only used in controlled environments.
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-22 07:14:18 +00:00
Evgenii Stratonikov
3e6fd4c611
[ #82 ] pilorama: Allow to store last sync height
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-13 11:25:44 +00:00
Alexey Vanin
20de74a505
Rename package name
...
Due to source code relocation from GitHub.
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2023-03-07 16:38:26 +03:00
Evgenii Stratonikov
58367e4df6
[ #2232 ] pilorama: Merge in-queue batches
...
To achieve high performance we must choose proper values for both
batch size and delay. For user operations we want to set low delay.
However it would prevent tree synchronization operations to form big
enough batches. For these operations, batching gives the most benefit
not only in terms of on-CPU execution cost, but also by speeding up
transaction persist (`fsync`).
In this commit we try merging batches that are already
_triggered_, but not yet _started to execute_. This way we can still
query batches for execution after the provided delay while also allowing
multiple formed batches to execute faster.
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-02-20 13:53:27 +03:00
Pavel Karpy
73bc1b0b68
[ #38 ] node: Fix linter warnings
...
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-02-06 17:27:54 +03:00
Evgenii Stratonikov
d65a95a2c6
[ #28 ] pilorama: Remove `LogMove` struct
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-25 15:31:47 +03:00
Evgenii Stratonikov
25d5995cef
[ #2210 ] pilorama: Allocate bucket name outside of batches
...
1. Reduce allocations inside transactions.
2. Do not encode container ID to string: it allocates a lot and takes more
space.
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-25 15:31:47 +03:00
Evgenii Stratonikov
165a600624
[ #2210 ] pilorama: Reduce the amount of keys per node
...
Under high load we are limited by the _amount_ of keys we need to update
in a single transaction. In this commit we try storing all state
with a single key.
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-25 15:31:47 +03:00
Evgenii Stratonikov
ac81c70c09
[ #1621 ] pilorama: Batch related operations
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-25 15:31:47 +03:00
Evgenii Stratonikov
cedbd380f2
[ #2197 ] pilorama: Close database in degraded mode
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-25 15:31:47 +03:00
Evgenii Stratonikov
b0ad1b9ed2
[ #2193 ] pilorama: Use `do` in `TreeMove`
...
It should be similar to a `TreeAddByPath`. `applyOperation` is used for
`Apply` when the operation can be inserted in the middle of a log.
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-25 15:31:47 +03:00
Evgenii Stratonikov
b4e90cdf51
[ #2165 ] pilorama: Optimize `TreeApply` when used for synchronization
...
Because synchronization _most likely_ will have apply already existing
operations, it is much faster to check their presence in a read
transaction. However, always doing this will degrade the perfomance
for normal `Apply`. And, let's be honest, it is already not good.
Thus we add a separate parameter which specifies whether this logic is
enabled.
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-30 11:07:35 +03:00
Evgenii Stratonikov
e1c3bdbfa6
[ #1621 ] pilorama: Remove `Timestamp` field from `nodeInfo`
...
It is already present in `Meta`.
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-12-30 11:07:35 +03:00
Evgenii Stratonikov
1044adbe94
[ #1621 ] pilorama: Improve memory allocation
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-12-30 11:07:35 +03:00
Evgenii Stratonikov
2539d466a6
[ #1621 ] pilorama: Seek after cursor invalidation
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-12-30 11:07:35 +03:00
Evgenii Stratonikov
e9ba8931f8
[ #1621 ] pilorama: Simplify bucket creation
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-12-30 11:07:35 +03:00
Evgenii Stratonikov
fe7ddfdc6a
[ #1621 ] pilorama: Compare memory forests properly
...
Node children are not sorted and could occur in any order.
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-12-30 11:07:35 +03:00
Evgenii Stratonikov
e5c304536b
[ #2161 ] pilorama: Do not apply already existing operations
...
Speeds up synchronization a bit.
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-30 11:07:35 +03:00
Pavel Karpy
923f84722a
Move to frostfs-node
...
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2022-12-28 15:04:29 +03:00
Evgenii Stratonikov
7335a52f29
[ #1732 ] pilorama: Improve logical error handling
...
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-30 16:58:52 +03:00
Anton Nikiforov
9a20498f34
[ #1940 ] Removing all trees by container ID if tree ID is empty in `pilorama.Forest.TreeDrop`
...
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2022-11-19 11:01:04 +03:00
Evgenii Stratonikov
a3e7365cbd
[ #1732 ] pilorama: Fill parent mark correctly
...
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-19 11:01:04 +03:00