Commit graph

18 commits

Author SHA1 Message Date
Ekaterina Pavlova
f5ea79d649 services: refactor NeoFS client naming
There is just NeoFS client/conn.

Signed-off-by: Ekaterina Pavlova <ekt@morphbits.io>
2024-12-16 11:12:46 +03:00
Ekaterina Pavlova
9e3f75e977 *: update neofs-sdk-go
Update to the version without pool panics
https://github.com/nspcc-dev/neofs-sdk-go/pull/643.

Signed-off-by: Ekaterina Pavlova <ekt@morphbits.io>
2024-12-13 17:07:52 +03:00
Ekaterina Pavlova
65bdc82da8 *: move constant and NeoFS related code into separate package
Signed-off-by: Ekaterina Pavlova <ekt@morphbits.io>
2024-12-13 11:43:19 +03:00
Ekaterina Pavlova
c79ffa967f services: add default values for NeoFSBlockFetcher configuration
The minimum sufficient configuration is Addresses and ContainerID,
example:
```
  NeoFSBlockFetcher:
    Enabled: true
    Addresses:
      - st1.storage.fs.neo.org:8080
      - st2.storage.fs.neo.org:8080
      - st3.storage.fs.neo.org:8080
      - st4.storage.fs.neo.org:8080
    ContainerID: "87JRc7vyWcjW8uS32LMoLTAj4ckCzFZWfKbacjU3sAob"
```

Close #3718

Signed-off-by: Ekaterina Pavlova <ekt@morphbits.io>
2024-12-13 11:39:22 +03:00
Ekaterina Pavlova
3010324c4f services: fix defer function in retry of NeoFSBlockFetcher
Drain the channel only if it has pending events. This avoids trying to
read from an already-empty channel.

Close #3734

Signed-off-by: Ekaterina Pavlova <ekt@morphbits.io>
2024-12-08 00:04:14 +03:00
Anna Shaleva
2244d2ad75 blockfetcher: decrease default pool deal timeout
Dealing is fast, we don't need 10 minutes to check that connection can
(or can't) be established. Improve documentation along the way.

Close #3721.

Signed-off-by: Anna Shaleva <shaleva.ann@nspcc.ru>
2024-12-04 16:34:09 +03:00
Ekaterina Pavlova
718ff9aa0a blockfetcher: add check on IndexFileSize of index files
Signed-off-by: Ekaterina Pavlova <ekt@morphbits.io>
2024-11-29 13:37:50 +03:00
Ekaterina Pavlova
14c980a685 network: refactor return value from NeoFSBlockFetcher service
Signed-off-by: Ekaterina Pavlova <ekt@morphbits.io>
2024-11-29 13:37:50 +03:00
Ekaterina Pavlova
119ca27994 blockfetcher: fix block enqueue logic
Previously, the `blockQueuer` routine, which enqueues blocks into
`bQueue`, could be blocked on enqueing newer blocks if older blocks
downloading is delayed by NeoFS.

The `blocksCh` channel, acting as a queue ordered by download speed,
conflicted with the BQueue requirement for strict sequential enqueuing
(expecting an exact range of blocks), resulting in a deadlock that
stalled the process.

Before with default config settings:
```
2024-11-27T17:12:19.348+0300	INFO	persisted to disk	{"blocks":
 0, "keys": 116, "headerHeight": 0, "blockHeight": 0, "took": "15
 .509083ms"}
2024-11-27T17:19:39.574+0300	INFO	persisted to disk	{"blocks":
 16, "keys": 11107, "headerHeight": 216768, "blockHeight": 216768,
 "took": "62.762041ms"}
```
Average block persistence speed: 492.40 block/s
Average blocks number for each persist log: 584.28

After:

```
2024-11-27T17:29:03.362+0300	INFO	persisted to disk	{"blocks":
 0, "keys": 116, "headerHeight": 0, "blockHeight": 0, "took": "19
 .485084ms"}
2024-11-27T17:34:58.527+0300	INFO	persisted to disk	{"blocks":
 16, "keys": 11109, "headerHeight": 216770, "blockHeight": 216769,
 "took": "52.43925ms"}
```
Average block persistence speed: 610.33 block/s
Average blocks number for each persist log: 752.61

Close #3699

Signed-off-by: Ekaterina Pavlova <ekt@morphbits.io>
2024-11-29 13:37:50 +03:00
Ekaterina Pavlova
df05cd2858 blockfetcher: change shutdown logic
Move the `isActive` check earlier to enable faster shutdown and prevent
the service from hanging.

Signed-off-by: Ekaterina Pavlova <ekt@morphbits.io>
2024-11-29 13:37:50 +03:00
Ekaterina Pavlova
8216f538c3 blockfetcher: check container and network Magic
Close #3644

Signed-off-by: Ekaterina Pavlova <ekt@morphbits.io>
2024-11-29 13:37:11 +03:00
Ekaterina Pavlova
92ff8409a9 blockfetcher: add retry for GET and SEARCH operations
Close #3564

Signed-off-by: Ekaterina Pavlova <ekt@morphbits.io>
2024-11-29 13:34:40 +03:00
Ekaterina Pavlova
9fa07d8d6d blockfetcher: fix invalid wallet test
Signed-off-by: Ekaterina Pavlova <ekt@morphbits.io>
2024-11-28 16:14:25 +03:00
Ekaterina Pavlova
c321eed8ee blockfetcher: use pool for GET and SEARCH operations
Use NeoFS storage nodes pool during blocks fetching to spread the load.

Close #3568

Signed-off-by: Ekaterina Pavlova <ekt@morphbits.io>
2024-11-28 16:14:25 +03:00
Ekaterina Pavlova
43609dd984 *: migrate to SEARCH with strict equality comparator
Close #3670

Signed-off-by: Ekaterina Pavlova <ekt@morphbits.io>
2024-11-25 13:19:35 +03:00
Ekaterina Pavlova
8a154d9189 blockfetcher: add more details to errors
Close #3629

Signed-off-by: Ekaterina Pavlova <ekt@morphbits.io>
2024-10-22 14:11:27 +03:00
Ekaterina Pavlova
59fab5d654 blockfetcher: adjust index file numeration from 0
Signed-off-by: Ekaterina Pavlova <ekt@morphbits.io>
2024-10-16 19:23:56 +03:00
Ekaterina Pavlova
0b31a29f39 services: add new service for fetching blocks from NeoFS
Close #3496

Co-authored-by: Anna Shaleva <shaleva.ann@nspcc.ru>
Signed-off-by: Ekaterina Pavlova <ekt@morphbits.io>
2024-09-11 08:50:31 +04:00