TrueCloudLab/neo-go
8
0
Fork 0
mirror of https://github.com/nspcc-dev/neo-go.git synced 2024-12-22 09:19:08 +00:00
Code Issues Projects Releases Packages Wiki Activity
8121 commits 38 branches 139 tags 46 MiB
Commit graph

893 commits

Author SHA1 Message Date
Anna Shaleva
6d20772714
Merge pull request #3763 from nspcc-dev/index-duplicates 
cli: add warning for duplicated index files in `upload-bin`
 2024-12-17 16:27:50 +03:00
Ekaterina Pavlova
c669ae343b cli: add warning for duplicated index files in upload-bin 
As we are not afraid of duplicates this is not a critical error anymore.
BlockFetcher will take the first returned by search.

Close #3762

Signed-off-by: Ekaterina Pavlova <ekt@morphbits.io>
 2024-12-17 16:19:07 +03:00
Anna Shaleva
0bb6c6e6bd cli: take care of upload-bin defaults 
Signed-off-by: Anna Shaleva <shaleva.ann@nspcc.ru>
 2024-12-16 15:54:15 +03:00
Ekaterina Pavlova
39cec7d0d6 cli: refactor neofs-sdk-go related operations in upload-bin 
Signed-off-by: Ekaterina Pavlova <ekt@morphbits.io>
 2024-12-13 19:19:15 +03:00
Ekaterina Pavlova
9e3f75e977 *: update neofs-sdk-go 
Update to the version without pool panics
https://github.com/nspcc-dev/neofs-sdk-go/pull/643.

Signed-off-by: Ekaterina Pavlova <ekt@morphbits.io>
 2024-12-13 17:07:52 +03:00
Ekaterina Pavlova
be4fc98041 cli: refactor upload-bin handler to reuse SDK built-in functionality 
Close #3714

Signed-off-by: Ekaterina Pavlova <ekt@morphbits.io>
 2024-12-13 16:36:37 +03:00
Ekaterina Pavlova
247ee831e5 cli: change block timestamp attribute of objects in upload-bin 
Close #3654

Signed-off-by: Ekaterina Pavlova <ekt@morphbits.io>
 2024-12-13 16:35:14 +03:00
Ekaterina Pavlova
46cbfab264 cli: add more debug info about retry to the upload-bin 
Signed-off-by: Ekaterina Pavlova <ekt@morphbits.io>
 2024-12-13 13:52:33 +03:00
Ekaterina Pavlova
65bdc82da8 *: move constant and NeoFS related code into separate package 
Signed-off-by: Ekaterina Pavlova <ekt@morphbits.io>
 2024-12-13 11:43:19 +03:00
Ekaterina Pavlova
62615f8c7e cli: update upload-bin to create index files during block uploading 
Close #3655
Close #3652

Signed-off-by: Ekaterina Pavlova <ekt@morphbits.io>
 2024-12-12 12:08:49 +03:00
Ekaterina Pavlova
ebb71497a5 *: move metric neogo_version out of pkg/network 
Close #3682

Signed-off-by: Ekaterina Pavlova <ekt@morphbits.io>
 2024-11-29 16:58:11 +03:00
Anna Shaleva
7ba295ca99
Merge pull request #3708 from nspcc-dev/rpcwrapper 
cli: add check config field for `generate-rpcwrapper`
 2024-11-29 14:34:02 +03:00
Ekaterina Pavlova
870421e950 cli: add check config field for generate-rpcwrapper 
Close #3646

Signed-off-by: Ekaterina Pavlova <ekt@morphbits.io>
 2024-11-29 13:53:40 +03:00
Ekaterina Pavlova
04516e7d26 neofs: add pool support for NeoFS operations 
Signed-off-by: Ekaterina Pavlova <ekt@morphbits.io>
 2024-11-28 16:13:36 +03:00
Ekaterina Pavlova
43609dd984 *: migrate to SEARCH with strict equality comparator 
Close #3670

Signed-off-by: Ekaterina Pavlova <ekt@morphbits.io>
 2024-11-25 13:19:35 +03:00
Ekaterina Pavlova
616c805ac5 cli: refactor index file search in upload-bin command 
Align index file searching logic with block search.

Signed-off-by: Ekaterina Pavlova <ekt@morphbits.io>
 2024-11-22 17:45:35 +03:00
Ekaterina Pavlova
5dab154582 cli: split searchBatchSize usage in upload-bin command 
Split searchBatchSize into constant for searching and uploading.

Signed-off-by: Ekaterina Pavlova <ekt@morphbits.io>
 2024-11-22 17:45:35 +03:00
Ekaterina Pavlova
9fd375d56d cli: fix nil error of fetchLatestMissingBlockIndex in upload-bin 
Close #3653

Signed-off-by: Ekaterina Pavlova <ekt@morphbits.io>
 2024-11-22 17:45:35 +03:00
Ekaterina Pavlova
b2bd8e4a0a manifest: support NEP-24 
Close #3451

Signed-off-by: Ekaterina Pavlova <ekt@morphbits.io>
 2024-11-20 14:45:28 +03:00
Ekaterina Pavlova
5f6284de05 cli: fix skip-blocks-uploading mode of upload-bin command 
If uploadBlocks succeeded uploadIndexFiles should work based on the
currentBlockHeight.

Signed-off-by: Ekaterina Pavlova <ekt@morphbits.io>
 2024-11-15 17:28:24 +03:00
Ekaterina Pavlova
58ed448f8d cli: add debug mode to upload-bin command 
Signed-off-by: Ekaterina Pavlova <ekt@morphbits.io>
 2024-11-15 17:18:44 +03:00
Anna Shaleva
b97d0b2326
Merge pull request #3684 from nspcc-dev/last-block 
cli: fix fetchLatestMissingBlockIndex and uploadIndexFiles in `upload-bin` command
 2024-11-15 15:12:30 +03:00
Ekaterina Pavlova
375b095f78 cli: add details to the logs in upload-bin 
Signed-off-by: Ekaterina Pavlova <ekt@morphbits.io>
 2024-11-15 14:34:36 +03:00
Ekaterina Pavlova
380d112599 cli: fix stopping of uploadIndexFiles in upload-bin 
The expected count of index files should be counted based on the already
uploaded object number.

Close #3669

Signed-off-by: Ekaterina Pavlova <ekt@morphbits.io>
 2024-11-15 14:34:36 +03:00
Ekaterina Pavlova
6e863e9a06 cli: fix fetchLatestMissingBlockIndex in upload-bin command 
Close #3667

Signed-off-by: Ekaterina Pavlova <ekt@morphbits.io>
 2024-11-15 14:34:10 +03:00
Roman Khimov
895a0ae624
Merge pull request #3677 from ixje/expose-slots 
vm: make slots public
 2024-11-15 13:16:48 +03:00
ixje
7ec0c1155c vm: expose Context slots, hide Set/ClearRefs on Slot, deprecate Dump*Slot methods 
Signed-off-by: ixje <erik@coz.io>
 2024-11-15 11:05:42 +01:00
ixje
d8ea4103c7 cli: add delete and list break point commands 
Close #3673

Signed-off-by: ixje <erik@coz.io>
 2024-11-14 09:52:25 +01:00
Ekaterina Pavlova
29e2d712f4 cli: add flag max-retries for upload-bin 
Signed-off-by: Ekaterina Pavlova <ekt@morphbits.io>
 2024-11-06 11:18:41 +03:00
Ekaterina Pavlova
18ed3bb3e8 cli: fix missing errors on closure of writer in upload-bin 
Ref #3658

Signed-off-by: Ekaterina Pavlova <ekt@morphbits.io>
 2024-11-06 11:16:46 +03:00
Anna Shaleva
8c4d9432d7
Merge pull request #3662 from nspcc-dev/improve-uploadbin-log 
cli: extend logs of index file construction for `upload-bin`
 2024-11-06 11:13:44 +03:00
Alexey Savchuk
df9247c00b services/rpcsrv: Return a new server by pointer 
Before, a new server was returned by value which could cause
a panic `unlock of unlocked mutex` on SIGHUP handling. It's
because the new server overwrites a locked mutex of the already
existing server.

oct‚ 22 13:51:15 node1 neo-go[1183338]: fatal error: sync: Unlock of unlocked RWMutex
oct‚ 22 13:51:15 node1 neo-go[1183338]: goroutine 538 [running]:
oct‚ 22 13:51:15 node1 neo-go[1183338]: sync.fatal({0xf83d64?, 0xc001085880?})
oct‚ 22 13:51:15 node1 neo-go[1183338]:         runtime/panic.go:1007 +0x18
oct‚ 22 13:51:15 node1 neo-go[1183338]: sync.(*RWMutex).Unlock(0xc00019a4c8)
oct‚ 22 13:51:15 node1 neo-go[1183338]:         sync/rwmutex.go:208 +0x45
oct‚ 22 13:51:15 node1 neo-go[1183338]: github.com/nspcc-dev/neo-go/pkg/services/rpcsrv.(*Server).dropSubscriber(0xc00019a2c8, 0xc000a77740)
oct‚ 22 13:51:15 node1 neo-go[1183338]:         github.com/nspcc-dev/neo-go/pkg/services/rpcsrv/server.go:825 +0xce
oct‚ 22 13:51:15 node1 neo-go[1183338]: github.com/nspcc-dev/neo-go/pkg/services/rpcsrv.(*Server).handleWsReads(0xc00019a2c8, 0xc0034478c0, 0xc000af5f80, 0xc000a7
7740)
oct‚ 22 13:51:15 node1 neo-go[1183338]:         github.com/nspcc-dev/neo-go/pkg/services/rpcsrv/server.go:810 +0x266
oct‚ 22 13:51:15 node1 neo-go[1183338]: github.com/nspcc-dev/neo-go/pkg/services/rpcsrv.(*Server).handleHTTPRequest(0xc00019a2c8, {0x11c3900, 0xc003437dc0}, 0xc00
31945a0)
oct‚ 22 13:51:15 node1 neo-go[1183338]:         github.com/nspcc-dev/neo-go/pkg/services/rpcsrv/server.go:582 +0x54a
oct‚ 22 13:51:15 node1 neo-go[1183338]: net/http.HandlerFunc.ServeHTTP(0x471779?, {0x11c3900?, 0xc003437dc0?}, 0xc000943b68?)
oct‚ 22 13:51:15 node1 neo-go[1183338]:         net/http/server.go:2171 +0x29
oct‚ 22 13:51:15 node1 neo-go[1183338]: net/http.serverHandler.ServeHTTP({0xc000a77680?}, {0x11c3900?, 0xc003437dc0?}, 0x6?)
oct‚ 22 13:51:15 node1 neo-go[1183338]:         net/http/server.go:3142 +0x8e
oct‚ 22 13:51:15 node1 neo-go[1183338]: net/http.(*conn).serve(0xc0032030e0, {0x11c5220, 0xc000a76960})
oct‚ 22 13:51:15 node1 neo-go[1183338]:         net/http/server.go:2044 +0x5e8
oct‚ 22 13:51:15 node1 neo-go[1183338]: created by net/http.(*Server).Serve in goroutine 534
oct‚ 22 13:51:15 node1 neo-go[1183338]:         net/http/server.go:3290 +0x4b4

Signed-off-by: Alexey Savchuk <alexey.a.savchuk@yandex.com>
 2024-11-05 15:05:25 +03:00
Roman Khimov
ddaf9c01ab cli: make nep1X balance commands work without a wallet, fix #3275 
Checking the balance should be easy without any wallets involved, the data
is available anyway.

Tests are extended a bit as well.

Adding the command to "query nep1X" is not so trivial, so let's have it this
way for now.

Signed-off-by: Roman Khimov <roman@nspcc.ru>
 2024-11-05 13:12:13 +03:00
Anna Shaleva
a85f3ce83f cli: rename uploader.go to upload_bin.go 
Follow the style of other files in the `util` package and existing
`dump_bin.go`.

Signed-off-by: Anna Shaleva <shaleva.ann@nspcc.ru>
 2024-11-05 12:43:20 +03:00
Anna Shaleva
22cb7feb1d cli: fix format 
Signed-off-by: Anna Shaleva <shaleva.ann@nspcc.ru>
 2024-11-05 11:25:08 +03:00
Anna Shaleva
82b508e8f3 cli: extend logs of index file construction for upload-bin 
These logs are needed to track the progress of uploaders.

Signed-off-by: Anna Shaleva <shaleva.ann@nspcc.ru>
 2024-11-05 11:24:57 +03:00
Ekaterina Pavlova
c955c1e4ea cli: use GET instead of HEAD in upload bin 
Will be reverted when nspcc-dev/neofs-node#2988 is fixed. Currently HEAD
request can return incorrect attributes, which can lead to incorrect
index files.

Signed-off-by: Ekaterina Pavlova <ekt@morphbits.io>
 2024-11-02 12:04:02 +03:00
Ekaterina Pavlova
35d12779d6 cli: adjust index file creation in upload-bin 
In case of incomplete search result it will try to find each missed oid
and process it. In case of duplicates the first found will be in index
file.

Close #3647

Signed-off-by: Ekaterina Pavlova <ekt@morphbits.io>
 2024-11-01 19:51:01 +03:00
Ekaterina Pavlova
365bbe08ed cli: refactor errors and comments in upload-bin 
Signed-off-by: Ekaterina Pavlova <ekt@morphbits.io>
 2024-11-01 14:58:48 +03:00
Ekaterina Pavlova
6a93f70728 cli: add flag for skipping block uploading in upload-bin 
Signed-off-by: Ekaterina Pavlova <ekt@morphbits.io>
 2024-10-31 18:39:40 +03:00
Ekaterina Pavlova
7b3eeb9061 cli: fix usage of upload-bin 
Signed-off-by: Ekaterina Pavlova <ekt@morphbits.io>
 2024-10-24 12:19:12 +03:00
Ekaterina Pavlova
36e855609d cli: add workers and searchers flags to upload-bin command 
Number of workers to fetch, upload and search blocks concurrently is now
configurable.

Close #3641

Signed-off-by: Ekaterina Pavlova <ekt@morphbits.io>
 2024-10-24 12:19:10 +03:00
Ekaterina Pavlova
a5e9ab6979 cli: verify index file construction in upload-bin command 
Verify that there are no empty OIDs in the constructed index file, as
empty payloads can result in improperly set attributes, leading to empty
OIDs being added to the index file.

Close #3628

Signed-off-by: Ekaterina Pavlova <ekt@morphbits.io>
 2024-10-23 15:30:12 +03:00
Ekaterina Pavlova
42c8e40eaa cli: add retry to all requests to NeoFS in upload-bin 
Add retry to `NetworkInfo` and `uploadObj`.

Signed-off-by: Ekaterina Pavlova <ekt@morphbits.io>
 2024-10-23 15:30:12 +03:00
Ekaterina Pavlova
8b43c33e44 cli: extend object attribute parsing error in upload-bin 
Signed-off-by: Ekaterina Pavlova <ekt@morphbits.io>
 2024-10-23 15:30:12 +03:00
Ekaterina Pavlova
5b793bcf1b cli: fix process termination in upload-bin command 
Add a return statement to properly handle errors, ensuring the process
terminates as expected.

Signed-off-by: Ekaterina Pavlova <ekt@morphbits.io>
 2024-10-23 15:30:12 +03:00
Ekaterina Pavlova
e83b3e4839 cli: fix error handling in upload-bin command 
Fix shared global error reuse.

Close #3634

Signed-off-by: Ekaterina Pavlova <ekt@morphbits.io>
 2024-10-23 15:30:12 +03:00
Ekaterina Pavlova
c47d4e6c5b BlockFetcher/BlockUploader: fix and add additional attributes 
Capitalize block objects and index file objects attributes.
Add network magic attribute to container.

Close #3631

Signed-off-by: Ekaterina Pavlova <ekt@morphbits.io>
 2024-10-23 13:52:06 +03:00
Ekaterina Pavlova
e00d7fec7f cli: fix empty block uploading in util upload-bin 
In case of uploading retry already drained bin buffer is returned in
`bw.Bytes`, which leads to empty block uploading.

Close #3630

Signed-off-by: Ekaterina Pavlova <ekt@morphbits.io>
 2024-10-21 22:44:06 +03:00
Ekaterina Pavlova
41873e35c6 cli: increase NeoFS pool deadlines for upload-bin command 
`defaultDialTimeout` and `defaultStreamTimeout` increased because of
`code = DeadlineExceeded desc = context deadline exceeded`

Close #3620

Signed-off-by: Ekaterina Pavlova <ekt@morphbits.io>
 2024-10-18 19:59:27 +03:00