TrueCloudLab/neo-go
8
0
Fork 0
mirror of https://github.com/nspcc-dev/neo-go.git synced 2024-12-22 19:19:09 +00:00
Code Issues Projects Releases Packages Wiki Activity
8121 commits 38 branches 139 tags 46 MiB
Commit graph

65 commits

Author SHA1 Message Date
Anna Shaleva
6d20772714
Merge pull request #3763 from nspcc-dev/index-duplicates 
cli: add warning for duplicated index files in `upload-bin`
 2024-12-17 16:27:50 +03:00
Ekaterina Pavlova
c669ae343b cli: add warning for duplicated index files in upload-bin 
As we are not afraid of duplicates this is not a critical error anymore.
BlockFetcher will take the first returned by search.

Close #3762

Signed-off-by: Ekaterina Pavlova <ekt@morphbits.io>
 2024-12-17 16:19:07 +03:00
Anna Shaleva
0bb6c6e6bd cli: take care of upload-bin defaults 
Signed-off-by: Anna Shaleva <shaleva.ann@nspcc.ru>
 2024-12-16 15:54:15 +03:00
Ekaterina Pavlova
39cec7d0d6 cli: refactor neofs-sdk-go related operations in upload-bin 
Signed-off-by: Ekaterina Pavlova <ekt@morphbits.io>
 2024-12-13 19:19:15 +03:00
Ekaterina Pavlova
9e3f75e977 *: update neofs-sdk-go 
Update to the version without pool panics
https://github.com/nspcc-dev/neofs-sdk-go/pull/643.

Signed-off-by: Ekaterina Pavlova <ekt@morphbits.io>
 2024-12-13 17:07:52 +03:00
Ekaterina Pavlova
be4fc98041 cli: refactor upload-bin handler to reuse SDK built-in functionality 
Close #3714

Signed-off-by: Ekaterina Pavlova <ekt@morphbits.io>
 2024-12-13 16:36:37 +03:00
Ekaterina Pavlova
247ee831e5 cli: change block timestamp attribute of objects in upload-bin 
Close #3654

Signed-off-by: Ekaterina Pavlova <ekt@morphbits.io>
 2024-12-13 16:35:14 +03:00
Ekaterina Pavlova
46cbfab264 cli: add more debug info about retry to the upload-bin 
Signed-off-by: Ekaterina Pavlova <ekt@morphbits.io>
 2024-12-13 13:52:33 +03:00
Ekaterina Pavlova
65bdc82da8 *: move constant and NeoFS related code into separate package 
Signed-off-by: Ekaterina Pavlova <ekt@morphbits.io>
 2024-12-13 11:43:19 +03:00
Ekaterina Pavlova
62615f8c7e cli: update upload-bin to create index files during block uploading 
Close #3655
Close #3652

Signed-off-by: Ekaterina Pavlova <ekt@morphbits.io>
 2024-12-12 12:08:49 +03:00
Ekaterina Pavlova
04516e7d26 neofs: add pool support for NeoFS operations 
Signed-off-by: Ekaterina Pavlova <ekt@morphbits.io>
 2024-11-28 16:13:36 +03:00
Ekaterina Pavlova
43609dd984 *: migrate to SEARCH with strict equality comparator 
Close #3670

Signed-off-by: Ekaterina Pavlova <ekt@morphbits.io>
 2024-11-25 13:19:35 +03:00
Ekaterina Pavlova
616c805ac5 cli: refactor index file search in upload-bin command 
Align index file searching logic with block search.

Signed-off-by: Ekaterina Pavlova <ekt@morphbits.io>
 2024-11-22 17:45:35 +03:00
Ekaterina Pavlova
5dab154582 cli: split searchBatchSize usage in upload-bin command 
Split searchBatchSize into constant for searching and uploading.

Signed-off-by: Ekaterina Pavlova <ekt@morphbits.io>
 2024-11-22 17:45:35 +03:00
Ekaterina Pavlova
9fd375d56d cli: fix nil error of fetchLatestMissingBlockIndex in upload-bin 
Close #3653

Signed-off-by: Ekaterina Pavlova <ekt@morphbits.io>
 2024-11-22 17:45:35 +03:00
Ekaterina Pavlova
5f6284de05 cli: fix skip-blocks-uploading mode of upload-bin command 
If uploadBlocks succeeded uploadIndexFiles should work based on the
currentBlockHeight.

Signed-off-by: Ekaterina Pavlova <ekt@morphbits.io>
 2024-11-15 17:28:24 +03:00
Ekaterina Pavlova
58ed448f8d cli: add debug mode to upload-bin command 
Signed-off-by: Ekaterina Pavlova <ekt@morphbits.io>
 2024-11-15 17:18:44 +03:00
Ekaterina Pavlova
375b095f78 cli: add details to the logs in upload-bin 
Signed-off-by: Ekaterina Pavlova <ekt@morphbits.io>
 2024-11-15 14:34:36 +03:00
Ekaterina Pavlova
380d112599 cli: fix stopping of uploadIndexFiles in upload-bin 
The expected count of index files should be counted based on the already
uploaded object number.

Close #3669

Signed-off-by: Ekaterina Pavlova <ekt@morphbits.io>
 2024-11-15 14:34:36 +03:00
Ekaterina Pavlova
6e863e9a06 cli: fix fetchLatestMissingBlockIndex in upload-bin command 
Close #3667

Signed-off-by: Ekaterina Pavlova <ekt@morphbits.io>
 2024-11-15 14:34:10 +03:00
Ekaterina Pavlova
29e2d712f4 cli: add flag max-retries for upload-bin 
Signed-off-by: Ekaterina Pavlova <ekt@morphbits.io>
 2024-11-06 11:18:41 +03:00
Ekaterina Pavlova
18ed3bb3e8 cli: fix missing errors on closure of writer in upload-bin 
Ref #3658

Signed-off-by: Ekaterina Pavlova <ekt@morphbits.io>
 2024-11-06 11:16:46 +03:00
Anna Shaleva
a85f3ce83f cli: rename uploader.go to upload_bin.go 
Follow the style of other files in the `util` package and existing
`dump_bin.go`.

Signed-off-by: Anna Shaleva <shaleva.ann@nspcc.ru>
 2024-11-05 12:43:20 +03:00
Anna Shaleva
22cb7feb1d cli: fix format 
Signed-off-by: Anna Shaleva <shaleva.ann@nspcc.ru>
 2024-11-05 11:25:08 +03:00
Anna Shaleva
82b508e8f3 cli: extend logs of index file construction for upload-bin 
These logs are needed to track the progress of uploaders.

Signed-off-by: Anna Shaleva <shaleva.ann@nspcc.ru>
 2024-11-05 11:24:57 +03:00
Ekaterina Pavlova
c955c1e4ea cli: use GET instead of HEAD in upload bin 
Will be reverted when nspcc-dev/neofs-node#2988 is fixed. Currently HEAD
request can return incorrect attributes, which can lead to incorrect
index files.

Signed-off-by: Ekaterina Pavlova <ekt@morphbits.io>
 2024-11-02 12:04:02 +03:00
Ekaterina Pavlova
35d12779d6 cli: adjust index file creation in upload-bin 
In case of incomplete search result it will try to find each missed oid
and process it. In case of duplicates the first found will be in index
file.

Close #3647

Signed-off-by: Ekaterina Pavlova <ekt@morphbits.io>
 2024-11-01 19:51:01 +03:00
Ekaterina Pavlova
365bbe08ed cli: refactor errors and comments in upload-bin 
Signed-off-by: Ekaterina Pavlova <ekt@morphbits.io>
 2024-11-01 14:58:48 +03:00
Ekaterina Pavlova
6a93f70728 cli: add flag for skipping block uploading in upload-bin 
Signed-off-by: Ekaterina Pavlova <ekt@morphbits.io>
 2024-10-31 18:39:40 +03:00
Ekaterina Pavlova
7b3eeb9061 cli: fix usage of upload-bin 
Signed-off-by: Ekaterina Pavlova <ekt@morphbits.io>
 2024-10-24 12:19:12 +03:00
Ekaterina Pavlova
36e855609d cli: add workers and searchers flags to upload-bin command 
Number of workers to fetch, upload and search blocks concurrently is now
configurable.

Close #3641

Signed-off-by: Ekaterina Pavlova <ekt@morphbits.io>
 2024-10-24 12:19:10 +03:00
Ekaterina Pavlova
a5e9ab6979 cli: verify index file construction in upload-bin command 
Verify that there are no empty OIDs in the constructed index file, as
empty payloads can result in improperly set attributes, leading to empty
OIDs being added to the index file.

Close #3628

Signed-off-by: Ekaterina Pavlova <ekt@morphbits.io>
 2024-10-23 15:30:12 +03:00
Ekaterina Pavlova
42c8e40eaa cli: add retry to all requests to NeoFS in upload-bin 
Add retry to `NetworkInfo` and `uploadObj`.

Signed-off-by: Ekaterina Pavlova <ekt@morphbits.io>
 2024-10-23 15:30:12 +03:00
Ekaterina Pavlova
8b43c33e44 cli: extend object attribute parsing error in upload-bin 
Signed-off-by: Ekaterina Pavlova <ekt@morphbits.io>
 2024-10-23 15:30:12 +03:00
Ekaterina Pavlova
5b793bcf1b cli: fix process termination in upload-bin command 
Add a return statement to properly handle errors, ensuring the process
terminates as expected.

Signed-off-by: Ekaterina Pavlova <ekt@morphbits.io>
 2024-10-23 15:30:12 +03:00
Ekaterina Pavlova
e83b3e4839 cli: fix error handling in upload-bin command 
Fix shared global error reuse.

Close #3634

Signed-off-by: Ekaterina Pavlova <ekt@morphbits.io>
 2024-10-23 15:30:12 +03:00
Ekaterina Pavlova
c47d4e6c5b BlockFetcher/BlockUploader: fix and add additional attributes 
Capitalize block objects and index file objects attributes.
Add network magic attribute to container.

Close #3631

Signed-off-by: Ekaterina Pavlova <ekt@morphbits.io>
 2024-10-23 13:52:06 +03:00
Ekaterina Pavlova
e00d7fec7f cli: fix empty block uploading in util upload-bin 
In case of uploading retry already drained bin buffer is returned in
`bw.Bytes`, which leads to empty block uploading.

Close #3630

Signed-off-by: Ekaterina Pavlova <ekt@morphbits.io>
 2024-10-21 22:44:06 +03:00
Ekaterina Pavlova
41873e35c6 cli: increase NeoFS pool deadlines for upload-bin command 
`defaultDialTimeout` and `defaultStreamTimeout` increased because of
`code = DeadlineExceeded desc = context deadline exceeded`

Close #3620

Signed-off-by: Ekaterina Pavlova <ekt@morphbits.io>
 2024-10-18 19:59:27 +03:00
Ekaterina Pavlova
87bbff9831 cli: adjust error of upload-bin command 
Miscalculation leads to negative values in errors:
```
2024-10-17 11:26:56.790	failed to fetch the latest missing block index
 from container: search of index files failed for batch with indexes
 from -260000 to -250000: error during object IDs iteration: context
 deadline exceeded
```

Close #3621

Signed-off-by: Ekaterina Pavlova <ekt@morphbits.io>
 2024-10-18 14:26:17 +03:00
Ekaterina Pavlova
6199240598 cli: add upload-bin 
This command is used for keeping container with blocks for
blockfetcher updated.

Close #3578

Signed-off-by: Ekaterina Pavlova <ekt@morphbits.io>
 2024-10-17 11:56:07 +03:00
Roman Khimov
1c1d77c9b8 *: make use of min/max where appropriate 
Signed-off-by: Roman Khimov <roman@nspcc.ru>
 2024-08-23 22:11:29 +03:00
Ekaterina Pavlova
fc2798f9d1 cli: fix canceltx usage text 
We do not have `--account` flag.

Signed-off-by: Ekaterina Pavlova <ekt@morphbits.io>
 2024-07-12 12:16:24 +03:00
Ekaterina Pavlova
39559b90e2 cli: add Required field for flags 
Close #2861

Signed-off-by: Ekaterina Pavlova <ekt@morphbits.io>
 2024-07-12 11:58:12 +03:00
Ekaterina Pavlova
acde7bd0de cli: upgrade urfave lib to v2 
Close #3097

Signed-off-by: Ekaterina Pavlova <ekt@morphbits.io>
 2024-07-12 11:58:12 +03:00
Ekaterina Pavlova
2f5c26f14b cli: use capital letter for Usage 
To unified cli help only capital letter is used in Usage field.

Signed-off-by: Ekaterina Pavlova <ekt@morphbits.io>
 2024-07-12 11:58:12 +03:00
Anna Shaleva
69bcd4b7d9 cli: unify commands description 
Always put blank line in the end of the command description.

Signed-off-by: Anna Shaleva <shaleva.ann@nspcc.ru>
 2024-04-10 19:21:40 +03:00
Anna Shaleva
b6a9c64c55 cli: ensure all outcomes are handled in TestAwaitUtilCancelTx 
Close #3365.

Signed-off-by: Anna Shaleva <shaleva.ann@nspcc.ru>
 2024-04-03 16:40:11 +03:00
Anna Shaleva
e0e7fdf810 cli: unify canceltx command output 
Always return "Target transaction accepted" error if the target transation was
either accepted by the moment of command run or during the command
handling. Exit with code 1 in both cases. Adjust TestAwaitUtilCancelTx
correspondingly, allow both cases in test. Simplify the test along the
way, remove useless code.

Close #3365.

Signed-off-by: Anna Shaleva <shaleva.ann@nspcc.ru>
 2024-04-02 15:23:25 +03:00
Ekaterina Pavlova
d80e14dcbb cli: fix TestAwaitUtilCancelTx failing 
Occasionally the block is being accepted right after main transaction
submission. Added two branches into this TestAwaitUtilCancelTx. One
branch handles the case of original transaction acceptance, the
other branch handles the conflicting transaction acceptance.

Close #3365

Signed-off-by: Ekaterina Pavlova <ekt@morphbits.io>
 2024-03-22 16:22:11 +03:00