Commit graph

18 commits

Author SHA1 Message Date
Anna Shaleva
a85f3ce83f cli: rename uploader.go to upload_bin.go
Follow the style of other files in the `util` package and existing
`dump_bin.go`.

Signed-off-by: Anna Shaleva <shaleva.ann@nspcc.ru>
2024-11-05 12:43:20 +03:00
Anna Shaleva
22cb7feb1d cli: fix format
Signed-off-by: Anna Shaleva <shaleva.ann@nspcc.ru>
2024-11-05 11:25:08 +03:00
Anna Shaleva
82b508e8f3 cli: extend logs of index file construction for upload-bin
These logs are needed to track the progress of uploaders.

Signed-off-by: Anna Shaleva <shaleva.ann@nspcc.ru>
2024-11-05 11:24:57 +03:00
Ekaterina Pavlova
c955c1e4ea cli: use GET instead of HEAD in upload bin
Will be reverted when nspcc-dev/neofs-node#2988 is fixed. Currently HEAD
request can return incorrect attributes, which can lead to incorrect
index files.

Signed-off-by: Ekaterina Pavlova <ekt@morphbits.io>
2024-11-02 12:04:02 +03:00
Ekaterina Pavlova
35d12779d6 cli: adjust index file creation in upload-bin
In case of incomplete search result it will try to find each missed oid
and process it. In case of duplicates the first found will be in index
file.

Close #3647

Signed-off-by: Ekaterina Pavlova <ekt@morphbits.io>
2024-11-01 19:51:01 +03:00
Ekaterina Pavlova
365bbe08ed cli: refactor errors and comments in upload-bin
Signed-off-by: Ekaterina Pavlova <ekt@morphbits.io>
2024-11-01 14:58:48 +03:00
Ekaterina Pavlova
6a93f70728 cli: add flag for skipping block uploading in upload-bin
Signed-off-by: Ekaterina Pavlova <ekt@morphbits.io>
2024-10-31 18:39:40 +03:00
Ekaterina Pavlova
36e855609d cli: add workers and searchers flags to upload-bin command
Number of workers to fetch, upload and search blocks concurrently is now
configurable.

Close #3641

Signed-off-by: Ekaterina Pavlova <ekt@morphbits.io>
2024-10-24 12:19:10 +03:00
Ekaterina Pavlova
a5e9ab6979 cli: verify index file construction in upload-bin command
Verify that there are no empty OIDs in the constructed index file, as
empty payloads can result in improperly set attributes, leading to empty
OIDs being added to the index file.

Close #3628

Signed-off-by: Ekaterina Pavlova <ekt@morphbits.io>
2024-10-23 15:30:12 +03:00
Ekaterina Pavlova
42c8e40eaa cli: add retry to all requests to NeoFS in upload-bin
Add retry to `NetworkInfo` and `uploadObj`.

Signed-off-by: Ekaterina Pavlova <ekt@morphbits.io>
2024-10-23 15:30:12 +03:00
Ekaterina Pavlova
8b43c33e44 cli: extend object attribute parsing error in upload-bin
Signed-off-by: Ekaterina Pavlova <ekt@morphbits.io>
2024-10-23 15:30:12 +03:00
Ekaterina Pavlova
5b793bcf1b cli: fix process termination in upload-bin command
Add a return statement to properly handle errors, ensuring the process
terminates as expected.

Signed-off-by: Ekaterina Pavlova <ekt@morphbits.io>
2024-10-23 15:30:12 +03:00
Ekaterina Pavlova
e83b3e4839 cli: fix error handling in upload-bin command
Fix shared global error reuse.

Close #3634

Signed-off-by: Ekaterina Pavlova <ekt@morphbits.io>
2024-10-23 15:30:12 +03:00
Ekaterina Pavlova
c47d4e6c5b BlockFetcher/BlockUploader: fix and add additional attributes
Capitalize block objects and index file objects attributes.
Add network magic attribute to container.

Close #3631

Signed-off-by: Ekaterina Pavlova <ekt@morphbits.io>
2024-10-23 13:52:06 +03:00
Ekaterina Pavlova
e00d7fec7f cli: fix empty block uploading in util upload-bin
In case of uploading retry already drained bin buffer is returned in
`bw.Bytes`, which leads to empty block uploading.

Close #3630

Signed-off-by: Ekaterina Pavlova <ekt@morphbits.io>
2024-10-21 22:44:06 +03:00
Ekaterina Pavlova
41873e35c6 cli: increase NeoFS pool deadlines for upload-bin command
`defaultDialTimeout` and `defaultStreamTimeout` increased because of
`code = DeadlineExceeded desc = context deadline exceeded`

Close #3620

Signed-off-by: Ekaterina Pavlova <ekt@morphbits.io>
2024-10-18 19:59:27 +03:00
Ekaterina Pavlova
87bbff9831 cli: adjust error of upload-bin command
Miscalculation leads to negative values in errors:
```
2024-10-17 11:26:56.790	failed to fetch the latest missing block index
 from container: search of index files failed for batch with indexes
 from -260000 to -250000: error during object IDs iteration: context
 deadline exceeded
```

Close #3621

Signed-off-by: Ekaterina Pavlova <ekt@morphbits.io>
2024-10-18 14:26:17 +03:00
Ekaterina Pavlova
6199240598 cli: add upload-bin
This command is used for keeping container with blocks for
blockfetcher updated.

Close #3578

Signed-off-by: Ekaterina Pavlova <ekt@morphbits.io>
2024-10-17 11:56:07 +03:00