Support file bulk upload for frostfs-cli #1552

New issue

Open

opened 2024-12-11 09:59:30 +00:00 by alexvanin · 4 comments

alexvanin commented

2024-12-11 09:59:30 +00:00

Owner

Sometimes I need to upload a bunch of files or directories into FrostFS Storage. To do that I have to run frostfs-cli in bash script.

Describe the solution you'd like

Consider having something like aws s3 sync command, that allows to synchronize selected files or directories. While it can delete files from storage as well, it would be nice to have at least bulk upload.

This new command can also provide both FilePath and FileName attributes.

Describe alternatives you've considered

Do nothing.

Additional context

For my demo, I needed to upload a bunch of video files so I modified frostfs-cli object put command to support dirs and wildcards, e.g. --file index*.ts

## Is your feature request related to a problem? Please describe. Sometimes I need to upload a bunch of files or directories into FrostFS Storage. To do that I have to run frostfs-cli in bash script. ## Describe the solution you'd like Consider having something like [aws s3 sync](https://docs.aws.amazon.com/cli/latest/reference/s3/sync.html) command, that allows to synchronize selected files or directories. While it can delete files from storage as well, it would be nice to have at least bulk upload. This new command can also provide both [FilePath](https://git.frostfs.info/TrueCloudLab/frostfs-api/src/commit/8540f0d18d5d26fd32ead801478e8405b8fbaaaf/object/types.proto#L145) and [FileName](https://git.frostfs.info/TrueCloudLab/frostfs-api/src/commit/8540f0d18d5d26fd32ead801478e8405b8fbaaaf/object/types.proto#L143) attributes. ## Describe alternatives you've considered Do nothing. ## Additional context For my demo, I needed to upload a bunch of video files so I modified `frostfs-cli object put` command to support dirs and wildcards, e.g. `--file index*.ts`

alexvanin added the

discussion

triage

labels 2024-12-11 09:59:30 +00:00

fyrchik commented

2024-12-11 10:18:22 +00:00

Owner

I would add a different command: it might have more parameters, related specifically to bulk upload. Not to clutter object put.
That said, the problem is easily solved via external utils, so let's disscuss whether the supposed additional value is big enought to justify implementing this command.

I would add a different command: it might have more parameters, related specifically to bulk upload. Not to clutter `object put`. That said, the problem is easily solved via external utils, so let's disscuss whether the supposed additional value is big enought to justify implementing this command.

fyrchik added the

enhancement

frostfs-cli

labels 2024-12-11 10:18:35 +00:00

nzinkevich commented

2024-12-18 07:45:54 +00:00

Member

Also it would be nice to have a bulk delete command in frostfs-cli. It can be useful to empty container or by a particular prefix after testing of bulk uploading, without having to recreate container or delete one-by-one

Also it would be nice to have a bulk delete command in `frostfs-cli`. It can be useful to empty container or by a particular prefix after testing of bulk uploading, without having to recreate container or delete one-by-one

fyrchik commented

2024-12-18 08:22:11 +00:00

Owner

Do you suggest removing objects based on some search filter?
There are special cases here, like filtering --user objects, while also removing incomplete big object uploads.

Do you suggest removing objects based on some search filter? There are special cases here, like filtering `--user` objects, while also removing incomplete big object uploads.

nzinkevich commented

2024-12-18 08:38:35 +00:00

Member

Yes, I mean deleting data, created by user with ability to filter. Maybe it can be done like in find command, where you can filter search result and specify action for each entry. Looks extensible for future needs

find / -name "*.old" -delete
find . -exec /bin/rm {} \;

Yes, I mean deleting data, created by user with ability to filter. Maybe it can be done like in `find` command, where you can filter search result and specify action for each entry. Looks extensible for future needs ``` find / -name "*.old" -delete find . -exec /bin/rm {} \; ```