TrueCloudLab/frostfs-http-gw
17
1
Fork 17
Code Issues 12 Pull requests 7 Projects Releases 9 Activity Actions

[#170] tar.gz extraction during upload #176

Open
nzinkevich wants to merge 1 commit from nzinkevich/frostfs-http-gw:zip_explode into master
pull from: nzinkevich/frostfs-http-gw:zip_explode
merge into: TrueCloudLab:master
Conversation 0 Commits 1 Files changed 5 +106 -45
nzinkevich commented 2024-12-06 12:04:23 +00:00
Member
Copy link
No description provided.
nzinkevich self-assigned this 2024-12-06 12:04:23 +00:00
nzinkevich added 3 commits 2024-12-06 12:04:24 +00:00 
During upload if "X-Attribute-Explode-Archive" is set, gate tries to read tar.gz archive and creates an object for each file. Each object acquires a FilePath attribute which is calculated relative to the archive root

Signed-off-by: Nikita Zinkevich <n.zinkevich@yadro.com>
Split DownloadZipped handler on methods. Add handler DownloadTar for downloading tar.gz archives. Make methods more universal for using in both implementations

Signed-off-by: Nikita Zinkevich <n.zinkevich@yadro.com>
[#170] Updated docs and configuration of archive section
Some checks failed
/ DCO (pull_request) Failing after 1m53s
Details
/ Vulncheck (pull_request) Successful in 2m54s
Details
/ Builds (pull_request) Successful in 3m1s
Details
/ Lint (pull_request) Successful in 4m25s
Details
/ Tests (pull_request) Successful in 3m3s
Details
0b16911a06
nzinkevich requested review from dkirillov 2024-12-06 12:04:24 +00:00
nzinkevich requested review from alexvanin 2024-12-06 12:04:24 +00:00
nzinkevich requested review from storage-services-committers 2024-12-06 12:05:19 +00:00
nzinkevich requested review from storage-services-developers 2024-12-06 12:05:23 +00:00
nzinkevich force-pushed zip_explode from 0b16911a06 to b00a4b4847 2024-12-06 12:06:15 +00:00 Compare
nzinkevich force-pushed zip_explode from b00a4b4847 to 240bf7bc72 2024-12-06 12:09:23 +00:00 Compare
nzinkevich changed title from [#170] GZIP tar downloading and uploading to [#170] tar.gz downloading and uploading 2024-12-06 12:09:55 +00:00
nzinkevich force-pushed zip_explode from 240bf7bc72 to 973373685e 2024-12-06 12:14:48 +00:00 Compare
nzinkevich changed title from [#170] tar.gz downloading and uploading to [#170] Archive extraction during upload 2024-12-06 12:17:25 +00:00
nzinkevich changed title from [#170] Archive extraction during upload to [#170] tar.gz extraction during upload 2024-12-06 12:17:35 +00:00
nzinkevich changed title from [#170] tar.gz extraction during upload to WIP: [#170] tar.gz extraction during upload 2024-12-06 12:29:36 +00:00
nzinkevich force-pushed zip_explode from 973373685e to 69ad977577 2024-12-06 12:38:54 +00:00 Compare
nzinkevich changed title from WIP: [#170] tar.gz extraction during upload to [#170] tar.gz extraction during upload 2024-12-06 12:39:35 +00:00
nzinkevich force-pushed zip_explode from 69ad977577 to 82271223ae 2024-12-10 06:34:43 +00:00 Compare
dkirillov reviewed 2024-12-11 09:52:21 +00:00
internal/handler/upload.go
@ -24,1 +28,3 @@
drainBufSize = 4096
jsonHeader = "application/json; charset=UTF-8"
drainBufSize = 4096
explodeArchiveHeader = "Explode-Archive"
dkirillov commented 2024-12-11 09:09:28 +00:00
Member
Copy link

Header should be X-Expode-Archive

Header should be `X-Expode-Archive`
internal/handler/upload.go
@ -73,3 +75,3 @@
log.Debug(
logs.CloseTemporaryMultipartFormFile,
zap.Stringer("address", addr),
zap.Stringer("container", bktInfo.CID),
dkirillov commented 2024-12-11 09:16:26 +00:00
Member
Copy link

Actually, as I remembered this log was aimed to print address of uploaded objects. It's better to keep such behavior. Maybe using some different approach but still

Actually, as I remembered this log was aimed to print address of uploaded objects. It's better to keep such behavior. Maybe using some different approach but still
dkirillov commented 2024-12-11 09:52:13 +00:00
Member
Copy link

Also, I don't think we should add attribute Explode-Archive to result objects

c.Request.Header.Peek(utils.UserAttributeHeaderPrefix + explodeArchiveHeader)
Also, I don't think we should add attribute `Explode-Archive` to result objects ```golang c.Request.Header.Peek(utils.UserAttributeHeaderPrefix + explodeArchiveHeader) ```
internal/handler/upload.go
@ -88,0 +121,4 @@
return
}
c.Response.Header.SetContentType(jsonHeader)
dkirillov commented 2024-12-11 09:37:12 +00:00
Member
Copy link

Despite this won't affect behavior, can we set header before setting body?

Despite this won't affect behavior, can we set header before setting body?
internal/handler/upload.go
@ -151,0 +198,4 @@
// explodeGzip read files from tar.gz archive and creates objects for each of them.
// Sets FilePath attribute with name from tar.Header.
func (h *Handler) explodeGzip(c *fasthttp.RequestCtx, log *zap.Logger, bktInfo *data.BucketInfo, file io.Reader) {
gzipReader, err := gzip.NewReader(file)
dkirillov commented 2024-12-11 09:50:52 +00:00
Member
Copy link

Why do we require archive be gziped? I supposed we can upload archive created like tar -cf dir.tar.gz dir but not it doesn't work:

create gzip reader: gzip: invalid header
Why do we require archive be gziped? I supposed we can upload archive created like `tar -cf dir.tar.gz dir` but not it doesn't work: ``` create gzip reader: gzip: invalid header ```
dkirillov commented 2024-12-11 11:33:46 +00:00
Member
Copy link

Oh, It seems I was wrong with command. I should use -z flag

But should we support not zipped tar?

Oh, It seems I was wrong with command. I should use `-z` flag But should we support not zipped tar?
internal/handler/upload.go
@ -175,0 +229,4 @@
if err != nil {
return
}
log.Debug(logs.ObjectUploaded, zap.String("object ID", idObj.EncodeToString()))
dkirillov commented 2024-12-11 09:41:19 +00:00
Member
Copy link

Let's log also object name

Let's log also object name
internal/logs/logs.go
@ -30,0 +32,4 @@
FailedToReadFileFromTar = "failed to read file from tar" // Error in ../../uploader/upload.go
FailedToFilterHeaders = "failed to filter headers" // Error in ../../uploader/upload.go
FailedToUploadObject = "failed to upload object" // Error in ../../uploader/upload.go
ObjectUploaded = "object uploaded" // Debug in ../../uploader/upload.go
dkirillov commented 2024-12-11 09:13:02 +00:00
Member
Copy link

It's better not to add comments at the end of lines

It's better not to add comments at the end of lines
alexvanin added this to the v0.33.0 milestone 2024-12-12 11:57:20 +00:00
All checks were successful
/ DCO (pull_request) Successful in 1m19s
Required
Details
/ Vulncheck (pull_request) Successful in 1m55s
Required
Details
/ Builds (pull_request) Successful in 2m10s
Required
Details
/ Lint (pull_request) Successful in 3m31s
Required
Details
/ Tests (pull_request) Successful in 1m56s
Required
Details
This pull request doesn't have enough approvals yet. 0 of 2 approvals granted.
This branch is out-of-date with the base branch
You are not authorized to merge this pull request.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u zip_explode:nzinkevich-zip_explode
git checkout nzinkevich-zip_explode
Sign in to join this conversation.
Reviewers
No reviewers
dkirillov
alexvanin
TrueCloudLab/storage-services-committers
TrueCloudLab/storage-services-developers
Labels
Clear labels   
P0

Highest

  
P1

High

  
P2

Medium

  
P3

Low

  
good first issue

Good for newcomers

  
Infrastructure

Infrastructure, Automation, CI/CD or DevOps related stuff

  
blocked

The issue or PR is blocked by something

  
bug

Something is not working

  
config

Configuration format update or breaking change

  
discussion

Talking about something in order to reach a decision or to exchange ideas

  
documentation

Documentation related

  
duplicate

This issue or pull request already exists

  
enhancement

New feature

  
go

Language features adoption

  
help wanted

Extra attention is needed

  
internal

   
invalid

Something is wrong

  
kludge

This is a kludge and we know it

  
observability

Related to metrics, tracing and logs

  
perfomance

Perfomance related

  
question

More information is needed

  
refactoring

Refactoring

  
wontfix

This won't be fixed

No labels
P0
P1
P2
P3
good first issue
Infrastructure
blocked
bug
config
discussion
documentation
duplicate
enhancement
go
help wanted
internal
invalid
kludge
observability
perfomance
question
refactoring
wontfix
Milestone
Clear milestone
No items
No milestone
v0.33.0
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
nzinkevich
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: TrueCloudLab/frostfs-http-gw#176
No description provided.
Delete branch "nzinkevich/frostfs-http-gw:zip_explode"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?