TrueCloudLab/frostfs-http-gw
17
1
Fork 18
Code Issues 9 Pull requests 3 Projects Releases 16 Activity Actions

[#170] tar.gz extraction during upload #176

Closed
nzinkevich wants to merge 1 commit from nzinkevich/frostfs-http-gw:zip_explode into master
pull from: nzinkevich/frostfs-http-gw:zip_explode
merge into: TrueCloudLab:master
Conversation 1 Commits 1 Files changed 7 +149 -59
nzinkevich commented 2024-12-06 12:04:23 +00:00
Member
Copy link
No description provided.
nzinkevich self-assigned this 2024-12-06 12:04:23 +00:00
nzinkevich added 3 commits 2024-12-06 12:04:24 +00:00 
During upload if "X-Attribute-Explode-Archive" is set, gate tries to read tar.gz archive and creates an object for each file. Each object acquires a FilePath attribute which is calculated relative to the archive root

Signed-off-by: Nikita Zinkevich <n.zinkevich@yadro.com>
Split DownloadZipped handler on methods. Add handler DownloadTar for downloading tar.gz archives. Make methods more universal for using in both implementations

Signed-off-by: Nikita Zinkevich <n.zinkevich@yadro.com>
[#170] Updated docs and configuration of archive section
Some checks failed
/ DCO (pull_request) Failing after 1m53s
Details
/ Vulncheck (pull_request) Successful in 2m54s
Details
/ Builds (pull_request) Successful in 3m1s
Details
/ Lint (pull_request) Successful in 4m25s
Details
/ Tests (pull_request) Successful in 3m3s
Details
0b16911a06
requested reviews from dkirillov, alexvanin, storage-services-committers, storage-services-developers 2024-12-06 12:05:23 +00:00
nzinkevich force-pushed zip_explode from 0b16911a06 to b00a4b4847 2024-12-06 12:06:15 +00:00 Compare
nzinkevich force-pushed zip_explode from b00a4b4847 to 240bf7bc72 2024-12-06 12:09:23 +00:00 Compare
nzinkevich changed title from [#170] GZIP tar downloading and uploading to [#170] tar.gz downloading and uploading 2024-12-06 12:09:55 +00:00
nzinkevich force-pushed zip_explode from 240bf7bc72 to 973373685e 2024-12-06 12:14:48 +00:00 Compare
nzinkevich changed title from [#170] tar.gz downloading and uploading to [#170] Archive extraction during upload 2024-12-06 12:17:25 +00:00
nzinkevich changed title from [#170] Archive extraction during upload to [#170] tar.gz extraction during upload 2024-12-06 12:17:35 +00:00
nzinkevich changed title from [#170] tar.gz extraction during upload to WIP: [#170] tar.gz extraction during upload 2024-12-06 12:29:36 +00:00
nzinkevich force-pushed zip_explode from 973373685e to 69ad977577 2024-12-06 12:38:54 +00:00 Compare
nzinkevich changed title from WIP: [#170] tar.gz extraction during upload to [#170] tar.gz extraction during upload 2024-12-06 12:39:35 +00:00
nzinkevich force-pushed zip_explode from 69ad977577 to 82271223ae 2024-12-10 06:34:43 +00:00 Compare
dkirillov reviewed 2024-12-11 09:52:21 +00:00
internal/handler/upload.go Outdated
@ -24,1 +28,3 @@
drainBufSize = 4096
jsonHeader = "application/json; charset=UTF-8"
drainBufSize = 4096
explodeArchiveHeader = "Explode-Archive"
dkirillov commented 2024-12-11 09:09:28 +00:00
Member
Copy link

Header should be X-Expode-Archive

Header should be `X-Expode-Archive`
dkirillov marked this conversation as resolved
internal/handler/upload.go Outdated
@ -73,3 +75,3 @@
log.Debug(
logs.CloseTemporaryMultipartFormFile,
zap.Stringer("address", addr),
zap.Stringer("container", bktInfo.CID),
dkirillov commented 2024-12-11 09:16:26 +00:00
Member
Copy link

Actually, as I remembered this log was aimed to print address of uploaded objects. It's better to keep such behavior. Maybe using some different approach but still

Actually, as I remembered this log was aimed to print address of uploaded objects. It's better to keep such behavior. Maybe using some different approach but still
dkirillov commented 2024-12-11 09:52:13 +00:00
Member
Copy link

Also, I don't think we should add attribute Explode-Archive to result objects

c.Request.Header.Peek(utils.UserAttributeHeaderPrefix + explodeArchiveHeader)
Also, I don't think we should add attribute `Explode-Archive` to result objects ```golang c.Request.Header.Peek(utils.UserAttributeHeaderPrefix + explodeArchiveHeader) ```
dkirillov marked this conversation as resolved
internal/handler/upload.go Outdated
@ -88,0 +121,4 @@
return
}
c.Response.Header.SetContentType(jsonHeader)
dkirillov commented 2024-12-11 09:37:12 +00:00
Member
Copy link

Despite this won't affect behavior, can we set header before setting body?

Despite this won't affect behavior, can we set header before setting body?
dkirillov marked this conversation as resolved
internal/handler/upload.go Outdated
@ -151,0 +198,4 @@
// explodeGzip read files from tar.gz archive and creates objects for each of them.
// Sets FilePath attribute with name from tar.Header.
func (h *Handler) explodeGzip(c *fasthttp.RequestCtx, log *zap.Logger, bktInfo *data.BucketInfo, file io.Reader) {
gzipReader, err := gzip.NewReader(file)
dkirillov commented 2024-12-11 09:50:52 +00:00
Member
Copy link

Why do we require archive be gziped? I supposed we can upload archive created like tar -cf dir.tar.gz dir but not it doesn't work:

create gzip reader: gzip: invalid header
Why do we require archive be gziped? I supposed we can upload archive created like `tar -cf dir.tar.gz dir` but not it doesn't work: ``` create gzip reader: gzip: invalid header ```
dkirillov commented 2024-12-11 11:33:46 +00:00
Member
Copy link

Oh, It seems I was wrong with command. I should use -z flag

But should we support not zipped tar?

Oh, It seems I was wrong with command. I should use `-z` flag But should we support not zipped tar?
nzinkevich commented 2024-12-13 05:41:53 +00:00
Author
Member
Copy link

Yes, I added support for this after you comment

Yes, I added support for this after you comment
dkirillov marked this conversation as resolved
internal/handler/upload.go Outdated
@ -175,0 +229,4 @@
if err != nil {
return
}
log.Debug(logs.ObjectUploaded, zap.String("object ID", idObj.EncodeToString()))
dkirillov commented 2024-12-11 09:41:19 +00:00
Member
Copy link

Let's log also object name

Let's log also object name
dkirillov marked this conversation as resolved
internal/logs/logs.go Outdated
@ -30,0 +32,4 @@
FailedToReadFileFromTar = "failed to read file from tar" // Error in ../../uploader/upload.go
FailedToFilterHeaders = "failed to filter headers" // Error in ../../uploader/upload.go
FailedToUploadObject = "failed to upload object" // Error in ../../uploader/upload.go
ObjectUploaded = "object uploaded" // Debug in ../../uploader/upload.go
dkirillov commented 2024-12-11 09:13:02 +00:00
Member
Copy link

It's better not to add comments at the end of lines

It's better not to add comments at the end of lines
dkirillov marked this conversation as resolved
alexvanin added this to the v0.33.0 milestone 2024-12-12 11:57:20 +00:00
nzinkevich force-pushed zip_explode from 82271223ae to beb4d6a646 2024-12-13 12:49:05 +00:00 Compare
nzinkevich force-pushed zip_explode from beb4d6a646 to 6e5189d6a9 2024-12-13 12:51:03 +00:00 Compare
nzinkevich closed this pull request 2024-12-14 08:21:56 +00:00
nzinkevich commented 2024-12-14 08:22:19 +00:00
Author
Member
Copy link

Moved changes to #177

Moved changes to #177
All checks were successful
/ DCO (pull_request) Successful in 4m4s
Required
Details
/ Builds (pull_request) Successful in 2m4s
Required
Details
/ Lint (pull_request) Successful in 2m54s
Required
Details
/ Tests (pull_request) Successful in 2m6s
Required
Details
/ Vulncheck (pull_request) Successful in 4m33s
Required
Details

Pull request closed

Please reopen this pull request to perform a merge.
Sign in to join this conversation.
Reviewers
No reviewers
dkirillov
alexvanin
TrueCloudLab/storage-services-committers
TrueCloudLab/storage-services-developers
Labels
Clear labels   
P0

Highest

  
P1

High

  
P2

Medium

  
P3

Low

  
good first issue

Good for newcomers

  
Infrastructure

Infrastructure, Automation, CI/CD or DevOps related stuff

  
blocked

The issue or PR is blocked by something

  
bug

Something is not working

  
config

Configuration format update or breaking change

  
discussion

Talking about something in order to reach a decision or to exchange ideas

  
documentation

Documentation related

  
duplicate

This issue or pull request already exists

  
enhancement

New feature

  
go

Language features adoption

  
help wanted

Extra attention is needed

  
internal

   
invalid

Something is wrong

  
kludge

This is a kludge and we know it

  
observability

Related to metrics, tracing and logs

  
perfomance

Perfomance related

  
question

More information is needed

  
refactoring

Refactoring

  
wontfix

This won't be fixed

No labels
P0
P1
P2
P3
good first issue
Infrastructure
blocked
bug
config
discussion
documentation
duplicate
enhancement
go
help wanted
internal
invalid
kludge
observability
perfomance
question
refactoring
wontfix
Milestone
Clear milestone
No items
No milestone
v0.33.0
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
nzinkevich
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: TrueCloudLab/frostfs-http-gw#176
No description provided.
Delete branch "nzinkevich/frostfs-http-gw:zip_explode"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?