TrueCloudLab/frostfs-http-gw
17
1
Fork 18
Code Issues 9 Pull requests 1 Projects Releases 13 Activity Actions

174-add_slash_clipping_when_search_by_filename_attr_master #205

Merged
alexvanin merged 2 commits from r.loginov/frostfs-http-gw:feature/174-add_slash_clipping_when_search_by_filename_attr_master into master 2025-02-07 12:14:11 +00:00
Conversation 6 Commits 2 Files changed 4 +52 -5
r.loginov commented 2025-02-04 10:26:42 +00:00
Member
Copy link

According to the FrostFS API specification,
the FileName attribute cannot contain a slash
at the beginning.

Test scenarios:

Container CID: 3TK3yYFQ9FqK9vGz5wfKpDS6KB81DYz8i3PinQiGLypW

Test scenario 1

Getting an object by the attribute FileName'. The value of the FileName` attribute has a "/" at the beginning.

Uploading an object:
# frostfs-cli object put -c cli.yml --cid 3TK3yYFQ9FqK9vGz5wfKpDS6KB81DYz8i3PinQiGLypW --file cat.jpg
 14396 / 14396 [==================================================================================================================================================================] 100.00% 0s
[cat.jpg] Object successfully stored
  OID: Hn9cVfXh6yMD41W9n4KhQ4UyYHQ1a2iCa4HGSyMvr3nq
  CID: 3TK3yYFQ9FqK9vGz5wfKpDS6KB81DYz8i3PinQiGLypW
By default, its attribute is FileName=cat.jpg :
# frostfs-cli object head -c cli.yml --cid 3TK3yYFQ9FqK9vGz5wfKpDS6KB81DYz8i3PinQiGLypW --oid Hn9cVfXh6yMD41W9n4KhQ4UyYHQ1a2iCa4HGSyMvr3nq
...
Attributes:
  FileName=cat.jpg
  Timestamp=1738665794 (2025-02-04 10:43:14 +0000 UTC)
...
We are trying to get an object using "/" at the beginning of the value of the attribute FileName:
# curl -i http://localhost:8888/get_by_attribute/3TK3yYFQ9FqK9vGz5wfKpDS6KB81DYz8i3PinQiGLypW/FileName//cat.jpg
HTTP/1.1 200 OK
Date: Tue, 04 Feb 2025 10:46:50 GMT
Content-Type: image/jpeg
Content-Length: 14396
X-Object-Id: Hn9cVfXh6yMD41W9n4KhQ4UyYHQ1a2iCa4HGSyMvr3nq
X-Owner-Id: NbToQ8qdTJERydwfsmt7AcPXi4xEpntZ2x
X-Container-Id: 3TK3yYFQ9FqK9vGz5wfKpDS6KB81DYz8i3PinQiGLypW
X-Attribute-FileName: cat.jpg
X-Attribute-Timestamp: 1738665794
Last-Modified: Tue, 04 Feb 2025 10:43:14 GMT
Content-Disposition: inline; filename=cat.jpg
Access-Control-Max-Age: 600
Logs of the frostfs-http-gw service:
2025-02-04T10:46:50.820Z        info        request        {"id": 85899345921, "remote": "127.0.0.1:51004", "method": "GET", "path": "/get_by_attribute/3TK3yYFQ9FqK9vGz5wfKpDS6KB81DYz8i3PinQiGLypW/FileName/cat.jpg", "query": ""}

Test scenario 2

We are retrieving an object using the FilePath attribute that does not exist. The value of the File Path attribute has a "/" at the beginning. Due to the fact that there is no such attribute, the gate will do a fallback search using the FileName attribute.

We are trying to get an object using the "/" at the beginning of the value of the "FilePath" attribute. (The bucket and the object from the previous test scenario):
# curl -i http://localhost:8888/get_by_attribute/3TK3yYFQ9FqK9vGz5wfKpDS6KB81DYz8i3PinQiGLypW/FilePath//cat.jpg
HTTP/1.1 200 OK
Date: Tue, 04 Feb 2025 10:52:38 GMT
Content-Type: image/jpeg
Content-Length: 14396
X-Object-Id: Hn9cVfXh6yMD41W9n4KhQ4UyYHQ1a2iCa4HGSyMvr3nq
X-Owner-Id: NbToQ8qdTJERydwfsmt7AcPXi4xEpntZ2x
X-Container-Id: 3TK3yYFQ9FqK9vGz5wfKpDS6KB81DYz8i3PinQiGLypW
X-Attribute-FileName: cat.jpg
X-Attribute-Timestamp: 1738665794
Last-Modified: Tue, 04 Feb 2025 10:43:14 GMT
Content-Disposition: inline; filename=cat.jpg
Access-Control-Max-Age: 600
Logs of the frostfs-http-gw service:
2025-02-04T10:52:38.881Z        info        request        {"id": 90194313217, "remote": "127.0.0.1:41800", "method": "GET", "path": "/get_by_attribute/3TK3yYFQ9FqK9vGz5wfKpDS6KB81DYz8i3PinQiGLypW/FilePath/cat.jpg", "query": ""}
2025-02-04T10:52:38.900Z        debug        object not found by filePath attribute, try search by fileName        {"id": 90194313217, "cid": "3TK3yYFQ9FqK9vGz5wfKpDS6KB81DYz8i3PinQiGLypW", "attr_key": "FilePath", "attr_val": "/cat.jpg"}

Signed-off-by: Roman Loginov r.loginov@yadro.com

According to the FrostFS API specification, the FileName attribute cannot contain a slash at the beginning. Test scenarios: Container CID: `3TK3yYFQ9FqK9vGz5wfKpDS6KB81DYz8i3PinQiGLypW` ### Test scenario 1 Getting an object by the attribute `FileName'. The value of the `FileName` attribute has a "/" at the beginning. ##### Uploading an object: ``` # frostfs-cli object put -c cli.yml --cid 3TK3yYFQ9FqK9vGz5wfKpDS6KB81DYz8i3PinQiGLypW --file cat.jpg 14396 / 14396 [==================================================================================================================================================================] 100.00% 0s [cat.jpg] Object successfully stored OID: Hn9cVfXh6yMD41W9n4KhQ4UyYHQ1a2iCa4HGSyMvr3nq CID: 3TK3yYFQ9FqK9vGz5wfKpDS6KB81DYz8i3PinQiGLypW ``` ##### By default, its attribute is `FileName=cat.jpg `: ``` # frostfs-cli object head -c cli.yml --cid 3TK3yYFQ9FqK9vGz5wfKpDS6KB81DYz8i3PinQiGLypW --oid Hn9cVfXh6yMD41W9n4KhQ4UyYHQ1a2iCa4HGSyMvr3nq ... Attributes: FileName=cat.jpg Timestamp=1738665794 (2025-02-04 10:43:14 +0000 UTC) ... ``` ##### We are trying to get an object using "/" at the beginning of the value of the attribute `FileName`: ``` # curl -i http://localhost:8888/get_by_attribute/3TK3yYFQ9FqK9vGz5wfKpDS6KB81DYz8i3PinQiGLypW/FileName//cat.jpg HTTP/1.1 200 OK Date: Tue, 04 Feb 2025 10:46:50 GMT Content-Type: image/jpeg Content-Length: 14396 X-Object-Id: Hn9cVfXh6yMD41W9n4KhQ4UyYHQ1a2iCa4HGSyMvr3nq X-Owner-Id: NbToQ8qdTJERydwfsmt7AcPXi4xEpntZ2x X-Container-Id: 3TK3yYFQ9FqK9vGz5wfKpDS6KB81DYz8i3PinQiGLypW X-Attribute-FileName: cat.jpg X-Attribute-Timestamp: 1738665794 Last-Modified: Tue, 04 Feb 2025 10:43:14 GMT Content-Disposition: inline; filename=cat.jpg Access-Control-Max-Age: 600 ``` ##### Logs of the frostfs-http-gw service: ``` 2025-02-04T10:46:50.820Z info request {"id": 85899345921, "remote": "127.0.0.1:51004", "method": "GET", "path": "/get_by_attribute/3TK3yYFQ9FqK9vGz5wfKpDS6KB81DYz8i3PinQiGLypW/FileName/cat.jpg", "query": ""} ``` ### Test scenario 2 We are retrieving an object using the `FilePath` attribute that does not exist. The value of the `File Path` attribute has a "/" at the beginning. Due to the fact that there is no such attribute, the gate will do a fallback search using the `FileName` attribute. ##### We are trying to get an object using the "/" at the beginning of the value of the "FilePath" attribute. (The bucket and the object from the previous test scenario): ``` # curl -i http://localhost:8888/get_by_attribute/3TK3yYFQ9FqK9vGz5wfKpDS6KB81DYz8i3PinQiGLypW/FilePath//cat.jpg HTTP/1.1 200 OK Date: Tue, 04 Feb 2025 10:52:38 GMT Content-Type: image/jpeg Content-Length: 14396 X-Object-Id: Hn9cVfXh6yMD41W9n4KhQ4UyYHQ1a2iCa4HGSyMvr3nq X-Owner-Id: NbToQ8qdTJERydwfsmt7AcPXi4xEpntZ2x X-Container-Id: 3TK3yYFQ9FqK9vGz5wfKpDS6KB81DYz8i3PinQiGLypW X-Attribute-FileName: cat.jpg X-Attribute-Timestamp: 1738665794 Last-Modified: Tue, 04 Feb 2025 10:43:14 GMT Content-Disposition: inline; filename=cat.jpg Access-Control-Max-Age: 600 ``` ##### Logs of the frostfs-http-gw service: ``` 2025-02-04T10:52:38.881Z info request {"id": 90194313217, "remote": "127.0.0.1:41800", "method": "GET", "path": "/get_by_attribute/3TK3yYFQ9FqK9vGz5wfKpDS6KB81DYz8i3PinQiGLypW/FilePath/cat.jpg", "query": ""} 2025-02-04T10:52:38.900Z debug object not found by filePath attribute, try search by fileName {"id": 90194313217, "cid": "3TK3yYFQ9FqK9vGz5wfKpDS6KB81DYz8i3PinQiGLypW", "attr_key": "FilePath", "attr_val": "/cat.jpg"} ``` Signed-off-by: Roman Loginov <r.loginov@yadro.com>
r.loginov self-assigned this 2025-02-04 10:26:42 +00:00
r.loginov added 1 commit 2025-02-04 10:26:43 +00:00
[#174] Add slash clipping for FileName attribute
All checks were successful
/ DCO (pull_request) Successful in 34s
Details
/ Vulncheck (pull_request) Successful in 59s
Details
/ Builds (pull_request) Successful in 1m9s
Details
/ OCI image (pull_request) Successful in 1m25s
Details
/ Lint (pull_request) Successful in 1m56s
Details
/ Tests (pull_request) Successful in 1m0s
Details
/ Integration tests (pull_request) Successful in 5m43s
Details
c6ad26a778 
According to the FrostFS API specification,
the FileName attribute cannot contain a slash
at the beginning.

Signed-off-by: Roman Loginov <r.loginov@yadro.com>
requested reviews from storage-services-developers, storage-services-committers 2025-02-04 10:26:43 +00:00
r.loginov changed title from 174-add_slash_clipping_when_search_by_filename_attr_master to WIP: 174-add_slash_clipping_when_search_by_filename_attr_master 2025-02-04 10:36:44 +00:00
r.loginov changed title from WIP: 174-add_slash_clipping_when_search_by_filename_attr_master to 174-add_slash_clipping_when_search_by_filename_attr_master 2025-02-04 11:11:35 +00:00
mbiryukova approved these changes 2025-02-05 10:08:02 +00:00
Dismissed
mbiryukova commented 2025-02-05 10:11:28 +00:00
Member
Copy link

Do we need to port release commit?

Do we need to port release commit?
👍 1
alexvanin commented 2025-02-05 11:29:49 +00:00
Owner
Copy link

Do we need to port release commit?

Yeah let's change changelog.md this way.

> Do we need to port release commit? Yeah let's change changelog.md this way.
👍 1
alexvanin added this to the v0.33.0 milestone 2025-02-05 11:31:41 +00:00
r.loginov added 1 commit 2025-02-05 11:41:07 +00:00
[#174] Port release v0.32.3 changelog
All checks were successful
/ DCO (pull_request) Successful in 29s
Details
/ Builds (pull_request) Successful in 1m6s
Details
/ Vulncheck (pull_request) Successful in 1m7s
Details
/ OCI image (pull_request) Successful in 1m26s
Details
/ Lint (pull_request) Successful in 2m6s
Details
/ Tests (pull_request) Successful in 1m0s
Details
/ Integration tests (pull_request) Successful in 5m44s
Details
6cdff1d1e2 
Signed-off-by: Roman Loginov <r.loginov@yadro.com>
r.loginov dismissed mbiryukova's review 2025-02-05 11:41:07 +00:00
Reason:

New commits pushed, approval review dismissed automatically according to repository settings

dkirillov approved these changes 2025-02-06 13:33:47 +00:00
Dismissed
dkirillov commented 2025-02-07 05:45:48 +00:00
Member
Copy link

As for me, I would be nice to have opportunity to search by attributes as it is. It can be different endpoint (or trimming slash only for get/<bucket>/<object> and not for `get_by_attributes///), config flag or something else.
What do you think?

As for me, I would be nice to have opportunity to search by attributes as it is. It can be different endpoint (or trimming slash only for `get/<bucket>/<object>` and not for `get_by_attributes/<bucket>/<attribute>/<value>), config flag or something else. What do you think?
r.loginov commented 2025-02-07 08:25:11 +00:00
Author
Member
Copy link

@dkirillov wrote in #205 (comment):

As for me, I would be nice to have opportunity to search by attributes as it is. It can be different endpoint (or trimming slash only for get/<bucket>/<object> and not for `get_by_attributes///), config flag or something else. What do you think?

Am I right that you want to do this for backward compatibility? Or in case the user suddenly wants to create objects with FileName attribute with leading slash?

@dkirillov wrote in https://git.frostfs.info/TrueCloudLab/frostfs-http-gw/pulls/205#issuecomment-66750: > As for me, I would be nice to have opportunity to search by attributes as it is. It can be different endpoint (or trimming slash only for `get/<bucket>/<object>` and not for `get_by_attributes///), config flag or something else. What do you think? Am I right that you want to do this for backward compatibility? Or in case the user suddenly wants to create objects with `FileName` attribute with leading slash?
dkirillov commented 2025-02-07 09:41:24 +00:00
Member
Copy link

If I already have such objects (with incorrect attributes) or if I will create such objects in future, I would like to be able to get them via http-gw somehow

If I already have such objects (with incorrect attributes) or if I will create such objects in future, I would like to be able to get them via http-gw somehow
👍 1
alexvanin force-pushed feature/174-add_slash_clipping_when_search_by_filename_attr_master from 6cdff1d1e2 to 466f3a9531 2025-02-07 12:06:31 +00:00 Compare
alexvanin dismissed dkirillov's review 2025-02-07 12:06:31 +00:00
Reason:

New commits pushed, approval review dismissed automatically according to repository settings

alexvanin commented 2025-02-07 12:09:46 +00:00
Owner
Copy link

@dkirillov #207

@dkirillov #207
alexvanin merged commit 466f3a9531 into master 2025-02-07 12:14:11 +00:00
alexvanin deleted branch feature/174-add_slash_clipping_when_search_by_filename_attr_master 2025-02-07 12:14:15 +00:00
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
P0

Highest

  
P1

High

  
P2

Medium

  
P3

Low

  
good first issue

Good for newcomers

  
Infrastructure

Infrastructure, Automation, CI/CD or DevOps related stuff

  
blocked

The issue or PR is blocked by something

  
bug

Something is not working

  
config

Configuration format update or breaking change

  
discussion

Talking about something in order to reach a decision or to exchange ideas

  
documentation

Documentation related

  
duplicate

This issue or pull request already exists

  
enhancement

New feature

  
go

Language features adoption

  
help wanted

Extra attention is needed

  
internal

   
invalid

Something is wrong

  
kludge

This is a kludge and we know it

  
observability

Related to metrics, tracing and logs

  
perfomance

Perfomance related

  
question

More information is needed

  
refactoring

Refactoring

  
wontfix

This won't be fixed

No labels
P0
P1
P2
P3
good first issue
Infrastructure
blocked
bug
config
discussion
documentation
duplicate
enhancement
go
help wanted
internal
invalid
kludge
observability
perfomance
question
refactoring
wontfix
Milestone
Clear milestone
No items
No milestone
v0.33.0
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
r.loginov
4 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: TrueCloudLab/frostfs-http-gw#205
No description provided.
Delete branch "r.loginov/frostfs-http-gw:feature/174-add_slash_clipping_when_search_by_filename_attr_master"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?