[#585] ListBuckets pagination support #591
No reviewers
TrueCloudLab/storage-services-developers
TrueCloudLab/storage-services-committers
Labels
No labels
P0
P1
P2
P3
good first issue
Infrastructure
blocked
bug
config
discussion
documentation
duplicate
enhancement
go
help wanted
internal
invalid
kludge
observability
perfomance
question
refactoring
wontfix
No milestone
No project
No assignees
2 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference: TrueCloudLab/frostfs-s3-gw#591
Loading…
Reference in a new issue
No description provided.
Delete branch "nzinkevich/frostfs-s3-gw:feature/list_buckets_pagination"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Add support for new ListBuckets query params for pagination. To test, use
--prefix
,--bucket-region
,--continuation-token
,--max-buckets
flags inaws cli
.fa66f9a135
to8466f66067
8466f66067
to9789edd296
9789edd296
tod11ace207a
d11ace207a
tofdbd3823bb
@ -299,3 +299,2 @@
func createBucket(hc *handlerContext, bktName string) *createBucketInfo {
box, key := createAccessBox(hc.t)
func createBucket(hc *handlerContext, prm bucketPrm) *createBucketInfo {
Consider params like:
or even don't change this function but create new one
@ -0,0 +15,4 @@
var (
own user.ID
res *ListBucketsResponse
ctx = r.Context()
nitpick: Please use
@ -0,0 +35,4 @@
own = result.Containers[0].Owner
}
res = &ListBucketsResponse{
Let's extract this to method/function
encodeListbuckets
or something like that (similar to object listing handlers).@ -0,0 +38,4 @@
res = &ListBucketsResponse{
Owner: Owner{
ID: own.String(),
DisplayName: own.String(),
Probably we can use
reqInfo.User
@ -0,0 +58,4 @@
}
func parseListBucketParams(r *http.Request) (params layer.ListBucketParams, err error) {
strMaxBuckets := r.URL.Query().Get("max-buckets")
It's better to use consts.
Probably even from middleware (
middleware.QueryMaxBuckets
)@ -0,0 +60,4 @@
func parseListBucketParams(r *http.Request) (params layer.ListBucketParams, err error) {
strMaxBuckets := r.URL.Query().Get("max-buckets")
if strMaxBuckets != "" {
if params.MaxBuckets, err = strconv.Atoi(r.URL.Query().Get("max-buckets")); err != nil {
We must check parsed value for negativeness and add test for that. Otherwise we will get panic, see:
@ -0,0 +120,4 @@
})
}
func listBuckets(hc *handlerContext, prm bucketPrm) ListBucketsResponse {
Consider using function signature like in
listObjectsV1
. Or at least create new param (don't reusebucketPrm
since a half of that fields aren't used in this function)@ -397,0 +406,4 @@
bktName string
query url.Values
box *accessbox.Box
key *keys.PrivateKey
Why do we need
box
andkey
in this struct if increateBucket
we always overwrite them?Remove
key
from params butbox
is needed forcreateBucketBase
and for now is not overwritten@ -397,0 +407,4 @@
query url.Values
box *accessbox.Box
key *keys.PrivateKey
body any
This should be more specific
@ -105,0 +110,4 @@
if info.CID.EncodeToString() == listParams.ContinuationToken {
tokenMet = true
}
if shouldSkipBucket(tokenMet, info, listParams) {
It seems before skip bucket we should list all and sort by bucket name
@ -79,3 +114,3 @@
objectErrors map[string]error
objectPutErrors map[string]error
containers map[string]*container.Container
containers containers
Why do we need this change?
I wanted to test list-bucket pagination, but
map[string]*container.Container
in mock has undetermined iterate order. When I call ListBucketHandler sequentially with continuation token, it's very likely that the test will failUPD: if we sorting containers now, it is not necessary anymore
@ -376,0 +387,4 @@
var result ListBucketResult
var err error
result.Containers, err = n.containerList(ctx, params)
Before listing we should check max-buckets. It it's 0 then just return empty result
fdbd3823bb
to19c771bb97
19c771bb97
to8fb6139774
8fb6139774
to601ff75d71
601ff75d71
toe0fa9e27d0
e0fa9e27d0
toa2a84b398d
View command line instructions
Checkout
From your project repository, check out a new branch and test the changes.