Sorting in listing doesn't work as expected on specific keys #561

New issue

Open

opened 2024-11-25 12:18:28 +00:00 by dkirillov · 0 comments

dkirillov commented

2024-11-25 12:18:28 +00:00

Member

According to spec https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectsV2.html listing should be lexicographical sorted.

Expected Behavior

Listing order always correct

Current Behavior

Listing order is not always correct. For example we have keys

bar baz/foo
bar/baz

From storage node (during sorted getSubTree) we gets them in the following order

bar/baz
bar baz/foo

That because of bar < bar baz

Possible Solution

Somehow change storage node to sort objects on several levels
List and cache node on one level to know for sure the order

Steps to Reproduce (for bugs)

Run test in handler package:

func TestListObjectsV2Sorting(t *testing.T) {
	tc := prepareHandlerContext(t)

	bktName := "bucket-for-listing"

	expectedObj := "bar baz/foo"
	otherObjPrefix := "bar/baz"

	bktInfo, _ := createBucketAndObject(tc, bktName, expectedObj)

	for i := 0; i < 1001; i++ {
		createTestObject(tc, bktInfo, otherObjPrefix+strconv.Itoa(i), encryption.Params{})
	}

	listV2Response2 := listObjectsV2(tc, bktName, "", "", "", "", -1)
	require.Equal(t, expectedObj, listV2Response2.Contents[0].Key)
}

Context

This leads to fatal error: Error during pagination: The same next token was received twice: {'ContinuationToken': 'H4hEvppC6XJRFafLriNy99Y8vaFkqtAKqLt26xecjqjb'} in aws s3 rm command

Regression

Probably yes

Your Environment

Version used: v0.29.3, v0.31.0-rc.6
Server setup and configuration: devenv

According to spec https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectsV2.html listing should be lexicographical sorted. ## Expected Behavior Listing order always correct ## Current Behavior Listing order is not always correct. For example we have keys * `bar baz/foo` * `bar/baz` From storage node (during sorted `getSubTree`) we gets them in the following order * `bar/baz` * `bar baz/foo` That because of `bar` < `bar baz` ## Possible Solution * Somehow change storage node to sort objects on several levels * List and cache node on one level to know for sure the order ## Steps to Reproduce (for bugs) Run test in `handler` package: ``` golang func TestListObjectsV2Sorting(t *testing.T) { tc := prepareHandlerContext(t) bktName := "bucket-for-listing" expectedObj := "bar baz/foo" otherObjPrefix := "bar/baz" bktInfo, _ := createBucketAndObject(tc, bktName, expectedObj) for i := 0; i < 1001; i++ { createTestObject(tc, bktInfo, otherObjPrefix+strconv.Itoa(i), encryption.Params{}) } listV2Response2 := listObjectsV2(tc, bktName, "", "", "", "", -1) require.Equal(t, expectedObj, listV2Response2.Contents[0].Key) } ``` ## Context This leads to `fatal error: Error during pagination: The same next token was received twice: {'ContinuationToken': 'H4hEvppC6XJRFafLriNy99Y8vaFkqtAKqLt26xecjqjb'}` in `aws s3 rm` command ## Regression Probably yes ## Your Environment  * Version used: `v0.29.3`, `v0.31.0-rc.6` * Server setup and configuration: devenv