metabase: Skip expired objects in ListWithCursor #1583

a-savchuk · 2024-12-25T19:30:24Z

a-savchuk commented

2024-12-25 19:30:24 +00:00

Before this change, the result of ListWithCursor included expired objects. This caused unnecessary attempts by the replicator to replicate these objects, leading to errors like this one below:

aug 06 11:56:12 node frostfs-node[4795]: 2024-08-06T11:56:12.630Z error replicator/process.go:78 could not replicate object {"component": "Object Replicator", "node": "03c5d517303e09d058371c03098becf5fc4089e301aa64d1e6f41602b11dd7b50f", "object": "H6esfqFaG3AcmdQnZ7UBKVQ38zpyUEt5nq4gAhHKDMFR/GQ3yizn5sCaTGrTGNzWgjQ6GADPRjCdANVRKyBiphHX2", "trace_id": "", "error": "(*putsvc.RemoteSender) could not send object: (*putsvc.remoteTarget) could not put single object to [/ip4/10.252.11.12/tcp/8080]: put single object via client: status: code = 1024 message = object has expired"}

Before this change, the result of `ListWithCursor` included expired objects. This caused unnecessary attempts by the replicator to replicate these objects, leading to errors like this one below: ``` aug 06 11:56:12 node frostfs-node[4795]: 2024-08-06T11:56:12.630Z error replicator/process.go:78 could not replicate object {"component": "Object Replicator", "node": "03c5d517303e09d058371c03098becf5fc4089e301aa64d1e6f41602b11dd7b50f", "object": "H6esfqFaG3AcmdQnZ7UBKVQ38zpyUEt5nq4gAhHKDMFR/GQ3yizn5sCaTGrTGNzWgjQ6GADPRjCdANVRKyBiphHX2", "trace_id": "", "error": "(*putsvc.RemoteSender) could not send object: (*putsvc.remoteTarget) could not put single object to [/ip4/10.252.11.12/tcp/8080]: put single object via client: status: code = 1024 message = object has expired"} ```

a-savchuk force-pushed do-not-list-expired-objects from c926fe4065 to 967072e52d

2024-12-25 19:32:53 +00:00

Compare

a-savchuk force-pushed do-not-list-expired-objects from 967072e52d to 2473257925

2024-12-25 19:38:32 +00:00

Compare

a-savchuk changed title from ~~WIP: metabase: Skip expired objects in ListWithCursor~~ to metabase: Skip expired objects in ListWithCursor

2024-12-25 19:50:36 +00:00

requested reviews from storage-core-committers, storage-core-developers

2024-12-25 19:50:37 +00:00

a-savchuk force-pushed do-not-list-expired-objects from 2473257925 to b49ce50daa

2024-12-25 19:59:55 +00:00

Compare

a-savchuk force-pushed do-not-list-expired-objects from b49ce50daa to dffae78eb2

2024-12-25 21:07:17 +00:00

Compare

fyrchik requested changes 2024-12-26 06:43:28 +00:00

Dismissed

fyrchik left a comment

Please, look at the already existing fix which was not in master for some reason 284b270c1f
It seems shorter and handles expired but locked objects (which thus should be available).
There should be tests for expired but locked objects too.

Please, look at the already existing fix which was not in master for some reason https://git.frostfs.info/fyrchik/frostfs-node/commit/284b270c1f1c5e17c6bd9a63f540d1dc14a633e9 It seems shorter and handles expired but locked objects (which thus should be available). There should be tests for expired but locked objects too.

pkg/local_object_storage/metabase/list.go Outdated

					
				@ -125,3 +128,3 @@

				}

				func (db *DB) listWithCursor(tx *bbolt.Tx, result []objectcore.Info, count int, cursor *Cursor) ([]objectcore.Info, *Cursor, error) {

				func (db *DB) listWithCursor(ctx context.Context, tx *bbolt.Tx, result []objectcore.Info, count int, cursor *Cursor) ([]objectcore.Info, *Cursor, error) {

fyrchik commented

2024-12-26 06:38:46 +00:00

Please, let's not have any ctx in unexported function of the metabase package.
It is unnecessary an complicates code.

Please, let's not have any `ctx` in unexported function of the metabase package. It is unnecessary an complicates code.

a-savchuk commented

2024-12-26 08:22:01 +00:00

Fixed

pkg/local_object_storage/metabase/list.go Outdated

					
				@ -225,6 +234,9 @@ func selectNFromBucket(bkt *bbolt.Bucket, // main bucket

					if !threshold {

						c.Seek(offset)

						// TODO(@a-savchuk): What'd be if the object with this offset was