TrueCloudLab/frostfs-sdk-go
32
0
Fork 21
Code Issues 31 Pull requests 3 Releases Activity Actions

pool: Support average request time for ListContainerStream #343

Merged
fyrchik merged 2 commits from elebedeva/frostfs-sdk-go:fix/pool-container-list-stream into master 2025-03-12 10:29:06 +00:00
Conversation 1 Commits 2 Files changed 3 +31 -6
elebedeva commented 2025-03-06 03:21:46 +00:00
Member
Copy link

Close #338

Added average request time estimation for ListContainerStream.

Tested with frostfs-node VSCode debugger. Attached file with code used for testing.

❯ go run main.go
Put container cid:  AWm2eSB7YKjUSMyt1XD7f1PDFHXMJ3oCjuCW6Wk5js8u
Read container list:
cid:  AWm2eSB7YKjUSMyt1XD7f1PDFHXMJ3oCjuCW6Wk5js8u
Put avg:  7.826346ms
ListStream avg:  912.977µs

Done

Signed-off-by: Ekaterina Lebedeva ekaterina.lebedeva@yadro.com

Close #338 Added average request time estimation for `ListContainerStream`. Tested with frostfs-node VSCode debugger. Attached file with code used for testing. ``` ❯ go run main.go Put container cid: AWm2eSB7YKjUSMyt1XD7f1PDFHXMJ3oCjuCW6Wk5js8u Read container list: cid: AWm2eSB7YKjUSMyt1XD7f1PDFHXMJ3oCjuCW6Wk5js8u Put avg: 7.826346ms ListStream avg: 912.977µs Done ``` Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
pool_cnr_stat.go.txt
3.7 KiB
elebedeva added 1 commit 2025-03-06 03:21:47 +00:00
[#338] pool: Support avg request time for ListContainerStream
Some checks failed
Code generation / Generate proto (pull_request) Successful in 41s
Details
DCO / DCO (pull_request) Successful in 38s
Details
Tests and linters / Lint (pull_request) Successful in 1m48s
Details
Tests and linters / Tests (pull_request) Failing after 10m30s
Details
8378394ad6 
Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
elebedeva force-pushed fix/pool-container-list-stream from 8378394ad6 to c9949b6e47 2025-03-06 03:22:27 +00:00 Compare
fyrchik reviewed 2025-03-06 04:18:41 +00:00
pool/pool.go Outdated
@ -602,0 +606,4 @@
return ResListStream{
r: cnrRdr,
elapsedTimeCallback: func(elapsed time.Duration) {
c.incRequests(elapsed, methodObjectGet)
fyrchik commented 2025-03-06 04:18:41 +00:00
Owner
Copy link

methodObjectGet?

`methodObjectGet`?
elebedeva commented 2025-03-06 07:28:27 +00:00
Author
Member
Copy link

Fixed 🙈

Fixed 🙈
fyrchik marked this conversation as resolved
elebedeva force-pushed fix/pool-container-list-stream from c9949b6e47 to ee6ff031d0 2025-03-06 07:26:58 +00:00 Compare
elebedeva force-pushed fix/pool-container-list-stream from ee6ff031d0 to 208dbab7fc 2025-03-07 05:47:23 +00:00 Compare
elebedeva changed title from WIP: pool: Support average request time for ListContainerStream to pool: Support average request time for ListContainerStream 2025-03-07 05:58:40 +00:00
requested reviews from storage-core-committers, storage-core-developers, storage-services-committers, storage-services-developers, dkirillov 2025-03-07 05:59:06 +00:00
aarifullin approved these changes 2025-03-07 08:52:13 +00:00
Dismissed
dstepanov-yadro approved these changes 2025-03-07 09:00:16 +00:00
Dismissed
fyrchik approved these changes 2025-03-07 09:37:03 +00:00
Dismissed
fyrchik commented 2025-03-07 09:37:22 +00:00
Owner
Copy link

@dkirillov could you also take a look, please?

@dkirillov could you also take a look, please?
acid-ant approved these changes 2025-03-10 12:12:21 +00:00
Dismissed
dkirillov reviewed 2025-03-10 14:26:27 +00:00
dkirillov left a comment
Member
Copy link

Please, update MethodIndex.String() method

Please, update `MethodIndex.String()` method
👍 1
pool/pool.go Outdated
@ -576,1 +579,3 @@
return x.r.Iterate(f)
start := time.Now()
err := x.r.Iterate(f)
x.elapsedTimeCallback(time.Since(start))
dkirillov commented 2025-03-10 14:25:38 +00:00
Member
Copy link

I'm not sure if we can elapse time for all iteration.

Probably something like that be more aligned with ResListStream.Read

diff --git a/pool/pool.go b/pool/pool.go
index e4b8027..dab92a6 100644
--- a/pool/pool.go
+++ b/pool/pool.go
@@ -577,8 +577,13 @@ func (x *ResListStream) Read(buf []cid.ID) (int, error) {
 // Returns an error if container can't be read.
 func (x *ResListStream) Iterate(f func(cid.ID) bool) error {
        start := time.Now()
-       err := x.r.Iterate(f)
-       x.elapsedTimeCallback(time.Since(start))
+       err := x.r.Iterate(func(id cid.ID) bool {
+               x.elapsedTimeCallback(time.Since(start))
+               stop := f(id)
+               start = time.Now()
+               return stop
+       })
+
        return err
 }
I'm not sure if we can elapse time for all iteration. Probably something like that be more aligned with `ResListStream.Read` ```diff diff --git a/pool/pool.go b/pool/pool.go index e4b8027..dab92a6 100644 --- a/pool/pool.go +++ b/pool/pool.go @@ -577,8 +577,13 @@ func (x *ResListStream) Read(buf []cid.ID) (int, error) { // Returns an error if container can't be read. func (x *ResListStream) Iterate(f func(cid.ID) bool) error { start := time.Now() - err := x.r.Iterate(f) - x.elapsedTimeCallback(time.Since(start)) + err := x.r.Iterate(func(id cid.ID) bool { + x.elapsedTimeCallback(time.Since(start)) + stop := f(id) + start = time.Now() + return stop + }) + return err } ```
elebedeva commented 2025-03-10 16:23:00 +00:00
Author
Member
Copy link

Fixed, thanks!

Fixed, thanks!
pool/pool.go Outdated
@ -597,1 +603,3 @@
res, err := cl.ContainerListInit(ctx, cliPrm)
start := time.Now()
cnrRdr, err := cl.ContainerListInit(ctx, cliPrm)
c.incRequests(time.Since(start), methodContainerListStream)
dkirillov commented 2025-03-10 14:26:25 +00:00
Member
Copy link

Probably we should not consider this invocation (like for ObjectGetInit). But I have no strong opinion on this

Probably we should not consider this invocation (like for `ObjectGetInit`). But I have no strong opinion on this
elebedeva commented 2025-03-10 15:51:20 +00:00
Author
Member
Copy link

We do estimate elapsed time for ObjectRangeInit invocation

Lines 1046 to 1048 in 749b4e9
start := time.Now()
res, err := cl.ObjectRangeInit(ctx, cliPrm)
c.incRequests(time.Since(start), methodObjectRange)


but, as you mentioned, for ObjectGetInit we don't.

Line 960 in 749b4e9
rObj, err := cl.ObjectGetInit(ctx, cliPrm)


I believe the estimation should be the same for this kind of invocations. Which approach should be used here? @fyrchik

We do estimate elapsed time for `ObjectRangeInit` invocation https://git.frostfs.info/TrueCloudLab/frostfs-sdk-go/src/commit/749b4e9ab592d23a1327a77ee7df91e7efab9944/pool/client.go#L1046-L1048 but, as you mentioned, for `ObjectGetInit` we don't. https://git.frostfs.info/TrueCloudLab/frostfs-sdk-go/src/commit/749b4e9ab592d23a1327a77ee7df91e7efab9944/pool/client.go#L960 I believe the estimation should be the same for this kind of invocations. Which approach should be used here? @fyrchik
elebedeva force-pushed fix/pool-container-list-stream from 208dbab7fc to 9346bd1f7d 2025-03-10 15:26:55 +00:00 Compare
elebedeva dismissed aarifullin's review 2025-03-10 15:26:55 +00:00
Reason:

New commits pushed, approval review dismissed automatically according to repository settings

elebedeva dismissed dstepanov-yadro's review 2025-03-10 15:26:55 +00:00
Reason:

New commits pushed, approval review dismissed automatically according to repository settings

elebedeva dismissed fyrchik's review 2025-03-10 15:26:55 +00:00
Reason:

New commits pushed, approval review dismissed automatically according to repository settings

elebedeva dismissed acid-ant's review 2025-03-10 15:26:55 +00:00
Reason:

New commits pushed, approval review dismissed automatically according to repository settings

elebedeva force-pushed fix/pool-container-list-stream from 9346bd1f7d to 208b654241 2025-03-10 16:22:13 +00:00 Compare
elebedeva force-pushed fix/pool-container-list-stream from 208b654241 to a262a0038f 2025-03-10 16:25:06 +00:00 Compare
dkirillov approved these changes 2025-03-12 09:22:40 +00:00
dkirillov left a comment
Member
Copy link

LGTM

LGTM
requested reviews from storage-core-committers, storage-core-developers 2025-03-12 10:18:02 +00:00
achuprov approved these changes 2025-03-12 10:24:47 +00:00
fyrchik approved these changes 2025-03-12 10:28:58 +00:00
fyrchik merged commit a262a0038f into master 2025-03-12 10:29:06 +00:00
fyrchik referenced this pull request from a commit 2025-03-12 10:29:06 +00:00
[#343] pool: Fix Yoda condition
Sign in to join this conversation.
Reviewers
No reviewers
dkirillov
achuprov
fyrchik
Labels
Clear labels   
P0

Highest

  
P1

High

  
P2

Medium

  
P3

Low

  
good first issue

Good for newcomers

  
pool

Tasks related to client pool

  
Infrastructure

Infrastructure, Automation, CI/CD or DevOps related stuff

  
blocked

The issue or PR is blocked by something

  
bug

Something is not working

  
config

Configuration format update or breaking change

  
discussion

Talking about something in order to reach a decision or to exchange ideas

  
documentation

Documentation related

  
duplicate

This issue or pull request already exists

  
enhancement

New feature

  
go

Language features adoption

  
help wanted

Extra attention is needed

  
internal

   
invalid

Something is wrong

  
kludge

This is a kludge and we know it

  
observability

Related to metrics, tracing and logs

  
perfomance

Perfomance related

  
question

More information is needed

  
refactoring

Refactoring

  
wontfix

This won't be fixed

No labels
P0
P1
P2
P3
good first issue
pool
Infrastructure
blocked
bug
config
discussion
documentation
duplicate
enhancement
go
help wanted
internal
invalid
kludge
observability
perfomance
question
refactoring
wontfix
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
7 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: TrueCloudLab/frostfs-sdk-go#343
No description provided.
Delete branch "elebedeva/frostfs-sdk-go:fix/pool-container-list-stream"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?