TrueCloudLab/frostfs-s3-gw
18
3
Fork 20
Code Issues 29 Pull requests 8 Projects Releases 23 Packages Activity Actions

feature/498-make_query_tracing_more_detailed #564

Open
r.loginov wants to merge 6 commits from r.loginov/frostfs-s3-gw:feature/498-make_query_tracing_more_detailed into master
pull from: r.loginov/frostfs-s3-gw:feature/498-make_query_tracing_more_detailed
merge into: TrueCloudLab:master
Conversation 0 Commits 6 Files changed 37 +527 -122
r.loginov commented 2024-11-26 11:23:30 +00:00
Member
Copy link

As part of this PR, additional spans have been added to the following layers:

  • handler
  • layer
  • tree
  • frostfs
  • middleware (Auth, FrostFSIDValidation, PolicyCheck)

Suggestions for expanding or reducing the detail are welcome.

The question may arise why there are no attributes in span. I did not add attributes, it seems that the information needed is in the logs. After these changes, the trace id is present in all logs. Therefore, the necessary information can be taken from the logs and if you need to look at the trace, then use the trace id to view it in jaeger. If you think it's worth duplicating (or just adding) some attributes to the span, let me know.

A few examples of what it looks like in jaeger:
CreateBucket:
image

PutObject:
image

If you suddenly want to make other requests and see how it looks in jaeger, you can do this using frostfs-dev-env (jaeger is there).

close #498

As part of this PR, additional spans have been added to the following layers: - handler - layer - tree - frostfs - middleware (Auth, FrostFSIDValidation, PolicyCheck) Suggestions for expanding or reducing the detail are welcome. The question may arise why there are no attributes in span. I did not add attributes, it seems that the information needed is in the logs. After [these changes](https://git.frostfs.info/TrueCloudLab/frostfs-s3-gw/issues/501), the trace id is present in all logs. Therefore, the necessary information can be taken from the logs and if you need to look at the trace, then use the trace id to view it in jaeger. If you think it's worth duplicating (or just adding) some attributes to the span, let me know. A few examples of what it looks like in jaeger: CreateBucket: ![image](/attachments/751ba189-8f3e-417f-8c66-fff7c5f88a09) PutObject: ![image](/attachments/6ea70d1c-5a63-4b95-9f82-fb918724e312) If you suddenly want to make other requests and see how it looks in jaeger, you can do this using frostfs-dev-env (jaeger is there). close #498
image.png
186 KiB
image.png
192 KiB
r.loginov self-assigned this 2024-11-26 11:23:30 +00:00
r.loginov added 5 commits 2024-11-26 11:23:31 +00:00 
Signed-off-by: Roman Loginov <r.loginov@yadro.com>
Signed-off-by: Roman Loginov <r.loginov@yadro.com>
Signed-off-by: Roman Loginov <r.loginov@yadro.com>
Signed-off-by: Roman Loginov <r.loginov@yadro.com>
[#498] middleware: Add spans to detail the trace
All checks were successful
/ DCO (pull_request) Successful in 1m52s
Details
/ Vulncheck (pull_request) Successful in 2m2s
Details
/ Builds (pull_request) Successful in 1m59s
Details
/ Lint (pull_request) Successful in 3m8s
Details
/ Tests (pull_request) Successful in 2m2s
Details
1e001820ef 
Spans are added only to the following middleware:
* PolicyCheck
* Auth
* FrostfsIDValidation

This is done this way because these middleware are basic and
they interact with frostfs-storage.

Signed-off-by: Roman Loginov <r.loginov@yadro.com>
r.loginov requested review from alexvanin 2024-11-26 11:23:31 +00:00
r.loginov requested review from dkirillov 2024-11-26 11:23:31 +00:00
r.loginov removed review request for alexvanin 2024-11-26 11:24:07 +00:00
r.loginov removed review request for dkirillov 2024-11-26 11:24:08 +00:00
r.loginov requested review from alexvanin 2024-11-26 11:30:38 +00:00
r.loginov requested review from dkirillov 2024-11-26 11:30:38 +00:00
r.loginov requested review from pogpp 2024-11-26 11:30:38 +00:00
r.loginov requested review from mbiryukova 2024-11-26 11:30:39 +00:00
r.loginov requested review from nzinkevich 2024-11-26 11:30:39 +00:00
dkirillov approved these changes 2024-11-26 12:43:33 +00:00
Dismissed
dkirillov left a comment
Member
Copy link

Overall LGTM, see minor comments

Overall LGTM, see minor comments
api/handler/multipart_upload.go Outdated
@ -105,3 +106,3 @@
func (h *handler) CreateMultipartUploadHandler(w http.ResponseWriter, r *http.Request) {
ctx := r.Context()
ctx, span := tracing.StartSpanFromContext(r.Context(), "handler.CreateMultipartUploadHandler")
dkirillov commented 2024-11-26 12:38:13 +00:00
Member
Copy link

I suggest write "handler.CreateMultipartUpload" instead of "handler.CreateMultipartUploadHandler"

I suggest write `"handler.CreateMultipartUpload"` instead of `"handler.CreateMultipartUploadHandler"`
👍 1
dkirillov marked this conversation as resolved
api/handler/util.go Outdated
@ -59,3 +63,4 @@
}
func (h *handler) ResolveCID(ctx context.Context, bucket string) (cid.ID, error) {
ctx, span := tracing.StartSpanFromContext(ctx, "handler.ResolveCID")
dkirillov commented 2024-11-26 12:43:10 +00:00
Member
Copy link

Do we really need create new span if we don't do anything significant (the same for handler.ResolveBucket)?
I have no strong opinion on this case.

Do we really need create new span if we don't do anything significant (the same for `handler.ResolveBucket`)? I have no strong opinion on this case.
👍 1
r.loginov commented 2024-11-26 13:58:58 +00:00
Author
Member
Copy link

I doubted whether to create spans for them or not, because we really don't do anything significant.

The arguments for this were:

  • handler.ResolveCID: the ability to track access to the store when resolving using DNS. Although for this you also need to add a span to frostfs.SystemDNS.
  • handler.ResolveBucket: the ability to track access to the storage when receiving information about the baguette.

I really think it would be better to just add a span to frostfs.SystemDNS, and remove the spans from handler.ResolveCID and handler.ResolveBucket.

I doubted whether to create spans for them or not, because we really don't do anything significant. The arguments for this were: - `handler.ResolveCID`: the ability to track access to the store when resolving using DNS. Although for this you also need to add a span to [frostfs.SystemDNS](https://git.frostfs.info/TrueCloudLab/frostfs-s3-gw/src/branch/master/internal/frostfs/frostfs.go#L453). - `handler.ResolveBucket`: the ability to track access to the storage when receiving information about the baguette. I really think it would be better to just add a span to `frostfs.SystemDNS`, and remove the spans from `handler.ResolveCID` and `handler.ResolveBucket`.
dkirillov marked this conversation as resolved
api/middleware/auth.go Outdated
@ -50,2 +52,2 @@
ctx := r.Context()
reqInfo := GetReqInfo(ctx)
reqCtx := r.Context()
ctx, span := tracing.StartSpanFromContext(reqCtx, "middlware.Auth")
dkirillov commented 2024-11-26 12:31:44 +00:00
Member
Copy link

Typo, must be middleware.Auth

Typo, must be `middleware.Auth`
👍 1
dkirillov marked this conversation as resolved
r.loginov force-pushed feature/498-make_query_tracing_more_detailed from 1e001820ef to d386476ae4 2024-11-26 14:47:05 +00:00 Compare
r.loginov dismissed dkirillov's review 2024-11-26 14:47:06 +00:00
Reason:

New commits pushed, approval review dismissed automatically according to repository settings

r.loginov force-pushed feature/498-make_query_tracing_more_detailed from d386476ae4 to 5c48bd136c 2024-11-26 14:54:52 +00:00 Compare
alexvanin reviewed 2024-11-27 08:22:25 +00:00
api/auth/center.go Outdated
@ -121,3 +121,3 @@
}
func (c *Center) Authenticate(r *http.Request) (*middleware.Box, error) {
func (c *Center) Authenticate(ctx context.Context, r *http.Request) (*middleware.Box, error) {
alexvanin commented 2024-11-27 08:22:25 +00:00
Owner
Copy link

After this change there are two sources of the context for developer: ctx and r.Context(). Is there any explicit reason to introduce ctx parameter?

For example, in #525 all handler attributes reused context from http.Request.

After this change there are two sources of the context for developer: `ctx` and `r.Context()`. Is there any explicit reason to introduce `ctx` parameter? For example, in #525 all handler attributes reused context from `http.Request`.
r.loginov commented 2024-11-27 08:57:33 +00:00
Author
Member
Copy link

Yes, there is a reason. The 'ctx` context additionally contains information about the current span. The hierarchy of spans is formed precisely due to the context in which there is information about the current span. This context needs to be thrown further before contacting the network so that the span in the network layers would understand who their parent span is.

For example (pay attention to the nesting of spans in middleware.Auth):
This is how backs look using ctx:
image
This is how backs look using r.Context:
image

However, a thought may arise - why don't we add information about span to the context of the request? The problem is that then, before completing the work of a specific middleware, we will need to remove the span information from the context, otherwise we will have the following picture (pay attention to the nesting):image

Yes, there is a reason. The 'ctx` context additionally contains information about the current span. The hierarchy of spans is formed precisely due to the context in which there is information about the current span. This context needs to be thrown further before contacting the network so that the span in the network layers would understand who their parent span is. For example (pay attention to the nesting of spans in `middleware.Auth`): This is how backs look using `ctx`: ![image](/attachments/23219347-e956-4c68-a3ad-507ec1c7b9db) This is how backs look using `r.Context`: ![image](/attachments/848f6ce4-342b-471a-ab2c-71f2bd5aaabc) However, a thought may arise - why don't we add information about span to the context of the request? The problem is that then, before completing the work of a specific middleware, we will need to remove the span information from the context, otherwise we will have the following picture (pay attention to the nesting):![image](/attachments/aa965899-7465-4447-8522-0bad6fa994c4)
image.png
54 KiB
image.png
54 KiB
image.png
80 KiB
alexvanin commented 2024-11-27 13:18:04 +00:00
Owner
Copy link

Oh I see, thank you for detailed comment on that.

Oh I see, thank you for detailed comment on that.
alexvanin marked this conversation as resolved
r.loginov added 1 commit 2024-11-28 08:15:43 +00:00
[OBJECT-13304] Update frostfs-observability version
Some checks failed
/ DCO (pull_request) Failing after 3m6s
Details
/ Vulncheck (pull_request) Successful in 4m10s
Details
/ Builds (pull_request) Successful in 4m25s
Details
/ Lint (pull_request) Successful in 5m53s
Details
/ Tests (pull_request) Successful in 3m35s
Details
63770bda39 
The new version of frostfs-observability has
improved the detail of tracing low-level rpc
calls by adding send and receive events.

Signed-off-by: Roman Loginov <r.loginov@yadro.com>
r.loginov force-pushed feature/498-make_query_tracing_more_detailed from 63770bda39 to 374018c707 2024-11-28 08:20:51 +00:00 Compare
dkirillov approved these changes 2024-11-28 13:31:59 +00:00
Dismissed
r.loginov force-pushed feature/498-make_query_tracing_more_detailed from 374018c707 to ef933df556 2024-11-29 09:41:56 +00:00 Compare
r.loginov dismissed dkirillov's review 2024-11-29 09:41:56 +00:00
Reason:

New commits pushed, approval review dismissed automatically according to repository settings

mbiryukova reviewed 2024-12-02 01:14:49 +00:00
api/middleware/auth.go Outdated
@ -97,3 +107,3 @@
if err = validateBearerToken(frostfsID, bd.Gate.BearerToken); err != nil {
reqLogOrDefault(ctx, log).Error(logs.FrostfsIDValidationFailed, zap.Error(err))
if _, wrErr := WriteErrorResponse(w, GetReqInfo(r.Context()), err); wrErr != nil {
if _, wrErr := WriteErrorResponse(w, GetReqInfo(ctx), err); wrErr != nil {
mbiryukova commented 2024-12-02 01:12:14 +00:00
Member
Copy link

Shouldn't r.Context() be used here and in policy check? Like in Auth middleware

Shouldn't `r.Context()` be used here and in policy check? Like in Auth middleware
r.loginov commented 2024-12-02 09:11:29 +00:00
Author
Member
Copy link

In middleware.auth context is eventually modified and added to http.Request, which is why there should be a separation between the use of the request context and the container with information about span. In the case of middleware.Frost fs ID Validation context is not saved further in http.Request and therefore it will not affect further spans in any way. In the middleware.PolicyCheck the situation is the same.

In `middleware.auth` context is eventually modified and [added](https://git.frostfs.info/TrueCloudLab/frostfs-s3-gw/src/branch/master/api/middleware/auth.go#L77) to `http.Request`, which is why there should be a separation between the use of the request context and the container with information about span. In the case of `middleware.Frost fs ID Validation` context is not saved further in `http.Request` and therefore it will not affect further spans in any way. In the `middleware.PolicyCheck` the situation is the same.
👍 1
r.loginov marked this conversation as resolved
pkg/service/tree/tree.go Outdated
@ -709,10 +731,16 @@ func (c *Tree) PutObjectTagging(ctx context.Context, bktInfo *data.BucketInfo, o
}
func (c *Tree) DeleteObjectTagging(ctx context.Context, bktInfo *data.BucketInfo, objVersion *data.NodeVersion) error {
ctx, span := tracing.StartSpanFromContext(ctx, "tree.GetObjectTagging")
mbiryukova commented 2024-12-02 00:56:25 +00:00
Member
Copy link

tree.DeleteObjectTagging

`tree.DeleteObjectTagging`
👍 1
r.loginov marked this conversation as resolved
mbiryukova reviewed 2024-12-02 01:57:12 +00:00
api/middleware/policy.go Outdated
@ -102,3 +106,2 @@
func policyCheck(r *http.Request, cfg PolicyConfig) error {
reqInfo := GetReqInfo(r.Context())
func policyCheck(ctx context.Context, r *http.Request, cfg PolicyConfig) error {
mbiryukova commented 2024-12-02 01:57:12 +00:00
Member
Copy link

Maybe we should also use span context in determineResourceTags?

Maybe we should also use span context in `determineResourceTags`?
👍 1
r.loginov commented 2024-12-02 09:12:24 +00:00
Author
Member
Copy link

Yes, you are right, because there may be requests in storage.

Yes, you are right, because there may be requests in storage.
r.loginov marked this conversation as resolved
r.loginov force-pushed feature/498-make_query_tracing_more_detailed from ef933df556 to 35a592438b 2024-12-02 09:12:58 +00:00 Compare
mbiryukova approved these changes 2024-12-02 09:33:18 +00:00
Dismissed
nzinkevich approved these changes 2024-12-02 12:49:58 +00:00
Dismissed
dkirillov approved these changes 2024-12-02 13:34:50 +00:00
Dismissed
alexvanin added this to the v0.33.0 milestone 2024-12-11 07:40:00 +00:00
r.loginov force-pushed feature/498-make_query_tracing_more_detailed from 35a592438b to 43fd7df66a 2024-12-11 13:25:09 +00:00 Compare
r.loginov dismissed mbiryukova's review 2024-12-11 13:25:10 +00:00
Reason:

New commits pushed, approval review dismissed automatically according to repository settings

r.loginov dismissed nzinkevich's review 2024-12-11 13:25:10 +00:00
Reason:

New commits pushed, approval review dismissed automatically according to repository settings

r.loginov dismissed dkirillov's review 2024-12-11 13:25:10 +00:00
Reason:

New commits pushed, approval review dismissed automatically according to repository settings

r.loginov force-pushed feature/498-make_query_tracing_more_detailed from 43fd7df66a to ca6fbb53ef 2024-12-11 13:34:05 +00:00 Compare
r.loginov force-pushed feature/498-make_query_tracing_more_detailed from ca6fbb53ef to 666a1ef3ed 2024-12-13 11:36:27 +00:00 Compare
dkirillov approved these changes 2024-12-13 12:09:03 +00:00
All checks were successful
/ DCO (pull_request) Successful in 3m51s
Required
Details
/ Vulncheck (pull_request) Successful in 4m1s
Required
Details
/ Builds (pull_request) Successful in 2m8s
Required
Details
/ Lint (pull_request) Successful in 3m25s
Required
Details
/ Tests (pull_request) Successful in 2m33s
Required
Details
This pull request has changes conflicting with the target branch.
  • go.mod
  • go.sum
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u feature/498-make_query_tracing_more_detailed:r.loginov-feature/498-make_query_tracing_more_detailed
git checkout r.loginov-feature/498-make_query_tracing_more_detailed
Sign in to join this conversation.
Reviewers
No reviewers
alexvanin
pogpp
dkirillov
Labels
Clear labels   
P0

Highest

  
P1

High

  
P2

Medium

  
P3

Low

  
good first issue

Good for newcomers

  
Infrastructure

Infrastructure, Automation, CI/CD or DevOps related stuff

  
blocked

The issue or PR is blocked by something

  
bug

Something is not working

  
config

Configuration format update or breaking change

  
discussion

Talking about something in order to reach a decision or to exchange ideas

  
documentation

Documentation related

  
duplicate

This issue or pull request already exists

  
enhancement

New feature

  
go

Language features adoption

  
help wanted

Extra attention is needed

  
internal

   
invalid

Something is wrong

  
kludge

This is a kludge and we know it

  
observability

Related to metrics, tracing and logs

  
perfomance

Perfomance related

  
question

More information is needed

  
refactoring

Refactoring

  
wontfix

This won't be fixed

No labels
P0
P1
P2
P3
good first issue
Infrastructure
blocked
bug
config
discussion
documentation
duplicate
enhancement
go
help wanted
internal
invalid
kludge
observability
perfomance
question
refactoring
wontfix
Milestone
Clear milestone
No items
No milestone
v0.33.0
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
r.loginov
5 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: TrueCloudLab/frostfs-s3-gw#564
No description provided.
Delete branch "r.loginov/frostfs-s3-gw:feature/498-make_query_tracing_more_detailed"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?