TrueCloudLab/frostfs-s3-lifecycler
12
0
Fork 4
Code Issues 1 Pull requests 1 Projects Releases 1 Packages Wiki Activity Actions

feature/20-add_logs #21

Merged
alexvanin merged 6 commits from feature/20-add_logs into master 2024-09-25 08:37:57 +00:00
Conversation 0 Commits 6 Files changed 12 +241 -47
dkirillov commented 2024-09-23 14:39:47 +00:00
Member
Copy link

close #20

close #20
dkirillov self-assigned this 2024-09-23 14:39:47 +00:00
dkirillov added 3 commits 2024-09-23 14:39:48 +00:00 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
[#20] Support logging with tags
Some checks failed
/ DCO (pull_request) Successful in 48s
Details
/ Vulncheck (pull_request) Successful in 1m17s
Details
/ Builds (pull_request) Failing after 1m5s
Details
/ Lint (pull_request) Successful in 2m32s
Details
/ Tests (pull_request) Successful in 1m7s
Details
2f7a73ee36 
Allow to log additionally "tagged" log entries.

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
dkirillov requested review from storage-services-committers 2024-09-23 14:39:57 +00:00
dkirillov requested review from storage-services-developers 2024-09-23 14:39:59 +00:00
dkirillov added 1 commit 2024-09-23 15:53:40 +00:00
[#20] Fix go.sum
All checks were successful
/ DCO (pull_request) Successful in 42s
Details
/ Vulncheck (pull_request) Successful in 1m18s
Details
/ Builds (pull_request) Successful in 57s
Details
/ Lint (pull_request) Successful in 2m27s
Details
/ Tests (pull_request) Successful in 1m6s
Details
52c2ec525c 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
dkirillov reviewed 2024-09-24 06:55:21 +00:00
cmd/s3-lifecycler/logger.go Outdated
@ -77,0 +111,4 @@
msg := entry.Message[4:] // len("tag:")
for _, tag := range tags {
if msg == tag {
dkirillov commented 2024-09-24 06:55:21 +00:00
Author
Member
Copy link

Probably we can use strings.HasPrefix(msg, tag) to support logs with same tag but different message

  • tag:delete remove object
  • tag:delete remove version
Probably we can use `strings.HasPrefix(msg, tag)` to support logs with same tag but different message * `tag:delete remove object` * `tag:delete remove version`
alexvanin commented 2024-09-24 11:36:42 +00:00
Owner
Copy link

Can we check if strings.HasPrefix affect a burst of 10k-100k concurrent log messages?

Can we check if `strings.HasPrefix` affect a burst of 10k-100k concurrent log messages?
alexvanin approved these changes 2024-09-24 11:40:37 +00:00
Dismissed
alexvanin left a comment
Owner
Copy link

Seems very good. Let's sync logger sampling options and maybe run some performance tests on string comparasion and prefix check.

Seems very good. Let's sync logger sampling options and maybe run some performance tests on string comparasion and prefix check.
cmd/s3-lifecycler/logger.go Outdated
@ -59,3 +51,1 @@
c := zap.NewProductionConfig()
c.EncoderConfig.EncodeTime = zapcore.ISO8601TimeEncoder
c.Level = zap.NewAtomicLevelAt(lvl)
func openZapSinks(cfg zap.Config) (zapcore.WriteSyncer, zapcore.WriteSyncer, error) {
alexvanin commented 2024-09-24 11:38:43 +00:00
Owner
Copy link

nitpick: I would place this function before or after both new*Logger() function.

nitpick: I would place this function before or after both `new*Logger()` function.
cmd/s3-lifecycler/settings.go
@ -41,0 +40,4 @@
cfgLoggerDestination = "logger.destination"
cfgLoggerSamplingEnabled = "logger.sampling.enabled"
cfgLoggerSamplingInitial = "logger.sampling.initial"
cfgLoggerSamplingThereafter = "logger.sampling.thereafter"
alexvanin commented 2024-09-24 11:37:15 +00:00
Owner
Copy link

https://git.frostfs.info/TrueCloudLab/frostfs-s3-gw/pulls/495/files introduces interval as well, what you think about it?

https://git.frostfs.info/TrueCloudLab/frostfs-s3-gw/pulls/495/files introduces `interval` as well, what you think about it?
👍 1
alexvanin marked this conversation as resolved
dkirillov added 2 commits 2024-09-24 12:56:28 +00:00 
Now different log messages can use the same tag:
* `tag:delete remove object`
* `tag:delete remove version`

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
[#20] Add metric to see number of dropped logs
Some checks failed
/ DCO (pull_request) Successful in 39s
Details
/ Builds (pull_request) Successful in 1m14s
Details
/ Vulncheck (pull_request) Successful in 1m16s
Details
/ Lint (pull_request) Failing after 1m4s
Details
/ Tests (pull_request) Successful in 1m14s
Details
1827d365c7 
Add new metric frostfs_s3_lifecycler_statistic_dropped_logs
Also, configuration sampling interval is added

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
dkirillov force-pushed feature/20-add_logs from 1827d365c7 to 8b6d93c94d 2024-09-24 14:47:14 +00:00 Compare
alexvanin approved these changes 2024-09-24 14:57:56 +00:00
alexvanin merged commit 8b6d93c94d into master 2024-09-25 08:37:57 +00:00
alexvanin deleted branch feature/20-add_logs 2024-09-25 08:37:59 +00:00
Sign in to join this conversation.
Reviewers
No reviewers
alexvanin
TrueCloudLab/storage-services-developers
Labels
Clear labels
  
Infrastructure

Infrastructure, Automation, CI/CD or DevOps related stuff

  
blocked

The issue or PR is blocked by something

  
bug

Something is not working

  
config

Configuration format update or breaking change

  
discussion

Talking about something in order to reach a decision or to exchange ideas

  
documentation

Documentation related

  
duplicate

This issue or pull request already exists

  
enhancement

New feature

  
go

Language features adoption

  
help wanted

Extra attention is needed

  
internal

   
invalid

Something is wrong

  
kludge

This is a kludge and we know it

  
observability

Related to metrics, tracing and logs

  
perfomance

Perfomance related

  
question

More information is needed

  
refactoring

Refactoring

  
wontfix

This won't be fixed

No labels
Infrastructure
blocked
bug
config
discussion
documentation
duplicate
enhancement
go
help wanted
internal
invalid
kludge
observability
perfomance
question
refactoring
wontfix
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
dkirillov
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: TrueCloudLab/frostfs-s3-lifecycler#21
No description provided.
Delete branch "feature/20-add_logs"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?