TrueCloudLab/frostfs-s3-gw
18
3
Fork 20
Code Issues 29 Pull requests 7 Projects Releases 23 Packages 1 Activity Actions

[#502] Add Dropped logs (by sampling) metric #557

Merged
alexvanin merged 1 commit from pogpp/frostfs-s3-gw:feature/502_sampling_metric into master 2024-12-03 14:20:58 +00:00
Conversation 0 Commits 1 Files changed 5 +75 -21
pogpp commented 2024-11-20 12:06:05 +00:00
Member
Copy link

close #502
Signed-off-by: Pavel Pogodaev p.pogodaev@yadro.com

close #502 Signed-off-by: Pavel Pogodaev <p.pogodaev@yadro.com>
pogpp added 1 commit 2024-11-20 12:06:06 +00:00
[#502] Add Dropped logs (by sampling) metric
All checks were successful
/ DCO (pull_request) Successful in 2m26s
Details
/ Vulncheck (pull_request) Successful in 2m27s
Details
/ Builds (pull_request) Successful in 2m36s
Details
/ Lint (pull_request) Successful in 2m30s
Details
/ Tests (pull_request) Successful in 2m9s
Details
80afebff7a 
Signed-off-by: Pavel Pogodaev <p.pogodaev@yadro.com>
pogpp requested review from alexvanin 2024-11-20 12:06:33 +00:00
pogpp requested review from dkirillov 2024-11-20 12:06:33 +00:00
pogpp requested review from mbiryukova 2024-11-20 12:06:35 +00:00
pogpp requested review from r.loginov 2024-11-20 12:06:36 +00:00
pogpp requested review from nzinkevich 2024-11-20 12:06:37 +00:00
pogpp force-pushed feature/502_sampling_metric from 80afebff7a to b5eb8d185d 2024-11-20 12:07:21 +00:00 Compare
pogpp self-assigned this 2024-11-20 12:10:47 +00:00
dkirillov reviewed 2024-11-21 06:57:56 +00:00
metrics/app.go Outdated
@ -43,2 +43,4 @@
}
func (m *AppMetrics) DroppedLogsInc() {
m.gate.Logs.DroppedLogsInc()
dkirillov commented 2024-11-21 06:57:53 +00:00
Member
Copy link

Please, use approach as for UsersAPIStats or APIStatMetrics or StateMetrics

Please, use approach as for `UsersAPIStats` or `APIStatMetrics` or `StateMetrics`
👍 1
pogpp commented 2024-11-21 12:56:46 +00:00
Author
Member
Copy link

Now it's similar to StateMetrics

Now it's similar to `StateMetrics`
dkirillov commented 2024-11-25 14:10:25 +00:00
Member
Copy link

I'm not sure if this

Lines 45 to 47 in pogpp/frostfs-s3-gw@99ff86a
func (m *AppMetrics) DroppedLogsInc() {
m.gate.Logs.DroppedLogsInc()
}


is similar to

Lines 59 to 65 in pogpp/frostfs-s3-gw@99ff86a
func (m *AppMetrics) State() *StateMetrics {
if !m.isEnabled() {
return nil
}
return m.gate.State
}

I'm not sure if this https://git.frostfs.info/pogpp/frostfs-s3-gw/src/commit/99ff86a0d1fa7697d94b21dd8e89d26e1fb48e9e/metrics/app.go#L45-L47 is similar to https://git.frostfs.info/pogpp/frostfs-s3-gw/src/commit/99ff86a0d1fa7697d94b21dd8e89d26e1fb48e9e/metrics/app.go#L59-L65
👍 1
dkirillov marked this conversation as resolved
pogpp force-pushed feature/502_sampling_metric from b5eb8d185d to 26e5be8918 2024-11-21 10:29:36 +00:00 Compare
pogpp force-pushed feature/502_sampling_metric from 26e5be8918 to 99ff86a0d1 2024-11-21 12:57:54 +00:00 Compare
dkirillov reviewed 2024-11-25 14:02:43 +00:00
cmd/s3-gw/app.go Outdated
@ -524,2 +525,4 @@
}
func (a *App) initLogger() {
coreWithContext := applyZapCoreMiddlewares(a.log.Core(), a.cfg, a.metrics)
dkirillov commented 2024-11-25 14:02:43 +00:00
Member
Copy link

I don't understand why do we applying new actually the same middleware that already has been added here

Line 1173 in pogpp/frostfs-s3-gw@99ff86a
func samplingEnabling(v *viper.Viper, core zapcore.Core) zapcore.Core {

I don't understand why do we applying new actually the same middleware that already has been added here https://git.frostfs.info/pogpp/frostfs-s3-gw/src/commit/99ff86a0d1fa7697d94b21dd8e89d26e1fb48e9e/cmd/s3-gw/app_settings.go#L1173
pogpp commented 2024-11-27 09:55:25 +00:00
Author
Member
Copy link

Init sequence of logger and metrics was sub-optimal, fixed

Init sequence of logger and metrics was sub-optimal, fixed
dkirillov marked this conversation as resolved
dkirillov reviewed 2024-11-25 14:07:33 +00:00
metrics/desc.go Outdated
@ -93,6 +93,13 @@ var appMetricsDesc = map[string]map[string]Description{
},
},
statisticSubsystem: {
droppedLogs: Description{
dkirillov commented 2024-11-25 14:07:33 +00:00
Member
Copy link

I would suggest (if we use statistic subsystem) add new metric to APIStatMetrics

I would suggest (if we use `statistic` subsystem) add new metric to `APIStatMetrics`
👍 1
dkirillov marked this conversation as resolved
dkirillov requested changes 2024-11-25 14:13:21 +00:00
Dismissed
dkirillov left a comment
Member
Copy link

#557 (comment)

https://git.frostfs.info/TrueCloudLab/frostfs-s3-gw/pulls/557#issuecomment-59074
pogpp force-pushed feature/502_sampling_metric from 99ff86a0d1 to e58ac06cc4 2024-11-27 10:29:57 +00:00 Compare
pogpp requested review from dkirillov 2024-11-27 10:30:07 +00:00
pogpp force-pushed feature/502_sampling_metric from e58ac06cc4 to e38442675e 2024-11-27 10:42:34 +00:00 Compare
pogpp force-pushed feature/502_sampling_metric from e38442675e to e1595e385b 2024-11-28 08:27:05 +00:00 Compare
dkirillov requested changes 2024-11-28 14:26:31 +00:00
Dismissed
dkirillov left a comment
Member
Copy link

Please consider something like:

diff --git a/cmd/s3-gw/app.go b/cmd/s3-gw/app.go
index cfddda0b..51b32189 100644
--- a/cmd/s3-gw/app.go
+++ b/cmd/s3-gw/app.go
@@ -83,11 +83,17 @@ type (
 		bucketResolver *resolver.BucketResolver
 		services       []*Service
 		settings       *appSettings
+		loggerSettings *loggerSettings
 
 		webDone chan struct{}
 		wrkDone chan struct{}
 	}
 
+	loggerSettings struct {
+		mu         sync.RWMutex
+		appMetrics *metrics.AppMetrics
+	}
+
 	appSettings struct {
 		logLevel            zap.AtomicLevel
 		httpLogging         s3middleware.LogHTTPConfig
@@ -132,7 +138,26 @@ type (
 	}
 )
 
-func newApp(ctx context.Context, log *Logger, v *viper.Viper) *App {
+func (s *loggerSettings) DroppedLogsInc() {
+	s.mu.RLock()
+	defer s.mu.RUnlock()
+
+	if s.appMetrics != nil {
+		s.appMetrics.Statistic().DroppedLogsInc()
+	}
+}
+
+func (s *loggerSettings) setMetrics(appMetrics *metrics.AppMetrics) {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+
+	s.appMetrics = appMetrics
+}
+
+func newApp(ctx context.Context, v *viper.Viper) *App {
+	logSettings := &loggerSettings{}
+	log := pickLogger(v, logSettings)
+
 	settings := newAppSettings(log, v)
 
 	objPool, treePool, key := getPools(ctx, log.logger, v, settings.dialerSource)
@@ -147,7 +172,8 @@ func newApp(ctx context.Context, log *Logger, v *viper.Viper) *App {
 		webDone: make(chan struct{}, 1),
 		wrkDone: make(chan struct{}, 1),
 
-		settings: settings,
+		settings:       settings,
+		loggerSettings: logSettings,
 	}
 
 	app.init(ctx)
@@ -163,7 +189,6 @@ func (a *App) init(ctx context.Context) {
 	a.initPolicyStorage(ctx)
 	a.initAPI(ctx)
 	a.initMetrics()
-	a.initLogger()
 	a.initServers(ctx)
 	a.initTracing(ctx)
 }
@@ -522,11 +547,7 @@ func (a *App) initMetrics() {
 
 	a.metrics = metrics.NewAppMetrics(cfg)
 	a.metrics.State().SetHealth(metrics.HealthStatusStarting)
-}
-
-func (a *App) initLogger() {
-	coreWithContext := applyZapCoreMiddlewares(a.log.Core(), a.cfg, a.metrics)
-	a.log = zap.New(coreWithContext, zap.AddStacktrace(zap.NewAtomicLevelAt(zap.FatalLevel)))
+	a.loggerSettings.setMetrics(a.metrics)
 }
 
 func (a *App) initFrostfsID(ctx context.Context) {
diff --git a/cmd/s3-gw/app_settings.go b/cmd/s3-gw/app_settings.go
index f6fd931a..f33de2e8 100644
--- a/cmd/s3-gw/app_settings.go
+++ b/cmd/s3-gw/app_settings.go
@@ -18,7 +18,6 @@ import (
 	"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/logs"
 	internalnet "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/net"
 	"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/version"
-	"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/metrics"
 	"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/netmap"
 	"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/pool"
 	"git.frostfs.info/TrueCloudLab/zapjournald"
@@ -1095,7 +1094,11 @@ func mergeConfig(v *viper.Viper, fileName string) error {
 	return v.MergeConfig(cfgFile)
 }
 
-func pickLogger(v *viper.Viper) *Logger {
+type LoggerAppSettings interface {
+	DroppedLogsInc()
+}
+
+func pickLogger(v *viper.Viper, settings LoggerAppSettings) *Logger {
 	lvl, err := getLogLevel(v)
 	if err != nil {
 		panic(err)
@@ -1105,9 +1108,9 @@ func pickLogger(v *viper.Viper) *Logger {
 
 	switch dest {
 	case destinationStdout:
-		return newStdoutLogger(lvl)
+		return newStdoutLogger(v, lvl, settings)
 	case destinationJournald:
-		return newJournaldLogger(lvl)
+		return newJournaldLogger(v, lvl, settings)
 	default:
 		panic(fmt.Sprintf("wrong destination for logger: %s", dest))
 	}
@@ -1127,19 +1130,21 @@ func pickLogger(v *viper.Viper) *Logger {
 // Logger records a stack trace for all messages at or above fatal level.
 //
 // See also zapcore.Level, zap.NewProductionConfig, zap.AddStacktrace.
-func newStdoutLogger(lvl zapcore.Level) *Logger {
+func newStdoutLogger(v *viper.Viper, lvl zapcore.Level, settings LoggerAppSettings) *Logger {
 	stdout := zapcore.AddSync(os.Stderr)
 	level := zap.NewAtomicLevelAt(lvl)
 
 	consoleOutCore := zapcore.NewCore(newLogEncoder(), stdout, level)
 
+	consoleOutCore = applyZapCoreMiddlewares(consoleOutCore, v, settings)
+
 	return &Logger{
 		logger: zap.New(consoleOutCore, zap.AddStacktrace(zap.NewAtomicLevelAt(zap.FatalLevel))),
 		lvl:    level,
 	}
 }
 
-func newJournaldLogger(lvl zapcore.Level) *Logger {
+func newJournaldLogger(v *viper.Viper, lvl zapcore.Level, settings LoggerAppSettings) *Logger {
 	level := zap.NewAtomicLevelAt(lvl)
 
 	encoder := zapjournald.NewPartialEncoder(newLogEncoder(), zapjournald.SyslogFields)
@@ -1151,6 +1156,8 @@ func newJournaldLogger(lvl zapcore.Level) *Logger {
 		zapjournald.SyslogPid(),
 	})
 
+	coreWithContext = applyZapCoreMiddlewares(coreWithContext, v, settings)
+
 	l := zap.New(coreWithContext, zap.AddStacktrace(zap.NewAtomicLevelAt(zap.FatalLevel)))
 
 	return &Logger{
@@ -1166,6 +1173,22 @@ func newLogEncoder() zapcore.Encoder {
 	return zapcore.NewConsoleEncoder(c)
 }
 
+func applyZapCoreMiddlewares(core zapcore.Core, v *viper.Viper, settings LoggerAppSettings) zapcore.Core {
+	if v.GetBool(cfgLoggerSamplingEnabled) {
+		core = zapcore.NewSamplerWithOptions(core,
+			v.GetDuration(cfgLoggerSamplingInterval),
+			v.GetInt(cfgLoggerSamplingInitial),
+			v.GetInt(cfgLoggerSamplingThereafter),
+			zapcore.SamplerHook(func(_ zapcore.Entry, dec zapcore.SamplingDecision) {
+				if dec&zapcore.LogDropped > 0 {
+					settings.DroppedLogsInc()
+				}
+			}))
+	}
+
+	return core
+}
+
 func getLogLevel(v *viper.Viper) (zapcore.Level, error) {
 	var lvl zapcore.Level
 	lvlStr := v.GetString(cfgLoggerLevel)
@@ -1200,19 +1223,3 @@ LOOP:
 	}
 	return validDomains
 }
-
-func applyZapCoreMiddlewares(core zapcore.Core, v *viper.Viper, appMetrics *metrics.AppMetrics) zapcore.Core {
-	if v.GetBool(cfgLoggerSamplingEnabled) {
-		core = zapcore.NewSamplerWithOptions(core,
-			v.GetDuration(cfgLoggerSamplingInterval),
-			v.GetInt(cfgLoggerSamplingInitial),
-			v.GetInt(cfgLoggerSamplingThereafter),
-			zapcore.SamplerHook(func(_ zapcore.Entry, dec zapcore.SamplingDecision) {
-				if dec&zapcore.LogDropped > 0 {
-					appMetrics.DroppedLogsInc()
-				}
-			}))
-	}
-
-	return core
-}
diff --git a/cmd/s3-gw/main.go b/cmd/s3-gw/main.go
index f76bced3..c6e3b97b 100644
--- a/cmd/s3-gw/main.go
+++ b/cmd/s3-gw/main.go
@@ -10,9 +10,8 @@ func main() {
 	g, _ := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)
 
 	v := newSettings()
-	l := pickLogger(v)
 
-	a := newApp(g, l, v)
+	a := newApp(g, v)
 
 	go a.Serve(g)
 
diff --git a/metrics/app.go b/metrics/app.go
index 4898d727..1a2ef077 100644
--- a/metrics/app.go
+++ b/metrics/app.go
@@ -42,14 +42,6 @@ func NewAppMetrics(cfg AppMetricsConfig) *AppMetrics {
 	}
 }
 
-func (m *AppMetrics) DroppedLogsInc() {
-	if !m.isEnabled() {
-		return
-	}
-
-	m.gate.Stats.DroppedLogsInc()
-}
-
 func (m *AppMetrics) SetEnabled(enabled bool) {
 	if !enabled {
 		m.logger.Warn(logs.MetricsAreDisabled)
Please consider something like: ```diff diff --git a/cmd/s3-gw/app.go b/cmd/s3-gw/app.go index cfddda0b..51b32189 100644 --- a/cmd/s3-gw/app.go +++ b/cmd/s3-gw/app.go @@ -83,11 +83,17 @@ type ( bucketResolver *resolver.BucketResolver services []*Service settings *appSettings + loggerSettings *loggerSettings webDone chan struct{} wrkDone chan struct{} } + loggerSettings struct { + mu sync.RWMutex + appMetrics *metrics.AppMetrics + } + appSettings struct { logLevel zap.AtomicLevel httpLogging s3middleware.LogHTTPConfig @@ -132,7 +138,26 @@ type ( } ) -func newApp(ctx context.Context, log *Logger, v *viper.Viper) *App { +func (s *loggerSettings) DroppedLogsInc() { + s.mu.RLock() + defer s.mu.RUnlock() + + if s.appMetrics != nil { + s.appMetrics.Statistic().DroppedLogsInc() + } +} + +func (s *loggerSettings) setMetrics(appMetrics *metrics.AppMetrics) { + s.mu.Lock() + defer s.mu.Unlock() + + s.appMetrics = appMetrics +} + +func newApp(ctx context.Context, v *viper.Viper) *App { + logSettings := &loggerSettings{} + log := pickLogger(v, logSettings) + settings := newAppSettings(log, v) objPool, treePool, key := getPools(ctx, log.logger, v, settings.dialerSource) @@ -147,7 +172,8 @@ func newApp(ctx context.Context, log *Logger, v *viper.Viper) *App { webDone: make(chan struct{}, 1), wrkDone: make(chan struct{}, 1), - settings: settings, + settings: settings, + loggerSettings: logSettings, } app.init(ctx) @@ -163,7 +189,6 @@ func (a *App) init(ctx context.Context) { a.initPolicyStorage(ctx) a.initAPI(ctx) a.initMetrics() - a.initLogger() a.initServers(ctx) a.initTracing(ctx) } @@ -522,11 +547,7 @@ func (a *App) initMetrics() { a.metrics = metrics.NewAppMetrics(cfg) a.metrics.State().SetHealth(metrics.HealthStatusStarting) -} - -func (a *App) initLogger() { - coreWithContext := applyZapCoreMiddlewares(a.log.Core(), a.cfg, a.metrics) - a.log = zap.New(coreWithContext, zap.AddStacktrace(zap.NewAtomicLevelAt(zap.FatalLevel))) + a.loggerSettings.setMetrics(a.metrics) } func (a *App) initFrostfsID(ctx context.Context) { diff --git a/cmd/s3-gw/app_settings.go b/cmd/s3-gw/app_settings.go index f6fd931a..f33de2e8 100644 --- a/cmd/s3-gw/app_settings.go +++ b/cmd/s3-gw/app_settings.go @@ -18,7 +18,6 @@ import ( "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/logs" internalnet "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/net" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/version" - "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/metrics" "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/netmap" "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/pool" "git.frostfs.info/TrueCloudLab/zapjournald" @@ -1095,7 +1094,11 @@ func mergeConfig(v *viper.Viper, fileName string) error { return v.MergeConfig(cfgFile) } -func pickLogger(v *viper.Viper) *Logger { +type LoggerAppSettings interface { + DroppedLogsInc() +} + +func pickLogger(v *viper.Viper, settings LoggerAppSettings) *Logger { lvl, err := getLogLevel(v) if err != nil { panic(err) @@ -1105,9 +1108,9 @@ func pickLogger(v *viper.Viper) *Logger { switch dest { case destinationStdout: - return newStdoutLogger(lvl) + return newStdoutLogger(v, lvl, settings) case destinationJournald: - return newJournaldLogger(lvl) + return newJournaldLogger(v, lvl, settings) default: panic(fmt.Sprintf("wrong destination for logger: %s", dest)) } @@ -1127,19 +1130,21 @@ func pickLogger(v *viper.Viper) *Logger { // Logger records a stack trace for all messages at or above fatal level. // // See also zapcore.Level, zap.NewProductionConfig, zap.AddStacktrace. -func newStdoutLogger(lvl zapcore.Level) *Logger { +func newStdoutLogger(v *viper.Viper, lvl zapcore.Level, settings LoggerAppSettings) *Logger { stdout := zapcore.AddSync(os.Stderr) level := zap.NewAtomicLevelAt(lvl) consoleOutCore := zapcore.NewCore(newLogEncoder(), stdout, level) + consoleOutCore = applyZapCoreMiddlewares(consoleOutCore, v, settings) + return &Logger{ logger: zap.New(consoleOutCore, zap.AddStacktrace(zap.NewAtomicLevelAt(zap.FatalLevel))), lvl: level, } } -func newJournaldLogger(lvl zapcore.Level) *Logger { +func newJournaldLogger(v *viper.Viper, lvl zapcore.Level, settings LoggerAppSettings) *Logger { level := zap.NewAtomicLevelAt(lvl) encoder := zapjournald.NewPartialEncoder(newLogEncoder(), zapjournald.SyslogFields) @@ -1151,6 +1156,8 @@ func newJournaldLogger(lvl zapcore.Level) *Logger { zapjournald.SyslogPid(), }) + coreWithContext = applyZapCoreMiddlewares(coreWithContext, v, settings) + l := zap.New(coreWithContext, zap.AddStacktrace(zap.NewAtomicLevelAt(zap.FatalLevel))) return &Logger{ @@ -1166,6 +1173,22 @@ func newLogEncoder() zapcore.Encoder { return zapcore.NewConsoleEncoder(c) } +func applyZapCoreMiddlewares(core zapcore.Core, v *viper.Viper, settings LoggerAppSettings) zapcore.Core { + if v.GetBool(cfgLoggerSamplingEnabled) { + core = zapcore.NewSamplerWithOptions(core, + v.GetDuration(cfgLoggerSamplingInterval), + v.GetInt(cfgLoggerSamplingInitial), + v.GetInt(cfgLoggerSamplingThereafter), + zapcore.SamplerHook(func(_ zapcore.Entry, dec zapcore.SamplingDecision) { + if dec&zapcore.LogDropped > 0 { + settings.DroppedLogsInc() + } + })) + } + + return core +} + func getLogLevel(v *viper.Viper) (zapcore.Level, error) { var lvl zapcore.Level lvlStr := v.GetString(cfgLoggerLevel) @@ -1200,19 +1223,3 @@ LOOP: } return validDomains } - -func applyZapCoreMiddlewares(core zapcore.Core, v *viper.Viper, appMetrics *metrics.AppMetrics) zapcore.Core { - if v.GetBool(cfgLoggerSamplingEnabled) { - core = zapcore.NewSamplerWithOptions(core, - v.GetDuration(cfgLoggerSamplingInterval), - v.GetInt(cfgLoggerSamplingInitial), - v.GetInt(cfgLoggerSamplingThereafter), - zapcore.SamplerHook(func(_ zapcore.Entry, dec zapcore.SamplingDecision) { - if dec&zapcore.LogDropped > 0 { - appMetrics.DroppedLogsInc() - } - })) - } - - return core -} diff --git a/cmd/s3-gw/main.go b/cmd/s3-gw/main.go index f76bced3..c6e3b97b 100644 --- a/cmd/s3-gw/main.go +++ b/cmd/s3-gw/main.go @@ -10,9 +10,8 @@ func main() { g, _ := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM) v := newSettings() - l := pickLogger(v) - a := newApp(g, l, v) + a := newApp(g, v) go a.Serve(g) diff --git a/metrics/app.go b/metrics/app.go index 4898d727..1a2ef077 100644 --- a/metrics/app.go +++ b/metrics/app.go @@ -42,14 +42,6 @@ func NewAppMetrics(cfg AppMetricsConfig) *AppMetrics { } } -func (m *AppMetrics) DroppedLogsInc() { - if !m.isEnabled() { - return - } - - m.gate.Stats.DroppedLogsInc() -} - func (m *AppMetrics) SetEnabled(enabled bool) { if !enabled { m.logger.Warn(logs.MetricsAreDisabled) ```
👍 1
cmd/s3-gw/app_settings.go Outdated
@ -1132,4 +1133,3 @@
consoleOutCore := zapcore.NewCore(newLogEncoder(), stdout, level)
consoleOutCore = samplingEnabling(v, consoleOutCore)
dkirillov commented 2024-11-28 13:57:55 +00:00
Member
Copy link

Now sampling doen't work at all.
See:

diff --git a/api/handler/head.go b/api/handler/head.go
index 314adfe2..fbd4bb90 100644
--- a/api/handler/head.go
+++ b/api/handler/head.go
@@ -123,6 +123,10 @@ func (h *handler) HeadObjectHandler(w http.ResponseWriter, r *http.Request) {
 }
 
 func (h *handler) HeadBucketHandler(w http.ResponseWriter, r *http.Request) {
+       for i := 0; i < 10; i++ {
+               h.log.Info("for sampling")
+       }
+
        ctx := r.Context()
        reqInf

config:

logger:
  level: debug
  sampling:
    enabled: true
    initial: 5
    thereafter: 5
    interval: 1s

and the following request:

$ aws s3api  head-bucket --bucket bucket-name

leads to logging all 10 message

2024-11-28T16:53:48.056+0300    info    for sampling
2024-11-28T16:53:48.056+0300    info    for sampling
2024-11-28T16:53:48.056+0300    info    for sampling
2024-11-28T16:53:48.056+0300    info    for sampling
2024-11-28T16:53:48.056+0300    info    for sampling
2024-11-28T16:53:48.056+0300    info    for sampling
2024-11-28T16:53:48.056+0300    info    for sampling
2024-11-28T16:53:48.056+0300    info    for sampling
2024-11-28T16:53:48.056+0300    info    for sampling
2024-11-28T16:53:48.056+0300    info    for sampling
Now sampling doen't work at all. See: ```diff diff --git a/api/handler/head.go b/api/handler/head.go index 314adfe2..fbd4bb90 100644 --- a/api/handler/head.go +++ b/api/handler/head.go @@ -123,6 +123,10 @@ func (h *handler) HeadObjectHandler(w http.ResponseWriter, r *http.Request) { } func (h *handler) HeadBucketHandler(w http.ResponseWriter, r *http.Request) { + for i := 0; i < 10; i++ { + h.log.Info("for sampling") + } + ctx := r.Context() reqInf ``` config: ```yaml logger: level: debug sampling: enabled: true initial: 5 thereafter: 5 interval: 1s ``` and the following request: ``` $ aws s3api head-bucket --bucket bucket-name ``` leads to logging all 10 message ``` 2024-11-28T16:53:48.056+0300 info for sampling 2024-11-28T16:53:48.056+0300 info for sampling 2024-11-28T16:53:48.056+0300 info for sampling 2024-11-28T16:53:48.056+0300 info for sampling 2024-11-28T16:53:48.056+0300 info for sampling 2024-11-28T16:53:48.056+0300 info for sampling 2024-11-28T16:53:48.056+0300 info for sampling 2024-11-28T16:53:48.056+0300 info for sampling 2024-11-28T16:53:48.056+0300 info for sampling 2024-11-28T16:53:48.056+0300 info for sampling ```
pogpp commented 2024-11-29 08:07:13 +00:00
Author
Member
Copy link

tested this way
image

tested this way ![image](/attachments/1abc8f7d-3cde-4b52-8a44-d130d6dcea26)
image.png
40 KiB
dkirillov commented 2024-11-29 08:13:20 +00:00
Member
Copy link

It turns out this isn't correct way of testing in this case

It turns out this isn't correct way of testing in this case
dkirillov marked this conversation as resolved
pogpp force-pushed feature/502_sampling_metric from e1595e385b to 724e6b601e 2024-11-29 08:59:00 +00:00 Compare
pogpp requested review from dkirillov 2024-11-29 08:59:19 +00:00
pogpp force-pushed feature/502_sampling_metric from 724e6b601e to dccf3c1d07 2024-11-29 09:01:12 +00:00 Compare
dkirillov approved these changes 2024-12-02 13:28:38 +00:00
r.loginov approved these changes 2024-12-02 14:18:00 +00:00
pogpp force-pushed feature/502_sampling_metric from dccf3c1d07 to 51322cccdf 2024-12-03 12:16:58 +00:00 Compare
alexvanin approved these changes 2024-12-03 14:20:54 +00:00
alexvanin merged commit 51322cccdf into master 2024-12-03 14:20:58 +00:00
alexvanin deleted branch feature/502_sampling_metric 2024-12-03 14:21:04 +00:00
Sign in to join this conversation.
Reviewers
No reviewers
mbiryukova
nzinkevich
dkirillov
r.loginov
alexvanin
Labels
Clear labels   
P0

Highest

  
P1

High

  
P2

Medium

  
P3

Low

  
good first issue

Good for newcomers

  
Infrastructure

Infrastructure, Automation, CI/CD or DevOps related stuff

  
blocked

The issue or PR is blocked by something

  
bug

Something is not working

  
config

Configuration format update or breaking change

  
discussion

Talking about something in order to reach a decision or to exchange ideas

  
documentation

Documentation related

  
duplicate

This issue or pull request already exists

  
enhancement

New feature

  
go

Language features adoption

  
help wanted

Extra attention is needed

  
internal

   
invalid

Something is wrong

  
kludge

This is a kludge and we know it

  
observability

Related to metrics, tracing and logs

  
perfomance

Perfomance related

  
question

More information is needed

  
refactoring

Refactoring

  
wontfix

This won't be fixed

No labels
P0
P1
P2
P3
good first issue
Infrastructure
blocked
bug
config
discussion
documentation
duplicate
enhancement
go
help wanted
internal
invalid
kludge
observability
perfomance
question
refactoring
wontfix
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
pogpp
4 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: TrueCloudLab/frostfs-s3-gw#557
No description provided.
Delete branch "pogpp/frostfs-s3-gw:feature/502_sampling_metric"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?