Add partial encoder #15

alexvanin · 2024-01-23T16:47:13Z

alexvanin commented

2024-01-23 16:47:13 +00:00

Partial encoder is an encoder wrapper which ignores specific fields. This is useful with zapjournald core to ignore syslog fields
in human-readable console output.

Before:

2024-01-23T15:11:20.383Z        info        message        {"SYSLOG_FACILITY": 3, "SYSLOG_IDENTIFIER": "app", "SYSLOG_PID": 617050}

After:

2024-01-23T16:40:08.198Z        info        message

Syslog fields are still indexed.

$ journalctl -u app.service -o json-pretty
{
       ...
        "SYSLOG_PID" : "618104",
        "SYSLOG_FACILITY" : "3",
        "MESSAGE" : "2024-01-23T16:40:08.198Z\tinfo\tmessage\n",
        "SYSLOG_IDENTIFIER" : "app",
       ...
}

To use encoder, just wrap partial encoder around console or any other encoder.

	c := zap.NewProductionConfig()
	c.EncoderConfig.EncodeTime = zapcore.ISO8601TimeEncoder
	c.Level = zap.NewAtomicLevelAt(lvl)

	encoder := zapjournald.NewPartialEncoder(zapcore.NewConsoleEncoder(c.EncoderConfig), zapjournald.SyslogFields)

	core := zapjournald.NewCore(zap.NewAtomicLevelAt(lvl), encoder, &journald.Journal{}, zapjournald.SyslogFields)
	coreWithContext := core.With([]zapcore.Field{
		zapjournald.SyslogFacility(zapjournald.LogDaemon),
		zapjournald.SyslogIdentifier(),
		zapjournald.SyslogPid(),
	})
	l := zap.New(coreWithContext, zap.AddStacktrace(zap.NewAtomicLevelAt(zap.FatalLevel)))

Encoder adds almost none overhead and even improves logging speed for entities with syslog fields.

goos: linux
goarch: amd64
pkg: git.frostfs.info/TrueCloudLab/zapjournald
cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz
BenchmarkEncoder
BenchmarkEncoder/console
BenchmarkEncoder/console/no_fields
BenchmarkEncoder/console/no_fields-8         	 1373727	       823.9 ns/op	      64 B/op	       3 allocs/op
BenchmarkEncoder/console/application_fields
BenchmarkEncoder/console/application_fields-8         	 1000000	      1042 ns/op	     192 B/op	       4 allocs/op
BenchmarkEncoder/console/journald_fields
BenchmarkEncoder/console/journald_fields-8            	  887916	      1344 ns/op	     144 B/op	       5 allocs/op
BenchmarkEncoder/partial
BenchmarkEncoder/partial/no_fields
BenchmarkEncoder/partial/no_fields-8                  	 1447774	       790.9 ns/op	      64 B/op	       3 allocs/op
BenchmarkEncoder/partial/application_fields
BenchmarkEncoder/partial/application_fields-8         	 1000000	      1033 ns/op	     192 B/op	       4 allocs/op
BenchmarkEncoder/partial/journald_fields
BenchmarkEncoder/partial/journald_fields-8            	  894090	      1262 ns/op	     144 B/op	       5 allocs/op

Partial encoder is an encoder wrapper which ignores specific fields. This is useful with zapjournald core to ignore syslog fields in human-readable console output. Before: ``` 2024-01-23T15:11:20.383Z info message {"SYSLOG_FACILITY": 3, "SYSLOG_IDENTIFIER": "app", "SYSLOG_PID": 617050} ``` After: ``` 2024-01-23T16:40:08.198Z info message ``` Syslog fields are still indexed. ``` $ journalctl -u app.service -o json-pretty { ... "SYSLOG_PID" : "618104", "SYSLOG_FACILITY" : "3", "MESSAGE" : "2024-01-23T16:40:08.198Z\tinfo\tmessage\n", "SYSLOG_IDENTIFIER" : "app", ... } ``` To use encoder, just wrap partial encoder around console or any other encoder. ```go c := zap.NewProductionConfig() c.EncoderConfig.EncodeTime = zapcore.ISO8601TimeEncoder c.Level = zap.NewAtomicLevelAt(lvl) encoder := zapjournald.NewPartialEncoder(zapcore.NewConsoleEncoder(c.EncoderConfig), zapjournald.SyslogFields) core := zapjournald.NewCore(zap.NewAtomicLevelAt(lvl), encoder, &journald.Journal{}, zapjournald.SyslogFields) coreWithContext := core.With([]zapcore.Field{ zapjournald.SyslogFacility(zapjournald.LogDaemon), zapjournald.SyslogIdentifier(), zapjournald.SyslogPid(), }) l := zap.New(coreWithContext, zap.AddStacktrace(zap.NewAtomicLevelAt(zap.FatalLevel))) ``` Encoder adds almost none overhead and even improves logging speed for entities with syslog fields. ``` goos: linux goarch: amd64 pkg: git.frostfs.info/TrueCloudLab/zapjournald cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz BenchmarkEncoder BenchmarkEncoder/console BenchmarkEncoder/console/no_fields BenchmarkEncoder/console/no_fields-8 1373727 823.9 ns/op 64 B/op 3 allocs/op BenchmarkEncoder/console/application_fields BenchmarkEncoder/console/application_fields-8 1000000 1042 ns/op 192 B/op 4 allocs/op BenchmarkEncoder/console/journald_fields BenchmarkEncoder/console/journald_fields-8 887916 1344 ns/op 144 B/op 5 allocs/op BenchmarkEncoder/partial BenchmarkEncoder/partial/no_fields BenchmarkEncoder/partial/no_fields-8 1447774 790.9 ns/op 64 B/op 3 allocs/op BenchmarkEncoder/partial/application_fields BenchmarkEncoder/partial/application_fields-8 1000000 1033 ns/op 192 B/op 4 allocs/op BenchmarkEncoder/partial/journald_fields BenchmarkEncoder/partial/journald_fields-8 894090 1262 ns/op 144 B/op 5 allocs/op ```

alexvanin referenced this pull request from a commit

2024-01-23 16:47:45 +00:00

[#15] Add partial encoder

alexvanin force-pushed feature/partial-encoder from f4efb4e3cf to 045b966ecb

2024-01-23 16:47:46 +00:00

Compare

alexvanin requested review from fyrchik 2024-01-23 16:48:20 +00:00

alexvanin requested review from dkirillov 2024-01-23 16:48:20 +00:00

alexvanin requested review from AlekseySVTN 2024-01-23 16:48:20 +00:00

alexvanin self-assigned this 2024-01-23 17:11:16 +00:00

alexvanin commented

2024-01-23 17:14:10 +00:00

Related to #14 but not sure it addresses exact issues.

fyrchik reviewed 2024-01-23 17:53:50 +00:00

partial_encoder.go

					
				@ -0,0 +16,4 @@

				// NewPartialEncoder wraps existing encoder to avoid output of some provided

				// fields. The main use case is to ignore SyslogFields that leak into

				// ConsoleEncoder and provide no additional info for the human.

				func NewPartialEncoder(enc zapcore.Encoder, ignore []string) zapcore.Encoder {

fyrchik commented

2024-01-23 17:53:48 +00:00

Here you write a wrapper. Have you considered the possibility of just copying relevant bits from another encoder and changing what is necessary (zapjournald.Field parameters are accepted but ignored instead of being printed).

Here you write a wrapper. Have you considered the possibility of just copying relevant bits from another encoder and changing what is necessary (`zapjournald.Field` parameters are accepted but ignored instead of being printed).

fyrchik commented

2024-01-23 17:57:35 +00:00

Oh, please, disregard, I have looked at cmd/main.go and it contains old code.

Oh, please, disregard, I have looked at `cmd/main.go` and it contains old code.

fyrchik marked this conversation as resolved

dkirillov approved these changes 2024-01-24 06:43:43 +00:00

pogpp approved these changes 2024-01-24 08:46:07 +00:00

fyrchik approved these changes 2024-01-24 09:24:25 +00:00

alexvanin merged commit 045b966ecb into master

2024-01-24 11:18:07 +00:00

alexvanin deleted branch feature/partial-encoder

2024-01-24 11:18:08 +00:00