* pkg/cache: fix race in Add() and Evict()
This fixes a race in Add() when the shard is at max capacity and the key
being added is already stored. Previously, the shard would evict a
random value - when all it needed to do was replace an existing value.
There was a race in how Evict() picked which key to remove, which would
cause concurrent calls to Evict() to remove the same key.
Additionally, this commit removes a lot of the lock contention and a
race around Add() and Evict() by changing them to immediately hold the
write lock. Previously, they would check conditions with the read lock
held and not re-check those conditions once the write lock was acquired
(this is a race).
* pkg/cache: code review comments
* pkg/cache: simplify Add() logic
* replacer: evaluate format once and improve perf by ~3x
This improves the performance of logging by almost 3x and reduces memory
usage by ~8x.
Benchmark results:
benchmark old ns/op new ns/op delta
BenchmarkReplacer-12 644 324 -49.69%
BenchmarkReplacer_CommonLogFormat-12 4228 1471 -65.21%
benchmark old allocs new allocs delta
BenchmarkReplacer-12 8 2 -75.00%
BenchmarkReplacer_CommonLogFormat-12 51 17 -66.67%
benchmark old bytes new bytes delta
BenchmarkReplacer-12 240 48 -80.00%
BenchmarkReplacer_CommonLogFormat-12 3723 446 -88.02%
* replacer: code review comments
* bufPool: document why we use a pointer to a slice
* parseFormat: fix confusing comment
* TestParseFormat_Nodes: rename to TestParseFormatNodes
* replacer: use a value for bufPool instead of a pointer
* replacer: remove comment
* replacer: replace labels with knownLabels
The previous slice of label names is no longer needed.
* Update Caddy to 1.0.1, and update import path
This fix updates caddy to 1.0.1 and also
updates the import path to github.com/caddyserver/caddy
This fix fixes 2959
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
* Also update plugin.cfg
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
* Update and bump zplugin.go
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
Add OnRestartFailed to the ready plugin and some various cleanups.
Document slightly better how things are supposed to work with multiple
`ready`'s in the multiple Server Blocks.
All manually tested with this Corefile:
~~~
. {
log
ready
}
example.org {
log
chaos
ready
}
~~~
And then `kill -SIGUSR1` and curling the ready endpoint. This works
well, the FailedReload is triggered by adding a syntax error in the
Corefile.
See #2659
Signed-off-by: Miek Gieben <miek@miek.nl>
* pkg/log: fix data race on d
Wrap d in a mutex to prevent data race. This makes is slower, but this
is a debugging aid anyway. It's not used normally.
Signed-off-by: Miek Gieben <miek@miek.nl>
* Fix tests compilation
Signed-off-by: Miek Gieben <miek@miek.nl>
* Fix test compile
Signed-off-by: Miek Gieben <miek@miek.nl>
This fixes a data race on the listener(s) that get started in the
metrics plugins.
It also restore pkg/uniq to its former glory and removes and state being
carried in there; this means for metrics that registry.go was to
replicate that behavior *with* locking (as pkg/uniq doesn't do, or need
that).
Also renamed uniqAddr to just u, to make it slightly shorter.
Signed-off-by: Miek Gieben <miek@miek.nl>
* Remove context.Context from request.Request
This removes the context from request.Request and makes all the changes
in the code to make it compile again. It's all mechanical. It did
unearth some weirdness in that the context was kept in handler structs
which may cause havoc with concurrently handling of requests.
Fixes#2721
Signed-off-by: Miek Gieben <miek@miek.nl>
* Make test compile
Signed-off-by: Miek Gieben <miek@miek.nl>
Add another test case for impossible DNS messages which should not be
cached. This adds a check for a message that denies its own existence.
Fixes#2724.
Signed-off-by: Miek Gieben <miek@miek.nl>
* pkg/replace: make it more efficient.
Remove the map that is allocated on every write and make it more static,
but just defining a function that gets called for a label and returns
its value.
Remove the interface definition and just implement what is needed in our
case. Add benchmark test for replace as well.
Extend metadata test to test multiple values (pretty sure this didn't
work, but there wasn't a test for it, so can't be sure).
Update all callers to use it - concurrent use should be fine as we pass
everything by value.
Benchmarks in replacer:
new: BenchmarkReplacer-4 300000 4717 ns/op 240 B/op 8 allocs/op
old: BenchmarkReplacer-4 300000 4368 ns/op 384 B/op 11 allocs/op
Added benchmark function to the old code to test it.
~~~
func BenchmarkReplacer(b *testing.B) {
w := dnstest.NewRecorder(&test.ResponseWriter{})
r := new(dns.Msg)
r.SetQuestion("example.org.", dns.TypeHINFO)
r.MsgHdr.AuthenticatedData = true
b.ResetTimer()
b.ReportAllocs()
repl := New(context.TODO(), r, w, "")
for i := 0; i < b.N; i++ {
repl.Replace("{type} {name} {size}")
}
}
~~~
New code contains (of course a different one). The amount of ops is
more, which might be good to look at some more. For all the allocations
is seems it was quite performant.
This looks to be 50% faster, and there is less allocations in log
plugin:
old: BenchmarkLogged-4 20000 70526 ns/op
new: BenchmarkLogged-4 30000 57558 ns/op
Signed-off-by: Miek Gieben <miek@miek.nl>
* Stickler bot
Signed-off-by: Miek Gieben <miek@miek.nl>
* Improve test coverage
Signed-off-by: Miek Gieben <miek@miek.nl>
* typo
Signed-off-by: Miek Gieben <miek@miek.nl>
* Add test for malformed log lines
Signed-off-by: Miek Gieben <miek@miek.nl>
This was added, but didn't see any use. For a large, complex chunk of
code we should have some users of it.
Remove all watch functionally from plugins, servers and packages.
Fixes: #2548
Signed-off-by: Miek Gieben <miek@miek.nl>
* Default to upstream to self
This is a backwards incompatible change.
This is a massive (cleanup) PR where we default to resolving external
names by the coredns process itself, instead of directly forwarding them
to some upstream.
This ignores any arguments `upstream` may have had and makes it depend
on proxy/forward configuration in the Corefile. This allows resolved
upstream names to be cached and we have better healthchecking of the
upstreams. It also means there is only one way to resolve names, by
either using the proxy or forward plugin.
The proxy/forward lookup.go functions have been removed. This also
lessen the dependency on proxy, meaning deprecating proxy will become
easier. Some tests have been removed as well, or moved to the top-level
test directory as they now require a full coredns process instead of
just the plugin.
For the etcd plugin, the entire StubZone resolving is *dropped*! This
was a hacky (but working) solution to say the least. If someone cares
deeply it can be brought back (maybe)?
The pkg/upstream is now very small and almost does nothing. Also the
New() function was changed to return a pointer to upstream.Upstream. It
also returns only one parameter, so any stragglers using it will
encounter a compile error.
All documentation has been adapted. This affected the following plugins:
* etcd
* file
* auto
* secondary
* federation
* template
* route53
A followup PR will make any upstream directives with arguments an error,
right now they are ignored.
Signed-off-by: Miek Gieben <miek@miek.nl>
* Fix etcd build - probably still fails unit test
Signed-off-by: Miek Gieben <miek@miek.nl>
* Slightly smarter lookup check in upstream
Signed-off-by: Miek Gieben <miek@miek.nl>
* Compilez
Signed-off-by: Miek Gieben <miek@miek.nl>
The miekg/dns API is a bit scruffy (documentation PR in flight soon),
but these both need to be set for the correct RCODE to be reflected in
the returned packet.
Signed-off-by: Miek Gieben <miek@miek.nl>
* Fix EDNS0 compliance
Do SizeAndDo in the server (ScrubWriter) and remove all uses of this
from the plugins. Also *always* do it. This is to get into compliance
for https://dnsflagday.net/.
The pkg/edns0 now exports the EDNS0 options we understand; this is
exported to allow plugins add things there. The *rewrite* plugin used
this to add custom EDNS0 option codes that the server needs to
understand.
This also needs a new release of miekg/dns because it triggered a
race-condition that was basicly there forever.
See:
* https://github.com/miekg/dns/issues/857
* https://github.com/miekg/dns/pull/859
Running a test instance and pointing the https://ednscomp.isc.org/ednscomp
to it shows the tests are now fixed:
~~~
EDNS Compliance Tester
Checking: 'miek.nl' as at 2018-12-01T17:53:15Z
miek.nl. @147.75.204.203 (drone.coredns.io.): dns=ok edns=ok edns1=ok edns@512=ok ednsopt=ok edns1opt=ok do=ok ednsflags=ok docookie=ok edns512tcp=ok optlist=ok
miek.nl. @2604:1380:2002:a000::1 (drone.coredns.io.): dns=ok edns=ok edns1=ok edns@512=ok ednsopt=ok edns1opt=ok do=ok ednsflags=ok docookie=ok edns512tcp=ok optlist=ok
All Ok
Codes
ok - test passed.
~~~
Signed-off-by: Miek Gieben <miek@miek.nl>
Signed-off-by: Miek Gieben <miek@miek.nl>
* typos in comments
Signed-off-by: Miek Gieben <miek@miek.nl>
* pkg/up: implement backoff
Every 2nd failure we double the interval until we hit 4 * interval. This
to have some sort of backoff, esp when a large cluster of coredns shares
an upstream (original intent of up package) they will hammer the
upstream. This put some back pressure on that.
Signed-off-by: Miek Gieben <miek@miek.nl>
* Update plugin/pkg/up/up.go
Co-Authored-By: miekg <miek@miek.nl>
* Redo the plugin log PR
Remove the code duplication and call of the "official" functions. This
is the second(?) time we forgot to update the other half, so remove that
problem entirely.
Also add a test if the correct (within limits) time in front of the log
line.
Signed-off-by: Miek Gieben <miek@miek.nl>
* Remove pFormat
Signed-off-by: Miek Gieben <miek@miek.nl>
999 chops of suffix zero, 000 allows for the milliseconds to be always
printed with 3 chars. This makes the log the printed with the same
columns.
(partial logs below)
2018-11-13T21:13:28.249Z [INFO] [::1]
2018-11-13T21:13:48.414Z [INFO] [::1]
2018-11-13T21:13:49.1Z [INFO] [::1]
vs:
2018-11-13T21:20:22.262Z [INFO] [::1]
2018-11-13T21:20:22.436Z [INFO] [::1]
2018-11-13T21:20:22.608Z [INFO] [::1]
Signed-off-by: Miek Gieben <miek@miek.nl>
* - UT on metrics verifying that all plugins of all blocs have their metrics collectors declared
* - fix error msg
* - redirect Registry of metric to the one that handle the listener
- allow duplicate of metrics collector on the same Registry (case of same plugin in 2 blocs listening metrics on the same address)
* - fix change of signature
* - ensure cleaning metrics before starting the test (metrics collectors are global vars .. and re-used by several tests)
* - I think I fixed this test. Ensure correct mn of hits and clean metrics before test.
* - fix typo in error msg - proposed at review
* - fix typo in comment
* - remove ResetMetrics functions
- change a way to test the numeric metrics : get the diff between begin and end of test
* - oops. removing debug logs
Default to using pkg/log for all logging and use a fixed time prefix
which is RFC3339Millli (doesn't exist in time, so we just extended
RFC3339), i.e. Nano might be pushing it.
Logs go from:
2018/10/30 19:14:55 [INFO] CoreDNS-1.2.5
2018/10/30 19:14:55 [INFO] linux/amd64, go1.11,
to:
2018-10-30T19:10:07.547Z [INFO] CoreDNS-1.2.5
2018-10-30T19:10:07.547Z [INFO] linux/amd64, go1.11,
Which includes the timezone - which oddly the std log package doesn't
natively do.
Signed-off-by: Miek Gieben <miek@miek.nl>
* plugin/kubernetes: remove bunch a string ops
This removes a bunch of appends to where not needed, makes dnsutil.Join
take variadic args which removes the need to wrap in a new string slice.
Signed-off-by: Miek Gieben <miek@miek.nl>
* Fix calls to dnsutil.Join
Signed-off-by: Miek Gieben <miek@miek.nl>
* Revert these
Signed-off-by: Miek Gieben <miek@miek.nl>
* Move functions from pkg/transport to pkg/parse
Although "parse" is a fairly generic name I believe this is somewhat
better named. pkg/transport keeps a few constants that are uses
throughout for the rest is is renaming a bunch (and the fallout from
there to make things compile again).
Signed-off-by: Miek Gieben <miek@miek.nl>
* Fix tests
Signed-off-by: Miek Gieben <miek@miek.nl>
Create plugin/pkg/transport that holds the transport related functions.
This needed to be a new pkg to prevent cyclic import errors.
This cleans up a bunch of duplicated code in core/dnsserver that also
tried to parse a transport (now all done in transport.Parse).
Signed-off-by: Miek Gieben <miek@miek.nl>
* Create test to verify correct listener behavior
* Create Unset function to remove todo items
* Reset address for prometheus listener before restarting
* Add inline documentation for Unset function
* Make shutdownTimeout a constant and change to five seconds
* Revert ForEach behavior in uniq package
* New plugin: loop
Add a plugin that detects loops. It does this by sending an unique query
to our selves. If we see the query more than twice we stop the process.
If there isn't a loop, the plugin disables it self and becomes a noop
plugin.
Signed-off-by: Miek Gieben <miek@miek.nl>
* bliep
Signed-off-by: Miek Gieben <miek@miek.nl>
* plugin/log: add log.Fatal[f]
Add log.Fatal(f) to mimic more of the log package. The first and only
use is in the (new) loop plugin.
Signed-off-by: Miek Gieben <miek@miek.nl>
* Clean up tests logging
This cleans up the travis logs so you can see the failures better.
Older tests in tests/ would call log.SetOutput(ioutil.Discard) in
a haphazard way. This add log.Discard and put an `init` function in each
package's dir (no way to do this globally). The cleanup in tests/ is
clear.
All plugins also got this init function to have some uniformity and kill
any (future) logging there in the tests as well.
There is a one-off in pkg/healthcheck because that does log.
Signed-off-by: Miek Gieben <miek@miek.nl>
* bring back original log_test.go
Signed-off-by: Miek Gieben <miek@miek.nl>
* suppress logging here as well
Signed-off-by: Miek Gieben <miek@miek.nl>
* DoH: put in pkg/doh
Factor out the DoH stuff into its own package, add function to request
a DoH response. This can be used by forward (and maybe proxy) to
implement DoH client support.
Signed-off-by: Miek Gieben <miek@miek.nl>
* lint
Signed-off-by: Miek Gieben <miek@miek.nl>
* ... and make it compile
Signed-off-by: Miek Gieben <miek@miek.nl>
This revert 17d807f0 and re-adds the metadata plugin as a plugin that
just sets a label to a value function.
Add package documentation on how to use the metadata package. Make it
clear that any caching is up to the Func implemented.
There are now - no in tree users. We could add the request metadata by
default under names that copy request.Request, i.e
request/ip - remote IP
request/port - remote port
Variables.go has been deleted.
Signed-off-by: Miek Gieben <miek@miek.nl>
* plugin/metadata: some cleanups
Name to provider.go as that's what being defined right now in the file.
Use request.Request because that's done in variables.go anyway. Name the
main storage M, because there is no further meaning behind.
Remove superfluous methods
Signed-off-by: Miek Gieben <miek@miek.nl>
* Fix test
Signed-off-by: Miek Gieben <miek@miek.nl>
* Fix max-age in http server
Move the minMsgTTL to dnsutil and rename it MinimalTTL, move some
constants there as well.
Use these new function in server_https to correctly set the max-age
HTTP header.
Fixes: #1823
* Linter
* Add part 1 watch functionality. (squashed)
* add funcs for service/endpoint fqdns
* add endpoints watch
* document exposed funcs
* only send subset deltas
* locking for watch map
* tests and docs
* add pod watch
* remove debugs prints
* feedback part 1
* add error reporting to proto
* inform clients of server stop+errors
* add grpc options param
* use proper context
* Review feedback:
* Removed client (will move to another repo)
* Use new log functions
* Change watchChan to be for string not []string
* Rework how k8s plugin stores watch tracking info to simplify
* Normalize the qname on watch request
* Add blank line back
* Revert another spurious change
* Fix tests
* Add stop channel.
Fix tests.
Better docs for plugin interface.
* fmt.Printf -> log.Warningf
* Move from dnsserver to plugin/pkg/watch
* gofmt
* remove dead client watches
* sate linter
* linter omg
add a test to see if we copy the rcode correctly. Some minor cleanup in
import ordering and renaming NewUpstream to New as we already are in the
upstream package.
