* pkg/replace: make it more efficient.
Remove the map that is allocated on every write and make it more static,
but just defining a function that gets called for a label and returns
its value.
Remove the interface definition and just implement what is needed in our
case. Add benchmark test for replace as well.
Extend metadata test to test multiple values (pretty sure this didn't
work, but there wasn't a test for it, so can't be sure).
Update all callers to use it - concurrent use should be fine as we pass
everything by value.
Benchmarks in replacer:
new: BenchmarkReplacer-4 300000 4717 ns/op 240 B/op 8 allocs/op
old: BenchmarkReplacer-4 300000 4368 ns/op 384 B/op 11 allocs/op
Added benchmark function to the old code to test it.
~~~
func BenchmarkReplacer(b *testing.B) {
w := dnstest.NewRecorder(&test.ResponseWriter{})
r := new(dns.Msg)
r.SetQuestion("example.org.", dns.TypeHINFO)
r.MsgHdr.AuthenticatedData = true
b.ResetTimer()
b.ReportAllocs()
repl := New(context.TODO(), r, w, "")
for i := 0; i < b.N; i++ {
repl.Replace("{type} {name} {size}")
}
}
~~~
New code contains (of course a different one). The amount of ops is
more, which might be good to look at some more. For all the allocations
is seems it was quite performant.
This looks to be 50% faster, and there is less allocations in log
plugin:
old: BenchmarkLogged-4 20000 70526 ns/op
new: BenchmarkLogged-4 30000 57558 ns/op
Signed-off-by: Miek Gieben <miek@miek.nl>
* Stickler bot
Signed-off-by: Miek Gieben <miek@miek.nl>
* Improve test coverage
Signed-off-by: Miek Gieben <miek@miek.nl>
* typo
Signed-off-by: Miek Gieben <miek@miek.nl>
* Add test for malformed log lines
Signed-off-by: Miek Gieben <miek@miek.nl>
This was added, but didn't see any use. For a large, complex chunk of
code we should have some users of it.
Remove all watch functionally from plugins, servers and packages.
Fixes: #2548
Signed-off-by: Miek Gieben <miek@miek.nl>
* Default to upstream to self
This is a backwards incompatible change.
This is a massive (cleanup) PR where we default to resolving external
names by the coredns process itself, instead of directly forwarding them
to some upstream.
This ignores any arguments `upstream` may have had and makes it depend
on proxy/forward configuration in the Corefile. This allows resolved
upstream names to be cached and we have better healthchecking of the
upstreams. It also means there is only one way to resolve names, by
either using the proxy or forward plugin.
The proxy/forward lookup.go functions have been removed. This also
lessen the dependency on proxy, meaning deprecating proxy will become
easier. Some tests have been removed as well, or moved to the top-level
test directory as they now require a full coredns process instead of
just the plugin.
For the etcd plugin, the entire StubZone resolving is *dropped*! This
was a hacky (but working) solution to say the least. If someone cares
deeply it can be brought back (maybe)?
The pkg/upstream is now very small and almost does nothing. Also the
New() function was changed to return a pointer to upstream.Upstream. It
also returns only one parameter, so any stragglers using it will
encounter a compile error.
All documentation has been adapted. This affected the following plugins:
* etcd
* file
* auto
* secondary
* federation
* template
* route53
A followup PR will make any upstream directives with arguments an error,
right now they are ignored.
Signed-off-by: Miek Gieben <miek@miek.nl>
* Fix etcd build - probably still fails unit test
Signed-off-by: Miek Gieben <miek@miek.nl>
* Slightly smarter lookup check in upstream
Signed-off-by: Miek Gieben <miek@miek.nl>
* Compilez
Signed-off-by: Miek Gieben <miek@miek.nl>
The miekg/dns API is a bit scruffy (documentation PR in flight soon),
but these both need to be set for the correct RCODE to be reflected in
the returned packet.
Signed-off-by: Miek Gieben <miek@miek.nl>
* Fix EDNS0 compliance
Do SizeAndDo in the server (ScrubWriter) and remove all uses of this
from the plugins. Also *always* do it. This is to get into compliance
for https://dnsflagday.net/.
The pkg/edns0 now exports the EDNS0 options we understand; this is
exported to allow plugins add things there. The *rewrite* plugin used
this to add custom EDNS0 option codes that the server needs to
understand.
This also needs a new release of miekg/dns because it triggered a
race-condition that was basicly there forever.
See:
* https://github.com/miekg/dns/issues/857
* https://github.com/miekg/dns/pull/859
Running a test instance and pointing the https://ednscomp.isc.org/ednscomp
to it shows the tests are now fixed:
~~~
EDNS Compliance Tester
Checking: 'miek.nl' as at 2018-12-01T17:53:15Z
miek.nl. @147.75.204.203 (drone.coredns.io.): dns=ok edns=ok edns1=ok edns@512=ok ednsopt=ok edns1opt=ok do=ok ednsflags=ok docookie=ok edns512tcp=ok optlist=ok
miek.nl. @2604:1380:2002:a000::1 (drone.coredns.io.): dns=ok edns=ok edns1=ok edns@512=ok ednsopt=ok edns1opt=ok do=ok ednsflags=ok docookie=ok edns512tcp=ok optlist=ok
All Ok
Codes
ok - test passed.
~~~
Signed-off-by: Miek Gieben <miek@miek.nl>
Signed-off-by: Miek Gieben <miek@miek.nl>
* typos in comments
Signed-off-by: Miek Gieben <miek@miek.nl>
* pkg/up: implement backoff
Every 2nd failure we double the interval until we hit 4 * interval. This
to have some sort of backoff, esp when a large cluster of coredns shares
an upstream (original intent of up package) they will hammer the
upstream. This put some back pressure on that.
Signed-off-by: Miek Gieben <miek@miek.nl>
* Update plugin/pkg/up/up.go
Co-Authored-By: miekg <miek@miek.nl>
* Redo the plugin log PR
Remove the code duplication and call of the "official" functions. This
is the second(?) time we forgot to update the other half, so remove that
problem entirely.
Also add a test if the correct (within limits) time in front of the log
line.
Signed-off-by: Miek Gieben <miek@miek.nl>
* Remove pFormat
Signed-off-by: Miek Gieben <miek@miek.nl>
999 chops of suffix zero, 000 allows for the milliseconds to be always
printed with 3 chars. This makes the log the printed with the same
columns.
(partial logs below)
2018-11-13T21:13:28.249Z [INFO] [::1]
2018-11-13T21:13:48.414Z [INFO] [::1]
2018-11-13T21:13:49.1Z [INFO] [::1]
vs:
2018-11-13T21:20:22.262Z [INFO] [::1]
2018-11-13T21:20:22.436Z [INFO] [::1]
2018-11-13T21:20:22.608Z [INFO] [::1]
Signed-off-by: Miek Gieben <miek@miek.nl>
* - UT on metrics verifying that all plugins of all blocs have their metrics collectors declared
* - fix error msg
* - redirect Registry of metric to the one that handle the listener
- allow duplicate of metrics collector on the same Registry (case of same plugin in 2 blocs listening metrics on the same address)
* - fix change of signature
* - ensure cleaning metrics before starting the test (metrics collectors are global vars .. and re-used by several tests)
* - I think I fixed this test. Ensure correct mn of hits and clean metrics before test.
* - fix typo in error msg - proposed at review
* - fix typo in comment
* - remove ResetMetrics functions
- change a way to test the numeric metrics : get the diff between begin and end of test
* - oops. removing debug logs
Default to using pkg/log for all logging and use a fixed time prefix
which is RFC3339Millli (doesn't exist in time, so we just extended
RFC3339), i.e. Nano might be pushing it.
Logs go from:
2018/10/30 19:14:55 [INFO] CoreDNS-1.2.5
2018/10/30 19:14:55 [INFO] linux/amd64, go1.11,
to:
2018-10-30T19:10:07.547Z [INFO] CoreDNS-1.2.5
2018-10-30T19:10:07.547Z [INFO] linux/amd64, go1.11,
Which includes the timezone - which oddly the std log package doesn't
natively do.
Signed-off-by: Miek Gieben <miek@miek.nl>
* plugin/kubernetes: remove bunch a string ops
This removes a bunch of appends to where not needed, makes dnsutil.Join
take variadic args which removes the need to wrap in a new string slice.
Signed-off-by: Miek Gieben <miek@miek.nl>
* Fix calls to dnsutil.Join
Signed-off-by: Miek Gieben <miek@miek.nl>
* Revert these
Signed-off-by: Miek Gieben <miek@miek.nl>
* Move functions from pkg/transport to pkg/parse
Although "parse" is a fairly generic name I believe this is somewhat
better named. pkg/transport keeps a few constants that are uses
throughout for the rest is is renaming a bunch (and the fallout from
there to make things compile again).
Signed-off-by: Miek Gieben <miek@miek.nl>
* Fix tests
Signed-off-by: Miek Gieben <miek@miek.nl>
Create plugin/pkg/transport that holds the transport related functions.
This needed to be a new pkg to prevent cyclic import errors.
This cleans up a bunch of duplicated code in core/dnsserver that also
tried to parse a transport (now all done in transport.Parse).
Signed-off-by: Miek Gieben <miek@miek.nl>
* Create test to verify correct listener behavior
* Create Unset function to remove todo items
* Reset address for prometheus listener before restarting
* Add inline documentation for Unset function
* Make shutdownTimeout a constant and change to five seconds
* Revert ForEach behavior in uniq package
* New plugin: loop
Add a plugin that detects loops. It does this by sending an unique query
to our selves. If we see the query more than twice we stop the process.
If there isn't a loop, the plugin disables it self and becomes a noop
plugin.
Signed-off-by: Miek Gieben <miek@miek.nl>
* bliep
Signed-off-by: Miek Gieben <miek@miek.nl>
* plugin/log: add log.Fatal[f]
Add log.Fatal(f) to mimic more of the log package. The first and only
use is in the (new) loop plugin.
Signed-off-by: Miek Gieben <miek@miek.nl>
* Clean up tests logging
This cleans up the travis logs so you can see the failures better.
Older tests in tests/ would call log.SetOutput(ioutil.Discard) in
a haphazard way. This add log.Discard and put an `init` function in each
package's dir (no way to do this globally). The cleanup in tests/ is
clear.
All plugins also got this init function to have some uniformity and kill
any (future) logging there in the tests as well.
There is a one-off in pkg/healthcheck because that does log.
Signed-off-by: Miek Gieben <miek@miek.nl>
* bring back original log_test.go
Signed-off-by: Miek Gieben <miek@miek.nl>
* suppress logging here as well
Signed-off-by: Miek Gieben <miek@miek.nl>
* DoH: put in pkg/doh
Factor out the DoH stuff into its own package, add function to request
a DoH response. This can be used by forward (and maybe proxy) to
implement DoH client support.
Signed-off-by: Miek Gieben <miek@miek.nl>
* lint
Signed-off-by: Miek Gieben <miek@miek.nl>
* ... and make it compile
Signed-off-by: Miek Gieben <miek@miek.nl>
This revert 17d807f0 and re-adds the metadata plugin as a plugin that
just sets a label to a value function.
Add package documentation on how to use the metadata package. Make it
clear that any caching is up to the Func implemented.
There are now - no in tree users. We could add the request metadata by
default under names that copy request.Request, i.e
request/ip - remote IP
request/port - remote port
Variables.go has been deleted.
Signed-off-by: Miek Gieben <miek@miek.nl>
* plugin/metadata: some cleanups
Name to provider.go as that's what being defined right now in the file.
Use request.Request because that's done in variables.go anyway. Name the
main storage M, because there is no further meaning behind.
Remove superfluous methods
Signed-off-by: Miek Gieben <miek@miek.nl>
* Fix test
Signed-off-by: Miek Gieben <miek@miek.nl>
* Fix max-age in http server
Move the minMsgTTL to dnsutil and rename it MinimalTTL, move some
constants there as well.
Use these new function in server_https to correctly set the max-age
HTTP header.
Fixes: #1823
* Linter
* Add part 1 watch functionality. (squashed)
* add funcs for service/endpoint fqdns
* add endpoints watch
* document exposed funcs
* only send subset deltas
* locking for watch map
* tests and docs
* add pod watch
* remove debugs prints
* feedback part 1
* add error reporting to proto
* inform clients of server stop+errors
* add grpc options param
* use proper context
* Review feedback:
* Removed client (will move to another repo)
* Use new log functions
* Change watchChan to be for string not []string
* Rework how k8s plugin stores watch tracking info to simplify
* Normalize the qname on watch request
* Add blank line back
* Revert another spurious change
* Fix tests
* Add stop channel.
Fix tests.
Better docs for plugin interface.
* fmt.Printf -> log.Warningf
* Move from dnsserver to plugin/pkg/watch
* gofmt
* remove dead client watches
* sate linter
* linter omg
add a test to see if we copy the rcode correctly. Some minor cleanup in
import ordering and renaming NewUpstream to New as we already are in the
upstream package.
* plugin/file: fix local CNAME lookup
Issue #1864 explains it will, when we serve the child zone as well we
should just recursive into ourself (upstream self). Thus relax the
IsSubDomain check in file/lookup.go and just query (even if the query
will hit a remote server).
I've looped over all other plugins that do something similar (CNAME
resolving) and they didn't do the IsSubDomain check; therefor I've
removed it from *file* as well.
Added test in file_upstream_test that shows this failed before but now
results in a reply.
Fixes#1864
* self does not need to be exported
* Fix test
We don't know if we had a valid reply. Check this.
Remove the code and remove the call in etcd and kubernetes handlers.
This does mean we should not add dups in the first place, which means
adding maps in backend_lookup to prevent dups from begin added.
This should cut down on the allocations because dnsutil.Dedup is very
expensive by converting everything to strings, we avoid doing that now.
* Current stage of the log files. Test need to be done as well as formatting of times.
* Finished testing. All altered classes test pass along with my additions
* Updated the replacer package to print the units as well. May take out.
* Changed the time units to be within the rules. Fixed the test as well.
* Fixed some tests, updated the readme, fixed the replacer class.
* Updates of standardizing only to seconds in response duration. Need to revert README.
* Reverted readme.
* Added a small test in new replacer.
* Changed replacer to inline the strconv for duration.
Uppercase all these test errors as well. And extend the presubmit to
check for these in the future. Also do a slightly smarter grep to only
get t.<something>. as (because dump regexp) this also grep over non test
files.
The DoH work (#1619) made changes to pkg/nonwriter.Writer that in
hindsight were not backwards compatible; it added override for the
LocalAddr() and RemoteAddr(). Instead of rolling back that PR, this PR
reverts those changes and creates a DoHWriter for use in the
https-server.go side of things.
This was only caught in the integration test making this hard to catch,
so we add a upstream_file_test.go that tries (doesn't work yet) to test
this in the unit tests as well. Esp. helpful when 'git bisecting'.
Fixes#1826
* WIP: make CoreDNS DoH Server
* It works
* Fix tests
* Review from Tom - on diff. PR
* correct mime type
* Cleanups and use the pkg/nonwriter
* rename and updates
* implement get
* implement GET
* Code review comments
* correct context
* tweaks
* code review
* Probe simplification
- the main reason of rework is that previous implementation hung
when calling Do() after Stop()
* replace atomics with mutex
* access Probe.interval under lock
* update docs
* plugins: use plugin specific logging
Hooking up pkg/log also changed NewWithPlugin to just take a string
instead of a plugin.Handler as that is more flexible and for instance
the Root "plugin" doesn't implement it fully.
Same logging from the reload plugin:
.:1043
2018/04/22 08:56:37 [INFO] CoreDNS-1.1.1
2018/04/22 08:56:37 [INFO] linux/amd64, go1.10.1,
CoreDNS-1.1.1
linux/amd64, go1.10.1,
2018/04/22 08:56:37 [INFO] plugin/reload: Running configuration MD5 = ec4c9c55cd19759ea1c46b8c45742b06
2018/04/22 08:56:54 [INFO] Reloading
2018/04/22 08:56:54 [INFO] plugin/reload: Running configuration MD5 = 9e2bfdd85bdc9cceb740ba9c80f34c1a
2018/04/22 08:56:54 [INFO] Reloading complete
* update docs
* better doc
