The health endpoint histogram has a large amount of cardinality for a
simple endpoint. Introduce a new "Slim" set of buckets for `/health` to
reduce the metrics load on large deployments. Especially those that have
per-node DNS caching services.
Add a metric to count internal health check failures rather than use the
timeout value as side effect monitor of the check error. This avoids
incorrectly recording the timeout value if there is an error that is not
a timeout (ex. refused)
Signed-off-by: SuperQ <superq@gmail.com>
Went over all generated manual pages and fixed some markdown issues,
mostly escaping "_" to avoid underlining entire paragraphs.
Some textual fixes in route53 and other cloud DNS plugins.
Regenerated the markdown with mmark.
Signed-off-by: Miek Gieben <miek@miek.nl>
Caught my eye, we name things directive still, esp when talking about
the prometheus *plugin*. Rename everything that needs to be plugin to
'plugin'. Also make sure Metrics is a H2 section (not H1).
Signed-off-by: Miek Gieben <miek@miek.nl>
Add OnReStartFailed which makes the health plugin stay up if the
Corefile is corrupt and we revert to the previous version.
Also needs a fix for the channel handling
See #2659
Testing it will log the following when restarting with a corrupted
Corefile
~~~
2019-05-04T18:01:59.431Z [INFO] linux/amd64, go1.12.4,
CoreDNS-1.5.0
linux/amd64, go1.12.4,
[INFO] SIGUSR1: Reloading
[INFO] Reloading
[ERROR] Restart failed: Corefile:5 - Error during parsing: Unknown directive 'bdhfhdhj'
[ERROR] SIGUSR1: starting with listener file descriptors: Corefile:5 - Error during parsing: Unknown directive 'bdhfhdhj'
~~~
After which the curl still works.
This also needed a change to reset the channel used for the metrics
go-routine which gets closed on shutdown, otherwise you'll see:
~~~
^C[INFO] SIGINT: Shutting down
panic: close of closed channel
goroutine 90 [running]:
github.com/coredns/coredns/plugin/health.(*health).OnFinalShutdown(0xc000089bc0, 0xc000063d88, 0x4afe6d)
~~~
Signed-off-by: Miek Gieben <miek@miek.nl>
Small, trivial cleanup: got triggered because I saw a comment on how
health plugins polls other plugins which isn't true.
* Remove useless newHealth function
* healthParse -> parse
* Remove useless constants
Net deletion of code.
Signed-off-by: Miek Gieben <miek@miek.nl>
* plugin/health: remove ability to poll other plugins
This mechanism defeats the purpose any plugin (mostly) caching can still
be alive, we can probably forward queries still. Don't poll plugins,
just tell the world we're up and running.
It was only actually used in kubernetes; and there specifically would
mean any network hiccup would NACK the entire server health.
Fixes: #2534
Signed-off-by: Miek Gieben <miek@miek.nl>
* update docs based on feedback
Signed-off-by: Miek Gieben <miek@miek.nl>
Prevent future; "remove trailing whitespace" PR, but adding a simple
presubmit that checks for this.
This presubmit flagged quite some offenders, remove all trailing
whitespace from. Apart from that there aren't any other changes.
Signed-off-by: Miek Gieben <miek@miek.nl>
* plugin/health: update README
Make more clear in the readme that health is limited to 1 server.
Fixes#1722
* rephrase and remove ~~~ corefile because it will fail
* reload: use OnRestart
Close the listener on OnRestart for health and metrics so the default
setup function can setup the listener when the plugin is "starting up".
Lightly test with some SIGUSR1-ing. Also checked the reload plugin with
this, seems fine:
.com.:1043
.:1043
2018/04/20 15:01:25 [INFO] CoreDNS-1.1.1
2018/04/20 15:01:25 [INFO] linux/amd64, go1.10,
CoreDNS-1.1.1
linux/amd64, go1.10,
2018/04/20 15:01:25 [INFO] Running configuration MD5 = aa8b3f03946fb60546ca1f725d482714
2018/04/20 15:02:01 [INFO] Reloading
2018/04/20 15:02:01 [INFO] Running configuration MD5 = b34a96d99e01db4015a892212560155f
2018/04/20 15:02:01 [INFO] Reloading complete
^C2018/04/20 15:02:06 [INFO] SIGINT: Shutting down
With this corefile:
.com {
proxy . 127.0.0.1:53
prometheus :9054
whoami
reload
}
. {
proxy . 127.0.0.1:53
prometheus :9054
whoami
reload
}
The prometheus port was 9053, changed that to 54 so reload would pick it
up.
From a cursory look it seems this also fixes:
Fixes#1604#1618#1686#1492
* At least make it test
* Use onfinalshutdown
* reload: add reload test
This test #1604 adn right now fails.
* Address review comments
* Add bug section explaining things a bit
* compile tests
* Fix tests
* fixes
* slightly less crazy
* try to make prometheus setup less confusing
* Use ephermal port for test
* Don't use the listener
* These are shared between goroutines, just use the boolean in the main
structure.
* Fix text in the reload README,
* Set addr to TODO once stopping it
* Morph fturb's comment into test, to test reload and scrape health and
metric endpoint
* plugin/health: make reload work
Remove the once.Do from the startup, so we can re-bind the HTTP
listener. Also clarify the usage of health in multiple server blocks
(this is not the best approach - but there isn't a generic solution at
this point).
Manual tested as we lack testing infra, i.e kill -SIGUSR1 and some
CURLing of the health endpoint.
* Readme test fix
* update
* dont need this
* plugin/health: add lameduck mode
Add a way to configure lameduck more, i.e. set health to false, stop
polling plugins. Then wait for a duration before shutting down. As the
health middleware is configured early on in the plugin list, it will
hold up all other shutdown, meaning we still answer queries.
* Add New
* More tests
* golint
* remove confusing text
* plugin/health: add 'overloaded metrics'
Query our on health endpoint and record (and export as a metric) the
time it takes. The Get has a 5s timeout, that, when reached, will set
the metric duration to 5s. The actually call "I'm I overloaded" is left
to an external entity.
* README
* golint and govet
* and the tests
* Add manual pages
Generate manual pages from the README and extend README with Name and
Description sections.
The generation requires 'ronn' which may not be available. Just check in
all generated manual pages.
Switched health and autopath plugin to allow any plugins to be used instead
of a hardcoded list. I did not switch federation over since it wasn't
obvious that anything other than kubernetes could be used with it.
Fixes#1291
Implement health.Healther in erratic and kubernetes plugin. The
kubernetes' healtcheck is only performed on startup - i.e. turn
healthy after the initial loading.
Erratic follow the drop count: every query%drop turns the healthcheck
unhealthy.
Fixes: #985
* doc update
Go through all README and fix mistakes, extend example and let more
corefile snippets be test for validity.
* Cant use spefic addr in test
* Rename middleware to plugin
first pass; mostly used 'sed', few spots where I manually changed
text.
This still builds a coredns binary.
* fmt error
* Rename AddMiddleware to AddPlugin
* Readd AddMiddleware to remain backwards compat
2017-09-14 09:36:06 +01:00
Renamed from middleware/health/README.md (Browse further)
