coredns/plugin/reload
Miek Gieben acbcad7b4e
reload: use OnRestart (#1709)
* reload: use OnRestart

Close the listener on OnRestart for health and metrics so the default
setup function can setup the listener when the plugin is "starting up".

Lightly test with some SIGUSR1-ing. Also checked the reload plugin with
this, seems fine:

.com.:1043
.:1043
2018/04/20 15:01:25 [INFO] CoreDNS-1.1.1
2018/04/20 15:01:25 [INFO] linux/amd64, go1.10,
CoreDNS-1.1.1
linux/amd64, go1.10,
2018/04/20 15:01:25 [INFO] Running configuration MD5 = aa8b3f03946fb60546ca1f725d482714
2018/04/20 15:02:01 [INFO] Reloading
2018/04/20 15:02:01 [INFO] Running configuration MD5 = b34a96d99e01db4015a892212560155f
2018/04/20 15:02:01 [INFO] Reloading complete
^C2018/04/20 15:02:06 [INFO] SIGINT: Shutting down

With this corefile:
.com {
  proxy . 127.0.0.1:53
  prometheus :9054
  whoami
  reload
}

. {
  proxy . 127.0.0.1:53
  prometheus :9054
  whoami
  reload
}

The prometheus port was 9053, changed that to 54 so reload would pick it
up.

From a cursory look it seems this also fixes:
Fixes #1604 #1618 #1686 #1492

* At least make it test

* Use onfinalshutdown

* reload: add reload test

This test #1604 adn right now fails.

* Address review comments

* Add bug section explaining things a bit

* compile tests

* Fix tests

* fixes

* slightly less crazy

* try to make prometheus setup less confusing

* Use ephermal port for test

* Don't use the listener

* These are shared between goroutines, just use the boolean in the main
  structure.
* Fix text in the reload README,
* Set addr to TODO once stopping it
* Morph fturb's comment into test, to test reload and scrape health and
  metric endpoint
2018-04-21 17:43:02 +01:00
..
OWNERS Add OWNERS file (#1486) 2018-02-08 10:55:51 +00:00
README.md reload: use OnRestart (#1709) 2018-04-21 17:43:02 +01:00
reload.go Update all plugins to use plugin/pkg/log (#1694) 2018-04-19 07:41:56 +01:00
setup.go add sync for proper termination (#1507) 2018-02-08 13:57:49 -05:00
setup_test.go Plugin/RELOAD - Tune usage of var global, add limit to options (#1457) 2018-02-02 13:15:56 -05:00

reload

Name

reload - allows automatic reload of a changed Corefile

Description

This plugin periodically checks if the Corefile has changed by reading it and calculating its MD5 checksum. If the file has changed, it reloads CoreDNS with the new Corefile. This eliminates the need to send a SIGHUP or SIGUSR1 after changing the Corefile.

The reloads are graceful - you should not see any loss of service when the reload happens. Even if the new Corefile has an error, CoreDNS will continue to run the old config and an error message will be printed to the log. But see the Bugs section for failure modes.

In some environments (for example, Kubernetes), there may be many CoreDNS instances that started very near the same time and all share a common Corefile. To prevent these all from reloading at the same time, some jitter is added to the reload check interval. This is jitter from the perspective of multiple CoreDNS instances; each instance still checks on a regular interval, but all of these instances will have their reloads spread out across the jitter duration. This isn't strictly necessary given that the reloads are graceful, and can be disabled by setting the jitter to 0s.

Jitter is re-calculated whenever the Corefile is reloaded.

This plugin can only be used once per Server Block.

Syntax

reload [INTERVAL] [JITTER]
  • The plugin will check for changes every INTERVAL, subject to +/- the JITTER duration
  • INTERVAL and JITTER are Golang (durations)[https://golang.org/pkg/time/#ParseDuration]
  • Default INTERVAL is 30s, default JITTER is 15s
  • Minimal value for INTERVAL is 2s, and for JITTER is 1s
  • If JITTER is more than half of INTERVAL, it will be set to half of INTERVAL

Examples

Check with the default intervals:

. {
    reload
    erratic
}

Check every 10 seconds (jitter is automatically set to 10 / 2 = 5 in this case):

. {
    reload 10s
    erratic
}

Bugs

The reload happens without data loss (i.e. DNS queries keep flowing), but there is a corner case where the reload fails, and you loose functionality. Consider the following Corefile:

. {
	health :8080
	whoami
}

CoreDNS starts and serves health from :8080. Now you change :8080 to :443 not knowing a process is already listening on that port. The process reloads and performs the following steps:

  1. close the listener on 8080
  2. reload and parse the config again
  3. fail to start a new listener on 443
  4. fail loading the new Corefile, abort and keep using the old process

After the aborted attempt to reload we are left with the old proceses running, but the listener is closed in step 1; so the health endpoint is broken. The same can hopen in the prometheus metrics plugin.

In general be careful with assigning new port and expecting reload to work fully.