NTP
All servers are time-synchronised using the NTP protocol. In pmtpa, this is managed by Puppet, with the ntp::client class in ntp.pp. The NTP servers are dobson and linne.
Testing
Testing is very important, as the November 2005 event demonstrated. It's easy to test whether ntpd is working.
/usr/sbin/ntpq -p
Here is the output from a happy server:
remote refid st t when poll reach delay offset jitter ============================================================================== *vl-2-0.csw1-pmt albert.pmtpa.wm 3 u 43 64 177 0.486 1.155 1.135
Note the asterisk in the first column, that tells you it's happy. It's synchronised to csw1, which is on stratum (st) 3, and the refid gives the stratum 2 server. The other important columns are:
- when: this tells you how long ago it received a response from the server, in this case 43 seconds
- offset: this tells you how far off the clock is, in milliseconds.
Here is the output from a server which is on its way to synchronisation:
remote refid st t when poll reach delay offset jitter ============================================================================== vl-2-0.csw1-pmt albert.pmtpa.wm 3 u 81 1024 7 0.573 -203.77 1.635
There's no asterisk, which means it hasn't synchronised yet. The offset is substantial, so it will take a while to get into sync. The fact that the remote, refid, st and when columns are reasonable tells you that it is actually working. Hopefully we check back later, offset should be smaller.
Here is the output from a completely broken server:
remote refid st t when poll reach delay offset jitter ============================================================================== vl-2-0.csw1-pmt .B▒▒. 16 u 16 64 0 0.000 0.000 4000.00
It seems to know what server it's meant to be reading from, but the other columns are just silly. There's no such thing as stratum 16, and I'm quite sure the network delay is meant to be more than zero. If you see something like this, you need to fix it.
ntpq can be run remotely. The output of ntpq -c peers csw1-pmtpa currently shows:
remote refid st t when poll reach delay offset jitter ============================================================================== 207.142.131.255 0.0.0.0 16 u - 64 0 0.000 0.000 16000.0 10.0.255.255 0.0.0.0 16 u - 64 0 0.000 0.000 16000.0 clock2.redhat.c .CDMA. 1 - 18d 64 0 73.200 1.322 16000.0 ntp-s1.cise.ufl 85.83.78.79 16 - 18d 1024 0 18.140 0.673 16000.0 raptor.tera-byt 0.0.0.0 16 - - 1024 0 0.000 0.000 16000.0 *albert.pmtpa.wm ntp-s1.cise.ufl 2 u 45 64 377 0.790 -3.597 0.400
Three broken external servers, two broadcast domains and albert, which is a working stratum 2 server. Finally, albert gives:
remote refid st t when poll reach delay offset jitter ============================================================================== *ntp-s1.cise.ufl .USNO. 1 u 86 128 377 22.090 12.095 5.867 +ip-207-145-113- .GPS. 1 u 89 128 377 76.639 20.227 5.468 +solarnet.ru hora.cs.tu-berl 2 u 92 128 377 180.877 11.987 3.481 -blah.jabber.dk ntp2.sth.netnod 2 u 97 128 377 135.076 22.336 5.524 LOCAL(0) LOCAL(0) 10 l 12 64 377 0.000 0.000 0.001
A nearby stratum 1 server at the University of Florida is selected as the reference, but I've configured three other servers from pool.ntp.org in case that one goes down. Two of them are contributing to the averaging process, the third is ignored because its clock doesn't agree with the others. If all 4 are unreachable, the local clock will be used. It's currently undesirable because it's been declared stratum 10.
Monitoring
Service Restart Check
A failure of this check indicates that the configuration file for ntp (ntp.conf) was changed but the service itself was not restarted. A CRITICAL alert is raised if the time delta between the configuration file change and service restart exceeds two (2) hours.
This check is meant to be a warning alert and does not signify an error in the service but should be corrected immediately by restarting ntpd.service. If you are unsure about the changes, please contact the Traffic team.
You can do so by using the cookbook: sre.dns.ntp-roll-restart. Simply run it by passing the relevant query/alias (such as --alias dnsbox) and it will automatically restart the service with sane defaults.