|Kevin Day||Dec 31, 2005 9:58 am|
|Kevin Day||Dec 31, 2005 11:31 am|
|Colin Johnston||Dec 31, 2005 11:32 am|
|Gerry Boudreaux||Dec 31, 2005 11:48 am|
|Deepak Jain||Dec 31, 2005 2:18 pm|
|Kevin Day||Dec 31, 2005 4:57 pm|
|Roy||Dec 31, 2005 5:06 pm|
|Jared Mauch||Dec 31, 2005 5:43 pm|
|Fergie||Dec 31, 2005 5:51 pm|
|Chris Adams||Dec 31, 2005 7:15 pm|
|Michael Loftis||Dec 31, 2005 11:29 pm|
|Saku Ytti||Jan 1, 2006 1:05 am|
|Charley Kline||Jan 2, 2006 2:29 pm|
|Subject:||Re: Leap second reminder - Check your NTP|
|From:||Kevin Day (toa...@dragondata.com)|
|Date:||Dec 31, 2005 4:57:22 pm|
Last NTP spam:
I'm by no means an NTP expert, if anyone else is, please pipe up.
About 30 minutes before the leap second should have occurred, several of our systems reported "xntpd: time reset 0.958385 s", which was really strange. They moved the wrong direction, and they did it early. Shortly after, those systems lost ntp association and began drifting. About 10 minutes after midnight all have regained sync. I wasn't checking things that early to see why, it's possible some of our NTP sources started disagreeing on what the correct time was, and would also match what other people have reported off-list, going back as far as 18 hours before midnight.
Several public NTP sources are now indicating a "leap second alarm" (setting the leap bits to 11), which will cause most NTP clients to rule them out as a source. ntp-2.gw.uiuc.edu is an example:
220.127.116.11: Server dropped: Leap not in sync server 18.104.22.168, port 123 stratum 2, precision -19, leap 11, trust 000 refid [22.214.171.124], delay 0.03357, dispersion 0.00049
According to ntpdate, its clock seems to have stopped about 5 minutes before midnight, and hasn't yet recovered.
Other NTP servers haven't cleared their "today is a leap second day" bit, which they should have by now. Some NTP implementations rule out servers that don't agree with what their "master" server thinks the leap second bits should be. My reading of the NTP spec says that at 00:00:00 the leap bits should have been returned to zero. Attempting to sync from one of these servers will produce a "Next leap second occurs at 00:00:00.000 UTC Sun Jan 01 2006" message, but that should be harmless as long as they correct themselves a while before midnight.
Still others have their clocks off by a significant amount(10+ minutes) and think they're still in sync, but since I started typing this email, they all have corrected themselves.
While I can't say anything broke on our network as a result of the leap second, a good percentage of our gear lost NTP sync or had some kind of NTP problem around midnight UTC. You may want to check your NTP status at some point, in case something drifted quite a way off and won't step itself back now because the difference is too great.