How Wind Nearly Took Down Boulder NTP

NTP is one of the most interesting and important, but all too forgotten, protocols that makes the internet tick. Accurate clock synchronization is required for everything ranging from cryptography to business and science. NTP is closely tied around a handful of atomic clocks, some in orbit on GPS satellites, and some in laboratories. So the near-failure of one such atomic clock sparked a rather large, and nerdy, internet debate.

On December 17, 2025, the Colorado front range experience a massive wind storm. The National Center for Atmospheric Reassure in Boulder recorded gusts in excess of 100 mph (about 85 knots or 160 kph). This storm was a real doozy, but gusts this strong are not unheard of in Boulder either. That is no small reason the National Renewable Energy Laboratory (now the National Laboratory of the Rockies) has a wind turbine testing facility in the neighborhood.

High winds and dry weather make for a particularly bad time.
Winds of this nature would not terribly interesting. However, the wind storm brought with it a particularly dangerous red flag warning outside of Boulder, a first for Colorado. Such high fire danger combined with damaged infrastructure prompted the local utility, Xcel Energy, to shut off power for hundreds of thousands of customers starting on December 17. Power was not regained until December 21 for many customers.

This outage came with all sorts of headaches to research institutions across Colorado. Not least of which was the National Institute of Science Technology’s (NIST) Boulder campus which houses a rather precise atomic clock. Due to predicted failure of NIST’s heat exchange, much of the normal monitoring equipment was unavailable to the scientists, further complicating the situation.

As was designed, once utility power failed, backup generators took over. But as the outage dragged on, indications came to the scientists in charge of the atomic clocks at NIST that one of the generators had failed. This prompted scientists to warn against relying on the Boulder NTP sources. The scientists running the clock feared complete failure of the hydrogen source clocks. Such failure would require a lengthy and complex re-start procedure once power was returned in the long term, and complete failure of a stratum one NTP source in the short term.

Further complicating the already bad situation was the fact that due to the dangers involved, the scientists could not reach the campus. So not only could they not confirm with certainty what issues the clocks may be experiencing, but they were unable to shut down the NTP servers. Fortunately, power was returned and the main source clock only drifted by a few microseconds. This is still far too much drift as would be preferred on a clock normally accurate in the range of nanoseconds, but perfectly usable for NTP which is only accurate to within a few milliseconds.

So this prompts the question, if such a key time source had failed, what would have happened? In short, not much. By nature of being so distributed, most servers have multiple NTP sources, often including GPS satellites. However, there would most certainly be any number of servers without multiple NTP sources configured. Websites hosted on such servers would be rendered inaccessible as HTTPS encryption handshakes require synchronized clocks. TOTP passkeys and FIDO hardware authenticators would likewise be unusable as both protocols rely on accurate time sources. So any two computers would be unable to properly execute protocols requiring synchronized time. Beyond the limited failures outlined above, its difficult to say what more the damage could be, but the effects are unlikely to be terribly dramatic.

If harnessing atoms to tell time sparks your interest, make sure to check out this atomic delay clock next! [Jeff Geerling] also has a nice discussion of this power outage that you might like.

hackaday.com/2025/12/29/how-wi…