outage 2017-01-16

January 16, 2017 posted by Taylor R Campbell

[Update, 2017-01-16 20:48 UTC: All issues have been resolved and should be functioning normally now.]

As some of you may have noticed, the DNS records are broken right now. Two independent incidents caused this to happen:

  1. UC Berkeley stopped providing secondary DNS service for at and on a few hours notice, before we had time to update the NS delegations in the ORG. zone. [Update, 2017-01-16 20:48 UTC: The Berkeley network administrators have graciously restored service until we can update our NS delegations. Thanks, Berkeley—for doing this on a US holiday, no less!]
  2. The network hosting our primary nameserver went offline at around 14:00 UTC for reasons unknown. [Update, 2017-01-16 20:48 UTC: The network and are back up.]

We have sent a request to our registrar to update the delegations to make primary and secondary. We asked them to expedite the process, but owing to the weekend, when the humans who process these requests are off work, it may take another day for that to take effect, and another day after that for the cached TTLs on the old NS delegations to Berkeley's nameservers to expire.

As a provisional stop gap, if you need access to the zone and you can prime your DNS cache with manually entered records, you can prime it with the following records to restore service at your site:     86400   IN      NS     86400   IN      NS  86400   IN      A  86400   IN      AAAA    2001:470:a085:999::53

(Presumably if you are reading this you have some DNS records in the zone cached, but they will eventually expire and you may need some others.)

[1 comment]



It should be noted that Berkeley's NOC was checking email on a national holiday, and got the zone live again once we contacted them. They said the original deletion was a mistake. We appreciate their quick response to get this sorted out.

Posted by David Maxwell on January 16, 2017 at 08:06 PM UTC #

Post a Comment:
Comments are closed for this entry.