Page MenuHomeMiraheze
Create Task
Maniphest T12538

4 Sept PowerDNS Outage across on some servers
Closed, ResolvedPublic
Actions

Related Objects
Search...

Event Timeline

RhinosF1 created this task.Wed, Sep 4, 06:55
RhinosF1 triaged this task as Unbreak Now! priority.
Herald added subscribers: Waki285, BrandonWM. · View Herald TranscriptWed, Sep 4, 06:55
RhinosF1 added projects: Technology-Team (Infrastructure), Cloud Infrastructure, DNS.Wed, Sep 4, 06:55
Void removed Void as the assignee of this task.Wed, Sep 4, 07:49
Void subscribed.

Tried rebooting some servers, but that doesn't seem to be a lasting solution. Not sure what the exact problem is. I suspect something network related, but all I really have to go off of is:

Sep  4 05:42:49 swiftproxy171 pdns-recursor[608]: msg="Failed to update . records" error="Too much time waiting for .|NS, timeouts: 5, throttles: 0, queries: 6, 7508msec" subsystem="housekeeping" level="0" prio="Error" tid="0" ts="1725428569.875" exception="ImmediateServFailException"
Sep  4 05:42:49 swiftproxy171 pdns-recursor[608]: msg="Failed to update . records" subsystem="housekeeping" level="0" prio="Warning" tid="0" ts="1725428569.875" rcode="-1"
Void added subscribers: Universal_Omega, OrangeStar.Wed, Sep 4, 17:55

Finished rebooting affected servers and we seem to be all up now. Barring a proper post-mortem, this is hopefully resolved. cc @OrangeStar @Universal_Omega

Void added a comment.Wed, Sep 4, 22:08

Tentatively resolved again, waiting for confirmation from the datacenter on what exactly happened. They did say it was resolved, but I would like to continue monitoring for a bit to ensure services don't go back down.

RhinosF1 lowered the priority of this task from Unbreak Now! to High.Thu, Sep 5, 07:25
OrangeStar closed this task as Resolved.Fri, Sep 6, 11:19

@Void yep, this is resolved and we have an explanation on what happened from FiberState.

It was related to neighbor discovery on one of our distribution switches. We've made a change to the VLAN that should take care of it.

Closing this.

OrangeStar added a comment.Fri, Sep 6, 11:21
In T12538#250747, @Void wrote:

Tried rebooting some servers, but that doesn't seem to be a lasting solution. Not sure what the exact problem is. I suspect something network related, but all I really have to go off of is:

Sep  4 05:42:49 swiftproxy171 pdns-recursor[608]: msg="Failed to update . records" error="Too much time waiting for .|NS, timeouts: 5, throttles: 0, queries: 6, 7508msec" subsystem="housekeeping" level="0" prio="Error" tid="0" ts="1725428569.875" exception="ImmediateServFailException"
Sep  4 05:42:49 swiftproxy171 pdns-recursor[608]: msg="Failed to update . records" subsystem="housekeeping" level="0" prio="Warning" tid="0" ts="1725428569.875" rcode="-1"

This is likely due to those servers losing IPv6 connectivity and thus being unable to query the DNS.