Page MenuHomeMiraheze

cp41 is out of disk space
Open, NormalPublic

Description

Error 503 Backend fetch failed via cp41.wikitide.net at Fri, 31 May 2024 03:11:29 GMT
Varnish XID 1073705392, serving 210.19.68.50, 127.0.0.1 (your IP!).

See https://grafana.wikitide.net/d/W9MIkA7iz/wikitide-cluster?orgId=1&var-job=node&var-node=cp41.wikitide.net&var-port=9100&viewPanel=152&from=now-2d&to=now-1m

Event Timeline

Now getting 502 Bad Gateway error when trying to navigate to the wiki

RhinosF1 renamed this task from Error 503 Backend fetch failed via cp41 to cp41 is out of disk space.Fri, May 31, 15:32
RhinosF1 triaged this task as Unbreak Now! priority.
RhinosF1 updated the task description. (Show Details)

Root cause is cp41 has no disk space left so it's gone grumpy

RhinosF1 lowered the priority of this task from Unbreak Now! to Normal.Sat, Jun 1, 07:15
RhinosF1 added a subscriber: Reception123.

@Reception123 cleared postfix logs which were full of errors on cp41. Root cause needs review.

Just to note that CosmicAlpha mentioned that postfix is indeed enabled on all servers intentionally as it's used to send out mail to root@ and other internal emails