Page MenuHomeMiraheze

Backups remain stored locally on db112; causing disk space full
Closed, DuplicatePublic

Description

Instead of being transferred to the backups server (or maybe they have and just not been automatically deleted locally), it seems the backups have remained locally on db112 causing the disk to be full every few hours and therefore render Matomo and Phabricator inaccessible unless someone deletes binlogs every few hours.

I'm not sure whether they've just not been deleted locally but have been actually backed up or if the backup didn't complete because of lack of space (which I suspect).

Leaving High as the current situation (down every few hours) is but if we decide to delete the backups for phab/matomo for now then this can be downgraded until a more permanent fix is found to for local storage (split them into two)?

Event Timeline

Reception123 created this task.
John claimed this task.

The backups are automatically deleted - backup/dbs was full of backups from early 2022, before the backup system even existed.

Oh, it never crossed my mind to check the timestamp as I didn't even imagine 2022 backups would be there.

Unknown Object (User) reopened this task as Open.Feb 24 2023, 06:33

This is the case, as it was the cause again today. In order to get it back up, I had to delete the piwik backups from db112.

Here is what it was before:

-rw-r--r-- 1 root root 26625302528 Feb 24 06:10 piwik.backup

So the idea itself is invalid - backups don't remain locally stored, the server runs out of space, causing the backup to fail.