Page MenuHomeMiraheze

Phabricator /tmp/ full error keeps recurring
Closed, ResolvedPublic

Description

We periodically get a message like this (like @revi got last night):

Unable to open lock '/tmp/phabricator-setup/setup-24878.1483103237.cache.lock' for writing!

And then I guess someone goes in and empties /tmp/? Is there any way we can prevent this from happening in the future?

Event Timeline

@John said there isn't really. I'm thinking about a cron job which would just rm -rf every two weeks but I'm not sure if there would be any issues with that or not.

It won't work. The bulk of them are generated within a 10 minute span when it gets full. We already have a cron doing it weekly that SPF committed.

@John What exactly is generated? Can't this be stopped or half of them get redirected to another place?

root@misc1:/tmp/phabricator-server# cat setup-9953.1483285350.cache
a:1:{s:21:"phabricator.in-flight";a:1:{s:3:"val";i:1;}}
root@misc1:/tmp/phabricator-setup# cat setup-9953.1483285350.cache
a:3:{s:49:"phabricator:cluster.db.health(81.4.125.112, 3306)";a:1:{s:3:"val";a:3:{s:2:"up";b:1;s:9:"lastCheck";d:1483285363.6285839;s:3:"log";a:2:{i:0;a:2:{s:9:"timestamp";d:1483285350.740164;s:2:"up";b:1;}i:1;a:2:{s:9:"timestamp";d:1483285363.6393931;s:2:"up";b:1;}}}}s:40:"phabricator:phabricator.setup.issue-keys";a:1:{s:3:"val";a:0:{}}s:42:"phabricator:phabricator.setup.needs-repair";a:1:{s:3:"val";b:1;}}

https://github.com/phacility/phabricator/blob/125fb332de722f769e330c37c73ae499fb167cbe/src/applications/cache/PhabricatorCaches.php#L253-L274 seems to be the issue.

It states it's APC but it downgrades to on-disk if APC isn't available. This then sets the keys (in /tmp/) and the code in use seems to deleteKey, but the keys aren't deleted.

John claimed this task.

Per a lot of advice, I've switched phabricator to use APC cache instead of disk based caching. Which when you think about it makes sense. As such, this issue is no long reproducible to me.