Page MenuHomeMiraheze

Redis-JobRunnerProject
ActivePublic

Members

  • This project does not have any members.
  • View All

Watchers

  • This project does not have any watchers.
  • View All

Details

Description

This project is for the Redis-based JobRunner service we operate.

Recent Activity

Tue, Mar 26

Universal_Omega removed hashtags from Redis-JobRunner: #jobrunner, #jobchron.
Tue, Mar 26, 16:33

Feb 1 2024

Universal_Omega closed T11458: Operate redis as two instances for jobrunner as Declined.

For now, I don't believe this is needed, but if it becomes apparent that is we can do so eventually.

Feb 1 2024, 11:50 · Redis-JobRunner, Infrastructure (SRE)

Jan 30 2024

Universal_Omega added a project to T11458: Operate redis as two instances for jobrunner: Redis-JobRunner.
Jan 30 2024, 17:21 · Redis-JobRunner, Infrastructure (SRE)

Jul 9 2023

Paladox closed T11046: Poor Redis performance since ~10:40 8 July as Resolved.

The update I did to jobrunner somehow broke the jobchron. I pulled in changes from upstream (starting as new) that broke things somehow.

Jul 9 2023, 11:39 · Infrastructure (SRE), Redis-JobRunner, MediaWiki (SRE)
RhinosF1 added a comment to T11046: Poor Redis performance since ~10:40 8 July.

@Paladox has made changes and it seems to have solved the issues.

Jul 9 2023, 11:25 · Infrastructure (SRE), Redis-JobRunner, MediaWiki (SRE)
Tali64 added a comment to T11046: Poor Redis performance since ~10:40 8 July.

Most of the wikis that have been created since redis was restarted have had no issues, but there was one that was created very recently that threw an error: https://meta.miraheze.org/wiki/Special:RequestWikiQueue/33416

Jul 9 2023, 04:24 · Infrastructure (SRE), Redis-JobRunner, MediaWiki (SRE)

Jul 8 2023

RhinosF1 renamed T11046: Poor Redis performance since ~10:40 8 July from CreateWiki throwing exceptions to Poor Redis performance since ~10:40 8 July.
Jul 8 2023, 16:26 · Infrastructure (SRE), Redis-JobRunner, MediaWiki (SRE)
RhinosF1 lowered the priority of T11046: Poor Redis performance since ~10:40 8 July from Unbreak Now! to Normal.

Nothing is exploding although commands/sec (and misses) are still elevated

Jul 8 2023, 16:25 · Infrastructure (SRE), Redis-JobRunner, MediaWiki (SRE)
RhinosF1 assigned T11046: Poor Redis performance since ~10:40 8 July to Paladox.

Redis was restarted. That may help.

Jul 8 2023, 15:41 · Infrastructure (SRE), Redis-JobRunner, MediaWiki (SRE)
RhinosF1 raised the priority of T11046: Poor Redis performance since ~10:40 8 July from Normal to Unbreak Now!.

Performance is much worse last day, wiki creations are fairly critical. Declaring UBN.

Jul 8 2023, 15:21 · Infrastructure (SRE), Redis-JobRunner, MediaWiki (SRE)

May 19 2023

MacFan4000 removed a member for Redis-JobRunner: John.
May 19 2023, 19:59

Aug 12 2021

Unknown Object (User) moved T7627: runJobs.php (via JobRunner) should not be able to cause load issues from To Triage to Bugs on the Redis-JobRunner board.
Aug 12 2021, 08:14 · MediaWiki (SRE), MediaWiki, Security
Unknown Object (User) moved T7627: runJobs.php (via JobRunner) should not be able to cause load issues from Backlog to Bugs on the MediaWiki board.
Aug 12 2021, 08:14 · MediaWiki (SRE), MediaWiki, Security

Jul 30 2021

RhinosF1 added a comment to T7627: runJobs.php (via JobRunner) should not be able to cause load issues.

Why is this a security task?

Jul 30 2021, 16:45 · MediaWiki (SRE), MediaWiki, Security

Jul 28 2021

Unknown Object (User) added a comment to T7627: runJobs.php (via JobRunner) should not be able to cause load issues.

Why is this a security task?

Jul 28 2021, 22:59 · MediaWiki (SRE), MediaWiki, Security

Jul 21 2021

Void added a comment to T7627: runJobs.php (via JobRunner) should not be able to cause load issues.

Probably not, I don't know if we can establish if this was caused by the job spawning multiple processes. Still worth investigating.

Jul 21 2021, 02:45 · MediaWiki (SRE), MediaWiki, Security

Jul 15 2021

Unknown Object (User) moved T7627: runJobs.php (via JobRunner) should not be able to cause load issues from Backlog to Short Term on the MediaWiki (SRE) board.
Jul 15 2021, 22:06 · MediaWiki (SRE), MediaWiki, Security

Jul 13 2021

RhinosF1 added a parent task for T7627: runJobs.php (via JobRunner) should not be able to cause load issues: T7633: Persistent resource consumption is causing all sorts.
Jul 13 2021, 07:32 · MediaWiki (SRE), MediaWiki, Security

Jul 11 2021

RhinosF1 added a comment to T7627: runJobs.php (via JobRunner) should not be able to cause load issues.

There's a --procs setting we can use? Should that be 1?

Jul 11 2021, 23:27 · MediaWiki (SRE), MediaWiki, Security
RhinosF1 created T7627: runJobs.php (via JobRunner) should not be able to cause load issues.
Jul 11 2021, 23:24 · MediaWiki (SRE), MediaWiki, Security

May 27 2021

John added a comment to T7373: Investigate cause of redis server error (socket error on read socket) when CreateWiki Extension creates a wiki.

Redis-JobRunner is additional software. It’s like saying MediaWiki is the cause of Matomo’s SSL certificate failing just because they have the same certificate - they’re entirely unrelated but confirmation bias suggests there’s a link because it’s easier to explain than an unknown cause.

May 27 2021, 06:21 · Universal Omega, MediaWiki (SRE), MediaWiki
RhinosF1 added a comment to T7373: Investigate cause of redis server error (socket error on read socket) when CreateWiki Extension creates a wiki.

Actually this may be a different error. I remember it first happened when John did work to Redis-JobRunner, so this must be a different error I guess.

Note: On this occasion, we got an actual error in the description whereas T7338 was a silent fail.

May 27 2021, 05:57 · Universal Omega, MediaWiki (SRE), MediaWiki
Dmehus added a comment to T7373: Investigate cause of redis server error (socket error on read socket) when CreateWiki Extension creates a wiki.

Actually this may be a different error. I remember it first happened when John did work to Redis-JobRunner, so this must be a different error I guess.

May 27 2021, 05:17 · Universal Omega, MediaWiki (SRE), MediaWiki
Dmehus moved T7373: Investigate cause of redis server error (socket error on read socket) when CreateWiki Extension creates a wiki from Backlog to Bugs on the CreateWiki board.
May 27 2021, 05:15 · Universal Omega, MediaWiki (SRE), MediaWiki
Dmehus moved T7373: Investigate cause of redis server error (socket error on read socket) when CreateWiki Extension creates a wiki from Unsorted to Short Term on the Universal Omega board.
May 27 2021, 05:15 · Universal Omega, MediaWiki (SRE), MediaWiki
Dmehus added a project to T7373: Investigate cause of redis server error (socket error on read socket) when CreateWiki Extension creates a wiki: CreateWiki.
May 27 2021, 05:15 · Universal Omega, MediaWiki (SRE), MediaWiki
Dmehus added a comment to T7373: Investigate cause of redis server error (socket error on read socket) when CreateWiki Extension creates a wiki.

I think this would be Infrastructure (SRE) task if it has to do with Redis-JobRunner causing the issue though not certain whether infrastructure or ourselves should handle/investigate this so leaving as is for now.

May 27 2021, 05:14 · Universal Omega, MediaWiki (SRE), MediaWiki
Unknown Object (User) added a comment to T7373: Investigate cause of redis server error (socket error on read socket) when CreateWiki Extension creates a wiki.

I think this would be Infrastructure (SRE) task if it has to do with Redis-JobRunner causing the issue though not certain whether infrastructure or ourselves should handle/investigate this so leaving as is for now.

May 27 2021, 03:08 · Universal Omega, MediaWiki (SRE), MediaWiki
Unknown Object (User) reopened T7373: Investigate cause of redis server error (socket error on read socket) when CreateWiki Extension creates a wiki as "Open".

Actually this may be a different error. I remember it first happened when John did work to Redis-JobRunner, so this must be a different error I guess.

May 27 2021, 02:58 · Universal Omega, MediaWiki (SRE), MediaWiki
Unknown Object (User) merged task T7373: Investigate cause of redis server error (socket error on read socket) when CreateWiki Extension creates a wiki into T7338: Investigate cause of wiki being created but not creation farmer log entry being created.
May 27 2021, 02:56 · Universal Omega, MediaWiki (SRE), MediaWiki
Dmehus moved T7373: Investigate cause of redis server error (socket error on read socket) when CreateWiki Extension creates a wiki from To Triage to Bugs on the Redis-JobRunner board.
May 27 2021, 02:42 · Universal Omega, MediaWiki (SRE), MediaWiki
Dmehus added a comment to T7373: Investigate cause of redis server error (socket error on read socket) when CreateWiki Extension creates a wiki.

Removed Production Error and added Redis-JobRunner as I'm guessing the former was not correct.

May 27 2021, 02:42 · Universal Omega, MediaWiki (SRE), MediaWiki
Dmehus edited projects for T7373: Investigate cause of redis server error (socket error on read socket) when CreateWiki Extension creates a wiki, added: Redis-JobRunner; removed Production Error.
May 27 2021, 02:41 · Universal Omega, MediaWiki (SRE), MediaWiki

Apr 11 2021

RhinosF1 created T7127: Add more jobrunner rate tasks to Grafana.
Apr 11 2021, 17:07 · MediaWiki (SRE), Monitoring
John closed T7108: Remove abandoned l-unclaimed entries as Resolved.

https://github.com/miraheze/jobrunner-service/compare/de7d72b68abc...7e6175d56b4e

Apr 11 2021, 15:02 · Redis-JobRunner, Infrastructure (SRE)

Apr 8 2021

John closed T7107: Remove :rootjobs: periodically as Resolved.
Apr 8 2021, 11:26 · Redis-JobRunner, Infrastructure (SRE)
John moved T7107: Remove :rootjobs: periodically from Incoming to Short Term on the Infrastructure (SRE) board.
Apr 8 2021, 11:21 · Redis-JobRunner, Infrastructure (SRE)
John moved T7108: Remove abandoned l-unclaimed entries from Incoming to Short Term on the Infrastructure (SRE) board.
Apr 8 2021, 11:21 · Redis-JobRunner, Infrastructure (SRE)
Reception123 triaged T7112: JobQueueError from line 778 of /srv/mediawiki/w/includes/jobqueue/JobQueueRedis.php: Redis server error: socket error on read socket as High priority.
Apr 8 2021, 07:24 · Infrastructure (SRE)

Apr 7 2021

John moved T7108: Remove abandoned l-unclaimed entries from To Triage to Bugs on the Redis-JobRunner board.
Apr 7 2021, 20:31 · Redis-JobRunner, Infrastructure (SRE)
John moved T7107: Remove :rootjobs: periodically from To Triage to Features on the Redis-JobRunner board.
Apr 7 2021, 20:31 · Redis-JobRunner, Infrastructure (SRE)
John triaged T7108: Remove abandoned l-unclaimed entries as Normal priority.
Apr 7 2021, 20:31 · Redis-JobRunner, Infrastructure (SRE)
John triaged T7107: Remove :rootjobs: periodically as Low priority.
Apr 7 2021, 20:26 · Redis-JobRunner, Infrastructure (SRE)
John set the image for Redis-JobRunner to F1420607: fa-briefcase-blue.png.
Apr 7 2021, 20:20
John created Redis-JobRunner.
Apr 7 2021, 20:20