Page MenuHomeMiraheze

Goal-2021-Jan-JunGoal
ArchivedPublic

Members (1)

Watchers

  • This project does not have any watchers.
  • View All

Details

Description

Miraheze's Goals for the first half of 2021 (January 1st ~ June 30th).

Tasks added to this project must be at least one of the following:

a long-term project that can realistically be finished before June 31st,
a development project that will have a positive impact on Miraheze communities,
an objective for Site Reliability Engineers (infrastructure, introducing a new service, major work etc.).
MediaWiki and development tasks can be added freely by anyone assuming someone has in the past hinted at the possibility they would be willing to work on it or have a noticeable impact on communities.

Site Reliability Engineering tasks should only be added by members of the Site Reliability Engineer team as they have full control over their priorities and workflow, not subject to community benefit.

< Goal-2020-Jul-Dec | Goal-2021-Jul-Dec >

Recent Activity

Jun 25 2022

Paladox closed T5044: Setup centralised logging for services as Resolved.

Resolved

Jun 25 2022, 15:54 · Monitoring, Goal-2022-Jan-Jun, Goal-2021-Jul-Dec, Infrastructure (SRE), Goal-2021-Jan-Jun, Goal-2020-Jul-Dec, Goal-2020-Jan-Jun
Paladox updated the task description for T5044: Setup centralised logging for services.
Jun 25 2022, 15:54 · Monitoring, Goal-2022-Jan-Jun, Goal-2021-Jul-Dec, Infrastructure (SRE), Goal-2021-Jan-Jun, Goal-2020-Jul-Dec, Goal-2020-Jan-Jun
John added a comment to T5044: Setup centralised logging for services.

@Paladox less than a week until end of goal period - do we have an update on this?

Jun 25 2022, 13:02 · Monitoring, Goal-2022-Jan-Jun, Goal-2021-Jul-Dec, Infrastructure (SRE), Goal-2021-Jan-Jun, Goal-2020-Jul-Dec, Goal-2020-Jan-Jun

May 9 2022

Unknown Object (User) moved T5044: Setup centralised logging for services from Backlog to Central Logging on the Monitoring board.
May 9 2022, 19:26 · Monitoring, Goal-2022-Jan-Jun, Goal-2021-Jul-Dec, Infrastructure (SRE), Goal-2021-Jan-Jun, Goal-2020-Jul-Dec, Goal-2020-Jan-Jun
Unknown Object (User) added a project to T5044: Setup centralised logging for services: Monitoring.
May 9 2022, 19:26 · Monitoring, Goal-2022-Jan-Jun, Goal-2021-Jul-Dec, Infrastructure (SRE), Goal-2021-Jan-Jun, Goal-2020-Jul-Dec, Goal-2020-Jan-Jun

Apr 16 2022

Paladox updated the task description for T5044: Setup centralised logging for services.
Apr 16 2022, 23:18 · Monitoring, Goal-2022-Jan-Jun, Goal-2021-Jul-Dec, Infrastructure (SRE), Goal-2021-Jan-Jun, Goal-2020-Jul-Dec, Goal-2020-Jan-Jun

Feb 21 2022

Paladox claimed T5044: Setup centralised logging for services.
Feb 21 2022, 15:09 · Monitoring, Goal-2022-Jan-Jun, Goal-2021-Jul-Dec, Infrastructure (SRE), Goal-2021-Jan-Jun, Goal-2020-Jul-Dec, Goal-2020-Jan-Jun
Paladox updated the task description for T5044: Setup centralised logging for services.
Feb 21 2022, 14:46 · Monitoring, Goal-2022-Jan-Jun, Goal-2021-Jul-Dec, Infrastructure (SRE), Goal-2021-Jan-Jun, Goal-2020-Jul-Dec, Goal-2020-Jan-Jun

Jan 1 2022

John added a comment to T5044: Setup centralised logging for services.

This task has taken a back foot, over other work which has higher priority currently such as T8469 T8350

Jan 1 2022, 10:45 · Monitoring, Goal-2022-Jan-Jun, Goal-2021-Jul-Dec, Infrastructure (SRE), Goal-2021-Jan-Jun, Goal-2020-Jul-Dec, Goal-2020-Jan-Jun
Unknown Object (User) moved T5044: Setup centralised logging for services from Backlog to Infrastructure on the Goal-2022-Jan-Jun board.
Jan 1 2022, 03:22 · Monitoring, Goal-2022-Jan-Jun, Goal-2021-Jul-Dec, Infrastructure (SRE), Goal-2021-Jan-Jun, Goal-2020-Jul-Dec, Goal-2020-Jan-Jun
Unknown Object (User) added a project to T5044: Setup centralised logging for services: Goal-2022-Jan-Jun.
Jan 1 2022, 03:15 · Monitoring, Goal-2022-Jan-Jun, Goal-2021-Jul-Dec, Infrastructure (SRE), Goal-2021-Jan-Jun, Goal-2020-Jul-Dec, Goal-2020-Jan-Jun

Dec 8 2021

Unknown Object (User) closed T7052: Make actual updates clearer on github rather than just existance as Resolved.

https://github.com/miraheze/mediawiki/blob/REL1_37/.github/workflows/label.yml
confirmed to work on https://github.com/miraheze/mediawiki/pull/4440

Dec 8 2021, 00:22 · Upstream, Universal Omega, SRE Automation, Goal-2021-Jan-Jun, Goal-2021-Jul-Dec, MediaWiki (SRE)

Dec 7 2021

Unknown Object (User) moved T7052: Make actual updates clearer on github rather than just existance from Unsorted to Short Term on the Universal Omega board.
Dec 7 2021, 22:41 · Upstream, Universal Omega, SRE Automation, Goal-2021-Jan-Jun, Goal-2021-Jul-Dec, MediaWiki (SRE)
Unknown Object (User) added a comment to T7052: Make actual updates clearer on github rather than just existance.

We can also add labels with:

pull_number=$(jq --raw-output .pull_request.number "$GITHUB_EVENT_PATH")
curl \
  -X POST \
  -H "Accept: application/vnd.github.v3+json" \
  -H 'Authorization: token ${{ secrets.GITHUB_TOKEN }}' \
  https://api.github.com/repos/${GITHUB_REPOSITORY}/issues/$pull_number/labels \
  -d '{"labels":["build"]}'
Dec 7 2021, 02:40 · Upstream, Universal Omega, SRE Automation, Goal-2021-Jan-Jun, Goal-2021-Jul-Dec, MediaWiki (SRE)
Unknown Object (User) added a comment to T7052: Make actual updates clearer on github rather than just existance.

And we can parse the list of files with:

git -c color.diff=false diff --submodule=diff $submodule | sed -n -e s/^diff\ --git\ a\\///p
Dec 7 2021, 01:36 · Upstream, Universal Omega, SRE Automation, Goal-2021-Jan-Jun, Goal-2021-Jul-Dec, MediaWiki (SRE)
Unknown Object (User) added a comment to T7052: Make actual updates clearer on github rather than just existance.

I have this to parse the dependabot branches into which submodule to check against, and properly get the full diff of them. I still don't know how to use this, convert it into a list of only files changed (as --name-only won't work with git diff for submodules). If we can convert it to a simple list somehow, then we still need to figure out how to use that list, add some regex checks on it, and add a label to it, to make it clearer what changes are made.

Dec 7 2021, 00:47 · Upstream, Universal Omega, SRE Automation, Goal-2021-Jan-Jun, Goal-2021-Jul-Dec, MediaWiki (SRE)

Dec 6 2021

Unknown Object (User) added a comment to T7052: Make actual updates clearer on github rather than just existance.

https://github.com/srvaroa/labeler/issues/35 was filed by you so hopefully this might get a better response

Dec 6 2021, 07:46 · Upstream, Universal Omega, SRE Automation, Goal-2021-Jan-Jun, Goal-2021-Jul-Dec, MediaWiki (SRE)
RhinosF1 added a project to T7052: Make actual updates clearer on github rather than just existance: Upstream.

https://github.com/srvaroa/labeler/issues/35 was filed by you so hopefully this might get a better response

Dec 6 2021, 07:41 · Upstream, Universal Omega, SRE Automation, Goal-2021-Jan-Jun, Goal-2021-Jul-Dec, MediaWiki (SRE)
Unknown Object (User) added a comment to T7052: Make actual updates clearer on github rather than just existance.

And I know nothing about the "go" Language so I can't attempt to fork these labeler actions to fix them to work with submodules.

Dec 6 2021, 07:34 · Upstream, Universal Omega, SRE Automation, Goal-2021-Jan-Jun, Goal-2021-Jul-Dec, MediaWiki (SRE)
Unknown Object (User) added a comment to T7052: Make actual updates clearer on github rather than just existance.

https://github.com/miraheze/mediawiki/pull/4385 is the new one by the way. It at least does stuff, but doesn't work for submodules still.

Dec 6 2021, 07:33 · Upstream, Universal Omega, SRE Automation, Goal-2021-Jan-Jun, Goal-2021-Jul-Dec, MediaWiki (SRE)
RhinosF1 added a comment to T7052: Make actual updates clearer on github rather than just existance.

What do you mean for other stuff?

We can definitely pull data in git submodules as we run tests on them.

The new one works for .github/test to label "build" whereas the original didn't.

As for tests, what tests do we run on them?

Actually not in our org at the moment but I'm pretty sure bots has examples. We did have it while we were attempting to add them to the mediawiki repo though.

Dec 6 2021, 07:32 · Upstream, Universal Omega, SRE Automation, Goal-2021-Jan-Jun, Goal-2021-Jul-Dec, MediaWiki (SRE)
Unknown Object (User) added a comment to T7052: Make actual updates clearer on github rather than just existance.

What do you mean for other stuff?

We can definitely pull data in git submodules as we run tests on them.

Dec 6 2021, 07:30 · Upstream, Universal Omega, SRE Automation, Goal-2021-Jan-Jun, Goal-2021-Jul-Dec, MediaWiki (SRE)
RhinosF1 added a comment to T7052: Make actual updates clearer on github rather than just existance.

Community forum and issue on repo got nothing

Dec 6 2021, 07:28 · Upstream, Universal Omega, SRE Automation, Goal-2021-Jan-Jun, Goal-2021-Jul-Dec, MediaWiki (SRE)
RhinosF1 added a comment to T7052: Make actual updates clearer on github rather than just existance.

As suggested to Reception given this is maintained by github staff, we could email support and hope that gets us better luck.

Dec 6 2021, 07:28 · Upstream, Universal Omega, SRE Automation, Goal-2021-Jan-Jun, Goal-2021-Jul-Dec, MediaWiki (SRE)
RhinosF1 added a comment to T7052: Make actual updates clearer on github rather than just existance.

What do you mean for other stuff?

Dec 6 2021, 07:25 · Upstream, Universal Omega, SRE Automation, Goal-2021-Jan-Jun, Goal-2021-Jul-Dec, MediaWiki (SRE)
Unknown Object (User) added a comment to T7052: Make actual updates clearer on github rather than just existance.

As far as I can tell now no GitHub action will work to get submodules files changed like the labeler action method was intended for. My new one works for other stuff, whereas before nothing worked, but submodules are still not working. I honestly have no idea how to continue here seeing as how the current actions are written in go which I have absolutely no knowledge or understanding of.

Dec 6 2021, 01:03 · Upstream, Universal Omega, SRE Automation, Goal-2021-Jan-Jun, Goal-2021-Jul-Dec, MediaWiki (SRE)
Unknown Object (User) added a comment to T7052: Make actual updates clearer on github rather than just existance.
In T7052#169436, @John wrote:

@Universal_Omega Do you think this can be done before December 31st?

Dec 6 2021, 00:11 · Upstream, Universal Omega, SRE Automation, Goal-2021-Jan-Jun, Goal-2021-Jul-Dec, MediaWiki (SRE)

Dec 5 2021

John added a comment to T7052: Make actual updates clearer on github rather than just existance.

@Universal_Omega Do you think this can be done before December 31st?

Dec 5 2021, 22:35 · Upstream, Universal Omega, SRE Automation, Goal-2021-Jan-Jun, Goal-2021-Jul-Dec, MediaWiki (SRE)
RhinosF1 reassigned T7052: Make actual updates clearer on github rather than just existance from RhinosF1 to Unknown Object (User).
Dec 5 2021, 22:16 · Upstream, Universal Omega, SRE Automation, Goal-2021-Jan-Jun, Goal-2021-Jul-Dec, MediaWiki (SRE)
RhinosF1 updated subscribers of T7052: Make actual updates clearer on github rather than just existance.
In T7052#169412, @John wrote:

Can we have an update on this goal please?

The solution we currently have is a bit buggy

Can we also have a plan for realistic completion or significant progress before this goal period is update December 31st please?

@Universal_Omega said he'd look at a fix and forking the actions/labeller tool soon

Dec 5 2021, 22:15 · Upstream, Universal Omega, SRE Automation, Goal-2021-Jan-Jun, Goal-2021-Jul-Dec, MediaWiki (SRE)
John added a comment to T7052: Make actual updates clearer on github rather than just existance.

Can we have an update on this goal please?

Dec 5 2021, 21:07 · Upstream, Universal Omega, SRE Automation, Goal-2021-Jan-Jun, Goal-2021-Jul-Dec, MediaWiki (SRE)
John added a comment to T5044: Setup centralised logging for services.

I am going to start progress on this task, firstly by cleaning up how we define all of this in puppet. I'll introduce simply logging stanzas that we can define over and over again for each log file, that handles all of the syslog-ng logic + logrotate configuration for the new system.

Dec 5 2021, 21:06 · Monitoring, Goal-2022-Jan-Jun, Goal-2021-Jul-Dec, Infrastructure (SRE), Goal-2021-Jan-Jun, Goal-2020-Jul-Dec, Goal-2020-Jan-Jun

Nov 7 2021

Unknown Object (User) merged T7135: Ingest PHP-FPM slowlogs into Graylog into T5044: Setup centralised logging for services.
Nov 7 2021, 00:59 · Monitoring, Goal-2022-Jan-Jun, Goal-2021-Jul-Dec, Infrastructure (SRE), Goal-2021-Jan-Jun, Goal-2020-Jul-Dec, Goal-2020-Jan-Jun

Oct 20 2021

John added a comment to T5044: Setup centralised logging for services.

New server list for checking the above plan against:

Oct 20 2021, 12:45 · Monitoring, Goal-2022-Jan-Jun, Goal-2021-Jul-Dec, Infrastructure (SRE), Goal-2021-Jan-Jun, Goal-2020-Jul-Dec, Goal-2020-Jan-Jun
John added a comment to T5044: Setup centralised logging for services.

Plan for resolving this task:

  • All services will have their logs ingested into Graylog, this isn't negotiable.
  • Where logs are ingested, we will maintain 24-48 hours of *local* logs on the server. This will be supported by log rotation.
Oct 20 2021, 12:33 · Monitoring, Goal-2022-Jan-Jun, Goal-2021-Jul-Dec, Infrastructure (SRE), Goal-2021-Jan-Jun, Goal-2020-Jul-Dec, Goal-2020-Jan-Jun

Oct 15 2021

John closed T5877: Revise MariaDB backup strategy as Resolved.

This is now resolved.

Oct 15 2021, 17:11 · Goal-2021-Jul-Dec, Infrastructure (SRE), Goal-2021-Jan-Jun, Database, Goal-2020-Jul-Dec

Oct 14 2021

RhinosF1 renamed T7052: Make actual updates clearer on github rather than just existance from Replace Dependabot with home grown monitoring to Make actual updates clearer on github rather than just existance.
Oct 14 2021, 10:05 · Upstream, Universal Omega, SRE Automation, Goal-2021-Jan-Jun, Goal-2021-Jul-Dec, MediaWiki (SRE)

Oct 13 2021

John added a comment to T5877: Revise MariaDB backup strategy.

db13:

  • Time taken: 2 hours and 20 minutes
  • Size: 33G
Oct 13 2021, 19:35 · Goal-2021-Jul-Dec, Infrastructure (SRE), Goal-2021-Jan-Jun, Database, Goal-2020-Jul-Dec
John added a comment to T5877: Revise MariaDB backup strategy.

https://github.com/miraheze/puppet/compare/6d6dcbc15b0e...139bf730eb26 automates this for daily, so we should have a live accessible copy for a 24 hour RPO - and bacula will store backups for a longer period of time (TBD).

Oct 13 2021, 13:27 · Goal-2021-Jul-Dec, Infrastructure (SRE), Goal-2021-Jan-Jun, Database, Goal-2020-Jul-Dec
John added a comment to T5877: Revise MariaDB backup strategy.

The backup ran for 14 hours before I killed it as it caused T8163.

Oct 13 2021, 10:04 · Goal-2021-Jul-Dec, Infrastructure (SRE), Goal-2021-Jan-Jun, Database, Goal-2020-Jul-Dec

Oct 12 2021

John added a comment to T5877: Revise MariaDB backup strategy.

Currently doing the above command but over an NFS mount to dbbackup1 which is in the US. This will take significantly longer - that is the main thing I am interested in right now.

Oct 12 2021, 20:14 · Goal-2021-Jul-Dec, Infrastructure (SRE), Goal-2021-Jan-Jun, Database, Goal-2020-Jul-Dec
John added a comment to T5877: Revise MariaDB backup strategy.
mydumper -G -E -R -m -v 3 -t 2 -c -x "^(?!([0-9a-z]+wiki.(objectcache|querycache|querycachetwo|recentchanges|searchindex)))" -L "/home/johnflewis/$(date +"%Y%m%d%H%M%S").log" --trx-consistency-only

On db12:

  • Time taken: 103 minutes (1 hour and 43 minutes)
  • Size: 30G
Oct 12 2021, 18:53 · Goal-2021-Jul-Dec, Infrastructure (SRE), Goal-2021-Jan-Jun, Database, Goal-2020-Jul-Dec

Oct 11 2021

John added a comment to T5877: Revise MariaDB backup strategy.

Trying to optimise the dump by reducing amount of data carried over (because not everything in MediaWiki is irreplaceable!)

Oct 11 2021, 22:12 · Goal-2021-Jul-Dec, Infrastructure (SRE), Goal-2021-Jan-Jun, Database, Goal-2020-Jul-Dec
John claimed T5044: Setup centralised logging for services.
Oct 11 2021, 18:08 · Monitoring, Goal-2022-Jan-Jun, Goal-2021-Jul-Dec, Infrastructure (SRE), Goal-2021-Jan-Jun, Goal-2020-Jul-Dec, Goal-2020-Jan-Jun
John added a comment to T5044: Setup centralised logging for services.

T7740 is likely to be influenced by work done on this task.

Oct 11 2021, 18:06 · Monitoring, Goal-2022-Jan-Jun, Goal-2021-Jul-Dec, Infrastructure (SRE), Goal-2021-Jan-Jun, Goal-2020-Jul-Dec, Goal-2020-Jan-Jun

Oct 9 2021

John claimed T5877: Revise MariaDB backup strategy.
Oct 9 2021, 20:26 · Goal-2021-Jul-Dec, Infrastructure (SRE), Goal-2021-Jan-Jun, Database, Goal-2020-Jul-Dec

Sep 28 2021

John placed T5877: Revise MariaDB backup strategy up for grabs.

De-assigned per lack of progress.

Sep 28 2021, 11:25 · Goal-2021-Jul-Dec, Infrastructure (SRE), Goal-2021-Jan-Jun, Database, Goal-2020-Jul-Dec

Sep 21 2021

John added a comment to T5877: Revise MariaDB backup strategy.

@Southparkfan Any updates on this task? If there isn't an update provided in a week, I'll reassign the task to ensure it gets completed.

Sep 21 2021, 20:14 · Goal-2021-Jul-Dec, Infrastructure (SRE), Goal-2021-Jan-Jun, Database, Goal-2020-Jul-Dec

Aug 10 2021

Paladox added a comment to T5044: Setup centralised logging for services.
In T5044#156437, @John wrote:

@Paladox has raised concerns with centralised-only logging. We should explore these concerns before pushing for things like nginx access logs as these are critical for debugging some traffic influx/DoS attacks.

Aug 10 2021, 16:57 · Monitoring, Goal-2022-Jan-Jun, Goal-2021-Jul-Dec, Infrastructure (SRE), Goal-2021-Jan-Jun, Goal-2020-Jul-Dec, Goal-2020-Jan-Jun
Reception123 added a comment to T5044: Setup centralised logging for services.

I agree with that. At least for some logs it's definitely useful to have logs stored locally in case something goes wrong and the logs don't get transmitted to graylog.

Aug 10 2021, 14:14 · Monitoring, Goal-2022-Jan-Jun, Goal-2021-Jul-Dec, Infrastructure (SRE), Goal-2021-Jan-Jun, Goal-2020-Jul-Dec, Goal-2020-Jan-Jun