Introducing SCSVG
The next steps in Miraheze's growth

In J16, we introduced the plan that Miraheze's Site Reliability Engineering team was pursuing to redesign and rewrite our infrastructure from the ground up by exploring other alternatives to how we are hosted. We are pleased to announce that since November, we have had our initial budget approved for hardware purchases and on Wednesday, 29th of December, 2021, deployed for the first time in Miraheze's history our own physical hardware!

You can see some pictures of them on Meta

To mark the occasion, we have also taken the approach of naming the 'datacentre' internally for us - rather than referring to it by providers name (as we have traditionally done for previous hosting locations such as DigitalOcean, Linode, RamNode, OVH). As such, I can now announce we have chosen to the call the new datacentre footprint, SCSVG. While it is not a catchy name at all, it can split up into:

  • SC - ServerChoice, the owners of the datacentre,
  • SVG - Stevenage, the 3-letter unique identifier for the nearest train station to the datacentre.

Is This Better?

For everyone, yes - on paper!

With the new infrastructure, we should see a drastic decrease in resource competition - particularly for CPU and processing time as we are more than doubling our available resources in this regard. Currently, our two general purpose cloud servers have two quad cores with hyperthreading enabled on them - this gives us a maximum of 32 threads (4 (cores) * 2 (hyperthreading) * 2 (CPUs) * 2 (servers)). We have deployed 3 general purposes servers now, each with 2 hex core CPUs with hyperthreading enabled - this gives us a new maximum of 72 threads! (6 (cores) * 2 (hyperthreading) * 2 (CPUs) * 3 (servers).) With this increased computing power available, we able to deploy more services with more power to handle the workload generated by the communities we service.

In addition, we also now deploy a hybrid setup of HDD and SSD disk devices. Each server has 1.8TB of HDD space and 960GB of SSD space available for us to use. Our plan is to leave non-critical services and disk space intensive services on our HDD disks, while moving our more critical and read intensive services (such as MediaWiki) onto our new SSD disk devices.

Finally, we are able to deploy a fully IPv6-only infrastructure at last - meaning we can split up our services even further to fine-tune the requirements and improve reliability/redundancy to all of our components that in the past we have been unable to because of the IPv4 address space limitations.

Our plan to improve the service we provide to each community will be supported by the above 3 main improvements - we can improve our MediaWiki response times vastly by:

  • reducing processing competition for CPU resources,
  • reduce disk latency for accessing files,
  • deploying more containerised hosts, keeping intensive operations off front end facing hosts.

What Is Next?

Currently we are working hard behind the scenes to get things set up, and to start planning the migration over between infrastructures. We will keep everyone up to date as more is known and things begin to put into place in terms of migration plans. You can follow the progress on T8469 and all the sub tasks - which will be proactively kept up to date!

There will be a period of degraded service across Miraheze during January - these will include read only times, potential total service disruption (we try to avoid it!) and a pause on image-related actions and wiki creations/changes. We will give at least one weeks notice before any of these things happen to ensure proper plans and notifications can be put into place to reach all of our communities.

Written by John on Jan 2 2022, 22:15.
Engineering Manager, Infrastructure

Event Timeline

This comment was removed by John.
This comment was removed by Void.