Just to clarify, what role will you assume, MediaWiki Engineer? Software Engineer?

I guess technically it would be Site Reliability Engineer on the MediaWiki team https://meta.miraheze.org/wiki/Miraheze_Vacancies#Site_Reliability_Engineers_(Infrastructure/MediaWiki)

Sorry, MediaWiki engineer seems like the more correct role.

Reception123 is out of office for a few days and I'm obviously not the Engineering Manager in any way but just to kickstart this process, I'm going to ask some routine questions.

1. What work do you plan on doing as a MediaWiki Engineer? Will you continue to be active as you have currently demonstrated?
2. Do you have previous experience in MediaWiki? What knowledge do you have of Linux CLI (command line interface) and MySQL?
3. If you were granted the sysadmin role on-wiki, in what situations would it be appropriate to use it?
4. A user emails sre@ about losing their 2FA device and recovery codes and you are the first to see the email and respond. How do you handle the situation?
5. A user complains that the wrong number of active users appears on Special:Statistics. How would you try to fix that issue? (the precise command you'd use)
6. How would you proceed with the import if a user requests an import on Phabricator and emails sre@ a file (say, eamplewiki.xml) for their wiki, examplewiki? (include all steps including how you would get the file on the server)
7. What would you do if you think you have the solution to an issue, but aren't really sure about it?
8. What commands would you use if you wanted to find out 1) the number of private wikis 2) the number of wikis that are in English 3) get a list of all closed wikis
9. What logs would you check if 1) you saw reports of 503s on all wikis? 2) someone reported an error like "Fatal exception of type "MWException""
10. How would you determine who is logged in on a server?
11. How would you determine what processes are running on a server?
12. What server(s) would you use to run maintenance scripts?
13. Miraheze uses Git to manage and update our MediaWiki installation and all extensions an skins. With this in mind, how would you update an extension via Git? (For example: update CreateWiki) - use the exact commands you would use.
14. Miraheze uses Git to manage and update its extensions. With this in mind, how would you install an extension? (provide commands you'd use and configuration files you'd change)
15. You see something like <system-message-1> appear on-wiki instead of an actual message, what script would you run to fix that?
16. How would you import images if a user requests an import on Phabricator of file examplewiki.xml for examplewiki? (include all steps including how you would get the file on the server)

In T10619#214018, @Agent_Isai wrote:

Reception123 is out of office for a few days and I'm obviously not the Engineering Manager in any way but just to kickstart this process, I'm going to ask some routine questions.

1. What work do you plan on doing as a MediaWiki Engineer? Will you continue to be active as you have currently demonstrated?

As I said above, the day-to-day type work. Stuff like "pls give me a dump of the wiki's images", "I can't edit the wiki, it shows me an error", and also PR reviews on the MediaWiki repositories, which UO was really the only person in the team doing. I'd also do the more feature request type tasks, depending on my ability and if I have time on my hands (real life is being kind of a PITA right now).

2. Do you have previous experience in MediaWiki? What knowledge do you have of Linux CLI (command line interface) and MySQL?

I've worked on some stuff here, most of it on the easy side to be honest, though I've tried top take over more complex tasks. My knowledge of the MediaWiki way of doing things technically might not be the best, but I'm confident in my ability to improve in this aspect.

As for Linux, I daily-drive Linux (currently running Alpine Linux on my desktop), and have administered Debian servers before. I would say I know what I'm doing in the CLI.

As for SQL, I know the basics, given good documentation of the database (basically know where stuff is, like how extensions and MediaWiki core document what database table they use at mediawiki.org) I'd be able to do as required. However I admit this is one of my weaker areas overall.

3. If you were granted the sysadmin role on-wiki, in what situations would it be appropriate to use it?

To assist in debugging for Phabricator tasks or handling the very unusual OAuth consumer review.

4. A user emails sre@ about losing their 2FA device and recovery codes and you are the first to see the email and respond. How do you handle the situation?

If that user doesn't have convincing proof of being who they claim to be, I wouldn't disable 2FA. As for convincing proof, for most users this will mean either the very rare user of OpenPGP digital signatures or a committed identity, per the Reset lost credentials page.

5. A user complains that the wrong number of active users appears on Special:Statistics. How would you try to fix that issue? (the precise command you'd use)

I'd run the initSiteStats script. sudo -u www-data php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=<dbname here> --active --update

6. How would you proceed with the import if a user requests an import on Phabricator and emails sre@ a file (say, eamplewiki.xml) for their wiki, examplewiki? (include all steps including how you would get the file on the server)

I'd SFTP the file over to my home directory on the server, then go from there like usual.

scp examplewiki.xml alex@dummydomain.miraheze.org:/home/alex/examplewiki.xml

(SSH into the server)

sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=examplewiki --no-updates --username-prefix=eaamplewiki /home/alex/eamplewiki.xml

sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=examplewiki

rm eamplewiki.xml

7. What would you do if you think you have the solution to an issue, but aren't really sure about it?

Try getting second opinions, also test it at betaheze if applicable.

8. What commands would you use if you wanted to find out 1) the number of private wikis 2) the number of wikis that are in English 3) get a list of all closed wikis

At the MariaDB CLI of mhglobal:

1: SELECT COUNT(*) FROM cw_wikis WHERE wiki_private != 0;
2: SELECT COUNT(*) FROM cw_wikis WHERE wiki_language = en;
3: SELECT COUNT(*) FROM cw_wikis WHERE wiki_closed != 0;

9. What logs would you check if 1) you saw reports of 503s on all wikis? 2) someone reported an error like "Fatal exception of type "MWException""

1: I'd check the #miraheze-sre channel and see if the icinga bot says the server is overloaded, aditionally the access log in case there's some IP just completely flooding the server with requests.
2: The NGINX error log.

10. How would you determine who is logged in on a server?

run the who command.

11. How would you determine what processes are running on a server?

Good-old top.

12. What server(s) would you use to run maintenance scripts?

Judging from https://meta.miraheze.org/wiki/Tech:Server_admin_log, it appears that these things are done at mwtask141 mostly.

13. Miraheze uses Git to manage and update our MediaWiki installation and all extensions an skins. With this in mind, how would you update an extension via Git? (For example: update CreateWiki) - use the exact commands you would use.

Given that extensions are managed as submodules, for the most part this will mean merging a PR from dependabot @ https://github.com/miraheze/mediawiki, then deploying like usual (sorry, I don't know how to do this part in Miraheze, first time messing with Puppet)

14. Miraheze uses Git to manage and update its extensions. With this in mind, how would you install an extension? (provide commands you'd use and configuration files you'd change)

First step is to cd into my fork of the mediawiki.

I'd add the extension as a submodule at the extensions folder. git checkout -b add-best-extension-ever, cd into the extensions folder, then git submodule add https://github.com/l33th4xx0r/best-extension-ever.

With the extension now at the extensions folder, I'd commit and push the changes, then open a PR at https://github.com/miraheze/mediawiki

Now I'd switch over to my fork of mw-config.

git checkout -b add-best-extension-ever, then add the extension to https://github.com/miraheze/mw-config/blob/master/ManageWikiExtensions.php. If it exists, add the new configuration to https://github.com/miraheze/mw-config/blob/master/ManageWikiSettings.php. Then open a PR.

Assuming everything went okay, I'd merge the 2 PRs, then deploy.

15. You see something like <system-message-1> appear on-wiki instead of an actual message, what script would you run to fix that?

I've actually never seen something like this happen, sorry but I don't know how to fix that.

16. How would you import images if a user requests an import on Phabricator of file examplewiki.xml for examplewiki? (include all steps including how you would get the file on the server)

But the XML file only contains revisions, there's no images in this situation. But assuming they gave a compressed zip file with images, I'd copy them over to my home directory at the server, uncompress, run ImportImages.php, then remove the images from my home dir.

Just some quick feedback from me on your responses.

Stuff like "pls give me a dump of the wiki's images",

For this, it is very complicated and requires numerous steps, and using wrong swift commands can be catastrophic to the entirety of Miraheze. How would you go about doing this? What Swift commands do you think you would use, or steps to take? P.S. It's probably fine to get wrong, someone can walk you through it, and I also had to explain things to Reception123 even for that when we migrated to Swift, since I was more involved in the migration as well.

At the MariaDB CLI of mhglobal:

MediaWiki engineers do not have access to this, it would be in sql.php with --wiki=loginwiki --wikidb=mhglobal

Given that extensions are managed as submodules, for the most part this will mean merging a PR from dependabot @ https://github.com/miraheze/mediawiki, then deploying like usual (sorry, I don't know how to do this part in Miraheze, first time messing with Puppet)

How would you do this with git commands? As dependabot isn't always reliable.

For this and adding submodules, there is documentation pages in the tech namespace on Meta that would helpful looking at deploy steps as well.

Now I'd switch over to my fork of mw-config.

git checkout -b add-best-extension-ever, then add the extension to https://github.com/miraheze/mw-config/blob/master/ManageWikiExtensions.php. If it exists, add the new configuration to https://github.com/miraheze/mw-config/blob/master/ManageWikiSettings.php. Then open a PR.

Don't forgot LocalSettings.php. Everything added to ManageWikiSettings.php needs added to LocalSettings.php also, the same is not the case for ManageWikiNamespaces.php, for that one only needs added directly there. In fact adding to LocalSettings.php also for that one sometimes breaks things.

Also, for #6, this is correct, but be mindful of the size of the import, if it is a small import request, advise them to use Special:RequestImportDump on Meta instead to make the process easier

Also for maintenance scripts, instead of using full command, you can use mwscript commands, which will automatically log to SAL also, otherwise don't forget to !log in #miraheze-sre on IRC.

In T10619#214317, @Universal_Omega wrote:

Just some quick feedback from me on your responses.

Stuff like "pls give me a dump of the wiki's images",

For this, it is very complicated and requires numerous steps, and using wrong swift commands can be catastrophic to the entirety of Miraheze. How would you go about doing this? What Swift commands do you think you would use, or steps to take? P.S. It's probably fine to get wrong, someone can walk you through it, and I also had to explain things to Reception123 even for that when we migrated to Swift, since I was more involved in the migration as well.

I don't actually know Swift, I kinda just assumed that it was easy (from a downloading files point of view).

After reading https://meta.miraheze.org/wiki/Tech:Swift, I should run the swift-env script to set the required envars for Swift, then search for the container of the wiki in question, which, judging by the example at that page (miraheze-examplewiki-local-public) should be easy. Then, to download the files, I should run swift download miraheze-examplewiki-local-public -D examplewiki-images, with the parameters changed as required of course.

Is this how to make a dump of the images? Please let me know if I got it right.

Given that extensions are managed as submodules, for the most part this will mean merging a PR from dependabot @ https://github.com/miraheze/mediawiki, then deploying like usual (sorry, I don't know how to do this part in Miraheze, first time messing with Puppet)

How would you do this with git commands? As dependabot isn't always reliable.

At the root of the mediawiki repository, run git submodule init extensions/DummyExtension, then git submodule update extensions/DummyExtension. Then just commit and push like usual.

For this and adding submodules, there is documentation pages in the tech namespace on Meta that would helpful looking at deploy steps as well.

Thanks, I found https://meta.miraheze.org/wiki/Tech:Deploy-mediawiki for how to deploy, and https://meta.miraheze.org/wiki/Tech:Updating_an_extension for updating submodules.

ALWAYS be VERY careful with swift commands, and never accidentally use the wrong one, or it could be catastrophic, usually it would be hard to use the wrong, but I'd advise against using swift delete commands when unnecessary.

Don't worry, I'll be sure to check thrice before running swift commands.

Thank you very much for your application @OrangeStar and for your detailed answers to Agent's questions. I think on the technical level you've been asked enough questions that demonstrate your knowledge. Two more non-technical questions:

1. When you say "step up and at least temporarily" does that mean that you don't see yourself staying as MWE very long / being active?
2. While not a pre-requisite in any way, would you be willing to join IRC and/or Discord channels occasionally in order to be able to communicate more effectively with team members, especially during maintenance (such as mw upgrades)?

In T10619#214432, @Reception123 wrote:

Thank you very much for your application @OrangeStar and for your detailed answers to Agent's questions. I think on the technical level you've been asked enough questions that demonstrate your knowledge. Two more non-technical questions:

1. When you say "step up and at least temporarily" does that mean that you don't see yourself staying as MWE very long / being active?

It's not activity, I mostly meant that in a "you never know what the future has for you" kind of way. I have some personal things going on and while I don't see why they would stop me from staying in SRE long term, you never know.

2. While not a pre-requisite in any way, would you be willing to join IRC and/or Discord channels occasionally in order to be able to communicate more effectively with team members, especially during maintenance (such as mw upgrades)?

Yes, in IRC.

Thank you for your quick answer and willingness to join IRC at some times, I will discuss this with @Void per the policy. In the meantime I would suggest reading https://meta.miraheze.org/wiki/Tech:Home if you have a chance.

This request gets a +1 from me.

Some notes based on the comments in this task:

Take a look at https://meta.miraheze.org/wiki/Tech:MediaWiki_appserver#mwscript when you get the chance. (Source can be found here if you're interested). It is useful mostly for automatic logging of all commands. Generally speaking, any command you run that impacts files or services that face the user should be logged. If you need to manually log a message, either run logsalmsg "message to be logged" on the server or !log message to be logged in #miraheze-sre on IRC.

If you are not familiar with the screen utility, I would recommend doing so now, as it is very helpful for long running scripts. You can disconnect from your SSH session while still having your command running inside the screen. When you connect back to the server, you can pull up the screen again to see the status and output of the screen session.

You see something like <system-message-1> appear on-wiki instead of an actual message, what script would you run to fix that?

I've actually never seen something like this happen, sorry but I don't know how to fix that.

This means that the localization cache is out of date or otherwise broken. I believe you can run deploy-mediawiki --l10n on mwtask141 to resolve this on all servers. (deploy-mediawiki source). This is preferable to rebuilding the localization cache on all servers.

This now just needs approval from @Reception123 as engineering manager.

Since Void has given his +1 I also approve of this request and this is now approved!

The onboarding can start once there is a confirmed NDA signed with Miraheze Limited.

NDA is on it's way.

@Owen will need to confirm the NDA has been received, as soon as that's done it can start.

@OrangeStar While we wait for the NDA process, here is a more comprehensive list of SRE policies. While quite short https://meta.miraheze.org/wiki/Tech:Organisation/mw-admins#Rules is quite relevant for the beginning as it's important to ask before doing before you are able to familiarize yourself with the processes.