I can now confirm that since notifications are fixed (thanks @Universal_Omega !) RequestSSL is operational.
What remains to be done is to add a check on-wiki for whether CNAME or NS is pointed (@Universal_Omega has an idea for how to do that easily) and then for the puppet API

In T7582#237196, @OrangeStar wrote:

@Reception123 If you want to get RequestSSL working right now, we could look at sending emails the way core does with Special:EmailUser

@Reception123 If you want to get RequestSSL working right now, we could look at sending emails the way core does with Special:EmailUser

In T7582#234826, @Reception123 wrote:

In T7582#234817, @OrangeStar wrote:

Regarding step 4:

This is too Miraheze-specific for inclusion in the RequestSSL codebase in my opinion. It is better suited as part of T11710. In the Miraheze-specific setup of this extension, once RequestSSL sends the request to puppet, the server program receiving the request should take care of determining if we should add new DNS zones. So I think steps 4 and 5 should be merged together.

Just to be clear, what you propose is the following? In my example, the domain is pointed via NS.
1: User requests SSL
2: RequestSSL checks (with puppet181's help) whether domain is pointed or not
3: RequestSSL submitted
4: ssl-certificate script once again checks whether domain is pointed and if it's pointed via NS, adds zone to GitHub
5: RequestSSL marked as completed

EDIT: in the fully automated version, steps 2 and 4 would probably be repetitive and would need merging

In T7582#234817, @OrangeStar wrote:

Regarding step 4:

This is too Miraheze-specific for inclusion in the RequestSSL codebase in my opinion. It is better suited as part of T11710. In the Miraheze-specific setup of this extension, once RequestSSL sends the request to puppet, the server program receiving the request should take care of determining if we should add new DNS zones. So I think steps 4 and 5 should be merged together.

Regarding step 4:

In T7582#234814, @MacFan4000 wrote:

I’m not sure that ssl-admins should be decommissioned, as cert removals would still be done manually, and this would allow us to investigate should something go wrong, and also somebody might not want letsencrypt.

I’m not sure that ssl-admins should be decommissioned, as cert removals would still be done manually, and this would allow us to investigate should something go wrong, and also somebody might not want letsencrypt.

@Reception123 I think it's about time we get a RequestSSL project and workboard on Phab. Also, add me as a member of the project if it will not have an open join policy please.

Thinking about it, it would still definitely be useful to check whether the domain is pointing before the request is submitting but the python script running on puppet141 would still be needed in the end in order to be able to create the DNS zone for wikis pointing NS. The script already exists but needs some adjustments.

@OrangeStar Thanks for the ideas. Indeed, it might be better to check whether the domain is pointed via PHP rather than having that in the python script and then having to contact the user afterwards and tell them it isn't. I guess what could be done then if it is not pointed is have an error display that clearly directs users to somewhere where they can get help pointing their domain.

Functional but not yet over! Can't do much right now since I'm waiting for T11680 which will introduce some utility functions (To avoid reinventing the wheel) that the server program that automates cert generation would need, but I want to cleanup some of my PRs (the ones yesterday were just to make stuff work), move strings into i18n, change existing i18n strings too, remove some ID leftovers I saw while skimming the code, and automate checking that the domain is pointed correctly at least.

Noting that the repo has been transferred to https://github.com/miraheze/requestssl

Thanks to @OrangeStar, RequestSSL is now functional! There's unfortunately one late issue that I thought of that will mean it can still not be made operational. For custom domains it is quite often the case that users don't point their domains properly and need guidance. My understanding is that RequestSSL uses Echo to notify users when there's a comment on their request and if they don't manually enable email notifications they might not get any and would not know that there's been a comment.

https://github.com/Reception123/RequestSSL/pull/5

In T7582#234462, @OrangeStar wrote:

In T7582#213501, @Reception123 wrote:

• Implement logging so that when RemoteWiki is executed with ManageWiki it logs it as if someone had changed managewiki on wiki

Does MediaWiki even have a concept of other wikis existing other than the one currently "running"? We could open the database for the remote wiki and manually write to the logs, but I don't think that's very good.

Assuming you want to log to the remote wiki's managewiki log.

In T7582#213501, @Reception123 wrote:

• Implement logging so that when RemoteWiki is executed with ManageWiki it logs it as if someone had changed managewiki on wiki

In T7582#234396, @OrangeStar wrote:

https://github.com/Reception123/RequestSSL/pull/4

https://github.com/Reception123/RequestSSL/pull/4

UO for the rescue! I was about to commit a grave sin against best practices.

In T7582#234392, @OrangeStar wrote:

In T7582#213501, @Reception123 wrote:

• Add a method so that $oldstatus and $newstatus is known in order for the updateManageWiki function to be executed only when the status is changed from something else to completed

So, I have an idea for this, but... it involves reading the $wgRequest global. Would this be acceptable? I'll keep searching for better ways to do this anyway.

In T7582#213501, @Reception123 wrote:

• Add a method so that $oldstatus and $newstatus is known in order for the updateManageWiki function to be executed only when the status is changed from something else to completed

In T7582#234273, @OrangeStar wrote:

Heads up: some custom domains are pointed using CNAME flattening. So any automation attempts should not only check for CNAME records or whether the authoritative nameservers are pointed to us, but if the A or AAAA returned points to the known IPs of cp* servers.

Heads up: some custom domains are pointed using CNAME flattening. So any automation attempts should not only check for CNAME records or whether the authoritative nameservers are pointed to us, but if the A or AAAA returned points to the known IPs of cp* servers.

Timestamps are no longer an issue. For any SRE that is a more competent developer than me (most will be!), the two remaining things for Step 3 are:

Due to the large number of tasks in this area and the particular use that automation can provide, moving this task to normal priority. It'd be nice if at least the remaining fixes for RequestSSL (Step 3) can be completed soon.

https://github.com/Reception123/RequestSSL has been created based on ImportDump. Things that are required to make it operational

• Add a method so that $oldstatus and $newstatus is known in order for the updateManageWiki function to be executed only when the status is changed from something else to completed
• Implement logging (can be copied from ManageWiki) so that when RemoteWiki is executed it logs it as if someone had changed managewiki on wiki
• Fix issues with timestamp (might just be a problem with the SQL implemented on beta)

Due to limited knowledge on my part, it would be preferable if someone else had a go at this.

• Check if all i18n messages make sense (can be done by anyone) [DONE!]

Just so we don't forget, the current idea would be to try using https://github.com/wikimedia/acme-chief and have an API backend for ManageWiki with the web app being MediaWiki.

While this hasn't been fully tested yet here is my not-so-perfect version of step 3 which I unfortunately didn't integrate into the ssl-certificate script due to lack of knowledge on how to do so properly and not wanting to make things too messy. Here is the current script that can be used in the meantime (not 100% sure if DNS works yet): https://phabricator.miraheze.org/P474 and hopefully be integrated into the main one soon so that step 4 can follow.

Opened https://github.com/miraheze/puppet/pull/2996.

Resolved

@Paladox less than a week until end of goal period - do we have an update on this?

SSL certificates are now generated on puppet111 directly and private keys are automatically copied.

Since there is no progress, I propose we simply move ssl-certificate to puppet111 (since no mw-admins have done any SSL tasks in recent years anyway) and have the private keys automatically added.

In T7582#172916, @Universal_Omega wrote:

I purpose that this task does not become a goal for the new goal period, as there was no progress on it last time, and does not seem that there is anyone who intends to work on it. Goals should ideally have someone fully committed to getting them done.

I purpose that this task does not become a goal for the new goal period, as there was no progress on it last time, and does not seem that there is anyone who intends to work on it. Goals should ideally have someone fully committed to getting them done.

This task has taken a back foot, over other work which has higher priority currently such as T8469 T8350

@Universal_Omega said he'd try to add some suggestions for this.

https://github.com/miraheze/mediawiki/blob/REL1_37/.github/workflows/label.yml
confirmed to work on https://github.com/miraheze/mediawiki/pull/4440

Everything seems to be going smoothly with this.

We can also add labels with:

pull_number=$(jq --raw-output .pull_request.number "$GITHUB_EVENT_PATH")
curl \
  -X POST \
  -H "Accept: application/vnd.github.v3+json" \
  -H 'Authorization: token ${{ secrets.GITHUB_TOKEN }}' \
  https://api.github.com/repos/${GITHUB_REPOSITORY}/issues/$pull_number/labels \
  -d '{"labels":["build"]}'

And we can parse the list of files with:

git -c color.diff=false diff --submodule=diff $submodule | sed -n -e s/^diff\ --git\ a\\///p

I have this to parse the dependabot branches into which submodule to check against, and properly get the full diff of them. I still don't know how to use this, convert it into a list of only files changed (as --name-only won't work with git diff for submodules). If we can convert it to a simple list somehow, then we still need to figure out how to use that list, add some regex checks on it, and add a label to it, to make it clearer what changes are made.

In T7052#169451, @RhinosF1 wrote:

https://github.com/srvaroa/labeler/issues/35 was filed by you so hopefully this might get a better response

https://github.com/srvaroa/labeler/issues/35 was filed by you so hopefully this might get a better response

And I know nothing about the "go" Language so I can't attempt to fork these labeler actions to fix them to work with submodules.

https://github.com/miraheze/mediawiki/pull/4385 is the new one by the way. It at least does stuff, but doesn't work for submodules still.

In T7052#169447, @Universal_Omega wrote:

In T7052#169444, @RhinosF1 wrote:

What do you mean for other stuff?

We can definitely pull data in git submodules as we run tests on them.

The new one works for .github/test to label "build" whereas the original didn't.

As for tests, what tests do we run on them?

Actually not in our org at the moment but I'm pretty sure bots has examples. We did have it while we were attempting to add them to the mediawiki repo though.

In T7052#169444, @RhinosF1 wrote:

What do you mean for other stuff?

We can definitely pull data in git submodules as we run tests on them.

Community forum and issue on repo got nothing

As suggested to Reception given this is maintained by github staff, we could email support and hope that gets us better luck.

What do you mean for other stuff?

As far as I can tell now no GitHub action will work to get submodules files changed like the labeler action method was intended for. My new one works for other stuff, whereas before nothing worked, but submodules are still not working. I honestly have no idea how to continue here seeing as how the current actions are written in go which I have absolutely no knowledge or understanding of.

In T7052#169436, @John wrote:

@Universal_Omega Do you think this can be done before December 31st?

@Universal_Omega Do you think this can be done before December 31st?

This can be closed once we successfully use it on Tuesday.

So far none. This is looking unlikely it'll be done this period.

In T7052#169412, @John wrote:

Can we have an update on this goal please?

The solution we currently have is a bit buggy

Can we also have a plan for realistic completion or significant progress before this goal period is update December 31st please?

@Universal_Omega said he'd look at a fix and forking the actions/labeller tool soon

Can we have an update on this goal please? Last update was on July 3rd.

Can we have an update on this goal please?

I am going to start progress on this task, firstly by cleaning up how we define all of this in puppet. I'll introduce simply logging stanzas that we can define over and over again for each log file, that handles all of the syslog-ng logic + logrotate configuration for the new system.

Just to acknowledge that I've seen the above and we will soon begin working on our SLOs.

Infrastructure SLOs have been completed on the internal site. This is now just waiting for MediaWiki (SRE)'s input for MediaWiki-led SLOs.

This is now a goal.

New server list for checking the above plan against:

Plan for resolving this task:

• All services will have their logs ingested into Graylog, this isn't negotiable.
• Where logs are ingested, we will maintain 24-48 hours of *local* logs on the server. This will be supported by log rotation.