Page MenuHomeMiraheze
Create Task
Maniphest T5675

AddTrust External Root Certificate Expired
Closed, ResolvedPublic
Actions

Description

Per https://support.sectigo.com/Com_KnowledgeDetailPage?Id=kA03l00000117LT, our AddTrust External CA root certificate expired.

This has caused a site-wide outage starting 10:51 UTC

Related Objects

Event Timeline

Kees_Langeveld created this task.May 30 2020, 10:53
Herald added a subscriber: Reception123. · View Herald TranscriptMay 30 2020, 10:53
TerrryTerrr added a subscriber: TerrryTerrr.May 30 2020, 10:59
RhinosF1 assigned this task to Southparkfan.May 30 2020, 11:28
RhinosF1 added a subscriber: RhinosF1.

Known

RhinosF1 triaged this task as Unbreak Now! priority.May 30 2020, 11:28
Herald added a project: Amanda Catherine. · View Herald TranscriptMay 30 2020, 11:28
Herald added subscribers: AmandaCath, • Examknow. · View Herald Transcript
RhinosF1 added a project: Site Reliability Engineering.May 30 2020, 11:28
RhinosF1 mentioned this in T1979: Error: 503 Backend fetch failed.
CircleyDoesExtracter added a subscriber: CircleyDoesExtracter.May 30 2020, 11:50

How much time before the error will go away?

RhinosF1 added a comment.May 30 2020, 11:51
In T5675#110767, @CircleyDoesExtracter wrote:

How much time before the error will go away?

We’re working on it now. We can’t give an exact time yet.

CircleyDoesExtracter added a comment.May 30 2020, 11:55

2 Hours?

RhinosF1 added a comment.May 30 2020, 11:56
In T5675#110769, @CircleyDoesExtracter wrote:

2 Hours?

We can’t say. We are testing a solution.

RhinosF1 added a comment.May 30 2020, 11:56

Please do not ask us when it will be fixed. If we have any news, we will let users know.

CircleyDoesExtracter added a comment.May 30 2020, 11:57

Ok

RhinosF1 added a comment.May 30 2020, 11:58

We are now attempting to deploy a workaround for the issue. We will let you know when this is complete.

CircleyDoesExtracter added a comment.May 30 2020, 12:00

Will it fail? I think not.

RhinosF1 added a comment.May 30 2020, 12:02
In T5675#110774, @CircleyDoesExtracter wrote:

Will it fail? I think not.

Hopefully not.

Note to everyone: Please keep general discussion to our discord or IRC. We will update you as we can but constant comments are not helpful.

CircleyDoesExtracter added a comment.May 30 2020, 12:03

Ok I'll stop sending messages.

CircleyDoesExtracter added a comment.May 30 2020, 12:35

Here's the reason why "We have identified the cause of the errors, which is the expiration of the "AddTrust External CA Root" certificate."

RhinosF1 renamed this task from Error 503 to AddTrust External Root Certificate Expired.May 30 2020, 12:39
RhinosF1 updated the task description. (Show Details)

I’ve updated the title to reflect the situation. We are working get the issue resolved.

CircleyDoesExtracter added a comment.May 30 2020, 12:40

Okay. How can you fix the expired add trust external root?

RhinosF1 added a comment.EditedMay 30 2020, 12:41
In T5675#110781, @CircleyDoesExtracter wrote:

Okay. How can you fix the expired add trust external root?

We are working on it and will publish an incident report with information later.

If you have discord, go to discord.is/miraheze and I’ll talk to you in real-time. We also have #miraheze on Freenode.

RhinosF1 closed this task as Resolved.May 30 2020, 13:25

Well done to SRE, the issue has been resolved.

CircleyDoesExtracter added a comment.May 30 2020, 13:26

YES! YES! THX!!!!!!!!!!

RhinosF1 added a subscriber: John.May 30 2020, 13:33

Incident Start: 11:51:04
Incident Over: 14:23:00
Total Time: 2 hours and 31 mins

(in UTC+1)

Responding SRE: @Reception123, @John and @Southparkfan (incident lead)

AmandaCath awarded a token.May 30 2020, 14:22

Dislike token is for the incident occurring to begin with... although I will say "good job"/"like" to those who resolved it.

RhinosF1 added a comment.May 30 2020, 14:25
In T5675#110788, @AmandaCath wrote:

Dislike token is for the incident occurring to begin with...

All incidents aren’t good but we were not aware of the issue until icinga started alerting so nothing we could have done.

although I will say "good job"/"like" to those who resolved it.

I’m sure everyone will appreciate it.