Page MenuHomeMiraheze

High cpu/mem on all cache proxies (10 Oct 2021 11:05)
Closed, ResolvedPublic


cpu/mem jumped on cp*, all depooled. Everything down.

Event Timeline

RhinosF1 raised the priority of this task from High to Unbreak Now!.Oct 10 2021, 10:09

Looks like cpu locked up, memory also jumped. Caused cache proxies to be unavailable

RhinosF1 renamed this task from High load on all cache proxies to High cpu/mem on all cache proxies (10 Oct 2021 11:05).Oct 10 2021, 11:18
RhinosF1 updated the task description. (Show Details)

Correcting title as cpu basic graph jumped not load

@John has confirmed it was a (D)DoS

It was distributed so not something we can block

root@cp15:~# cat /var/log/nginx/access.log | awk '{ print $7 }' | sort | uniq -c | sort -nr | head -n 1
root@cp12:~# cat /var/log/nginx/access.log | awk '{ print $7 }' | sort | uniq -c | sort -nr | head -n 1
root@cp13:~# cat /var/log/nginx/access.log | awk '{ print $7 }' | sort | uniq -c | sort -nr | head -n 1
root@cp14:~# cat /var/log/nginx/access.log | awk '{ print $7 }' | sort | uniq -c | sort -nr | head -n 1

This equates to 816620 requests over 16 minutes. ~51038/min, ~850/s

cp13 was mitigated by OVH at 10:04.

IPs are random and not in a net - so can't easily mitigate.

@Owen: Is this worth logging with the NCSC?

It requires a mobile number, I'm happy to fill out the form but don't want to put my personal one.

Please ping here / on email as I'm out so no discord access.

It may be worth reporting.

SRE should have a joint/common UK phone number that can be used by someone in the management team - which should be used in these instances.

NCSC incident 101021-1 refers

Reception123 claimed this task.

Due to the fact that IPs are random there isn't much else we can do about this currently, so I'm resolving for now.

RhinosF1 changed the visibility from "Custom Policy" to "Public (No Login Required)".Oct 10 2021, 15:36
RhinosF1 changed the edit policy from "Custom Policy" to "All Users".

The report to Action Fruad has been filed (copy to be sent via email to SRE & Owen).

The NFIB will investigate to see if there is any reasonable action.