cpu/mem jumped on cp*, all depooled. Everything down.
Description
Description
Status | Assigned | Task | ||
---|---|---|---|---|
Resolved | None | T8140 High cpu/mem on all cache proxies (10 Oct 2021 11:05) | ||
Resolved | RhinosF1 | T8141 All wikis are down (10 Oct 2021) |
Event Timeline
Comment Actions
root@cp15:~# cat /var/log/nginx/access.log | awk '{ print $7 }' | sort | uniq -c | sort -nr | head -n 1 208350 https://miraheze.org/"
root@cp12:~# cat /var/log/nginx/access.log | awk '{ print $7 }' | sort | uniq -c | sort -nr | head -n 1 226925 https://miraheze.org/"
root@cp13:~# cat /var/log/nginx/access.log | awk '{ print $7 }' | sort | uniq -c | sort -nr | head -n 1 198560 https://miraheze.org/"
root@cp14:~# cat /var/log/nginx/access.log | awk '{ print $7 }' | sort | uniq -c | sort -nr | head -n 1 182785 https://miraheze.org/"
This equates to 816620 requests over 16 minutes. ~51038/min, ~850/s
cp13 was mitigated by OVH at 10:04.
Comment Actions
@Owen: Is this worth logging with the NCSC?
It requires a mobile number, I'm happy to fill out the form but don't want to put my personal one.
Please ping here / on email as I'm out so no discord access.
Comment Actions
It may be worth reporting.
SRE should have a joint/common UK phone number that can be used by someone in the management team - which should be used in these instances.
Comment Actions
Due to the fact that IPs are random there isn't much else we can do about this currently, so I'm resolving for now.
Comment Actions
The report to Action Fruad has been filed (copy to be sent via email to SRE & Owen).
The NFIB will investigate to see if there is any reasonable action.