Page MenuHomeMiraheze

Varnish/Nginx? returning 429 due to DDoS/SQLi mitigations when rendering Math Images in some cases
Closed, ResolvedPublic

Description

I have a wiki page with a lot of math formulas, many of which won't load. Opening network analysis in browser developer tools reveals that the queries for the formula images return 429 "Too many requests" errors. Is this a bug, or is there a limit on how many formulas can be loaded for one wiki page?

The URL of the wiki page is: https://schuelerwiki.miraheze.org/wiki/Trigonometrie_an_rechtwinkligen_Dreiecken

Thanks in advance!
Jan12

Event Timeline

Reception123 triaged this task as Normal priority.Sep 5 2020, 12:51
Reception123 added a project: Extensions.
Reception123 added a subscriber: Paladox.
RhinosF1 raised the priority of this task from Normal to High.Sep 7 2020, 17:09

Moving to high, a lot of wikis use this.

RhinosF1 added a subscriber: Southparkfan.

@Southparkfan, @Paladox: Could this be the rate limiter kicking into play?

RhinosF1 renamed this task from Extension:Math formulas return 429 errors to Varnish/Nginx? returning 429 when rendering Math Images in some cases.Sep 7 2020, 17:13
RhinosF1 renamed this task from Varnish/Nginx? returning 429 when rendering Math Images in some cases to Varnish/Nginx? returning 429 due to DDoS/SQLi mitigations when rendering Math Images in some cases.

Yes, because the browser is downloading the images.

Assigning to @Southparkfan since he added the rate limit to nginx.

@Southparkfan I think we should move the rate limit to varnish where you can also do it base on url (or to exclude it based on the url).

Also wasn't this suppose to be temporary? Do we really want to rate limit like this (rate limiting everything)?

If we do we should probably increase how many requests a browser can make.

Waiting for response from @Paladox regarding nginx changes.

I've had to revert my change as it was causing icinga to warn because nginx access logs were filled with 429s. Though I did raise the limit to 50r/s still not enough.

Since the client can make external requests which counts toward the rate limiting.

It seems this problem will prolong indefinitely or should we expect a solution soon? Thank you (ps: some days ago I was able to load my page without problem but then went back to the same, was the rate limiter lifted at some point?)

Edit: @RhinosF1 thanks,no pressure intended I know you all are working hard but I did need to ask since there seems to be no solution in sight from what I understand.

It seems this problem will prolong indefinitely or should we expect a solution soon?

@Paladox and @Southparkfan are working to deal with this. The rate limiter will probably exist for the foreseeable future unless we are happy are resources can cope with traffic that was causing issues (DDoS/SQLis) but we are working to improve the solution so it doesn't affect legitimate traffic.

causing icinga to warn because nginx access logs were filled with 429s.

It was doing this with the old system anyway on occasion. I assume you mean more frequently. Are 429s something we can exclude from the check? Would that be better?

Probably doesn't help to resolve the issue but just in case since it hasn't been mentioned, the rate limit due to DDoS/SQLis mitigation seems to affect only the desktop view while the mobile view works normally, so maybe the configuration of the later could give some light about the former hopefully.

In OP URL case:
https://schuelerwiki.miraheze.org/w/index.php?title=Trigonometrie_an_rechtwinkligen_Dreiecken&mobileaction=toggle_view_desktop
https://schuelerwiki.miraheze.org/w/index.php?title=Trigonometrie_an_rechtwinkligen_Dreiecken&mobileaction=toggle_view_mobile
(Is better to open one view in incognito then close and repeat with the other link so that the history from one won't affect the other)

Permanent fixes have been implemented for this issue, closing the task. If you still have issues, please reopen this task.

The problem wasn't solved in my case
https://login.miraheze.org/w/index.php?title=User:Nomalias&mobileaction=toggle_view_desktop
https://login.miraheze.org/w/index.php?title=User:Nomalias&mobileaction=toggle_view_mobile (works fine)

I think the mobile view works fine because there is no rate limit there while for the desktop view I think my page will always hit the rate limit and I will not be able to see my page as it used to be.

I was working fine but now I suppose I'm suddenly beyond the expected user case and will have to see what I do with my stuff and in any case it doesn't seems worth solving this for only one person.

I get why this task can't be really resolved but I'm still curious about why there is such a difference between the desktop and mobile view since I'm only guessing it would be nice if that can explained if possible

You shouldn't be sticking everything that could take up a whole wiki on a single userpage anyway.

I'd strongly suggest requesting or using a wiki for your content.

We're never going to account for an extreme (mis)use of resources.

Thanks, it was no extreme (mis)use until some days ago what changed is that there is a new rate limit for DDoS/SQLi but latex is not such so it could be a lot bigger and there would be no issue if it weren't for the latex interaction which doesn't happens on the mobile view which is why I feel curious.

https://nomalias.miraheze.org/ (requested as previously suggested, doesn't loads latex)
https://login.miraheze.org/w/index.php?title=User:Nomalias&mobileaction=toggle_view_desktop (loads latex partially)
https://login.miraheze.org/w/index.php?title=User:Nomalias&mobileaction=toggle_view_mobile (loads latex entirely)

You'll need to turn Extension:Math on in ManageWiki/settings on nomaliaswiki then split it up into smaller pages. Please do this within the next 2-3 days so we can delete your loginwiki page.

It should have never have been on loginwiki and should have been on a wiki anyway but given it's never caused an issue, no one has bothered to make you move it.

We've just taken action now that's broke it inadvertently.

Thanks, yes but it is not broken in the mobile view since there latex doesn't trigger a DDoS/SQLi mitigation as it should be.

On the wiki I have turned on the math extension but now I only get "No Output prod".

Tried splitting following what it says in https://template.miraheze.org/wiki/Template:Split but with {{Split|Article 1|Article 2|Article 3|date=November 2020}} I get only "Template:Split" displayed which is not what should appear as I understand.

I don't have and edit button on the wiki, I can only edit from the loginwiki.