Page MenuHomeMiraheze

Varnish/Nginx? returning 429 due to DDoS/SQLi mitigations when rendering Math Images in some cases
Open, HighPublic

Description

I have a wiki page with a lot of math formulas, many of which won't load. Opening network analysis in browser developer tools reveals that the queries for the formula images return 429 "Too many requests" errors. Is this a bug, or is there a limit on how many formulas can be loaded for one wiki page?

The URL of the wiki page is: https://schuelerwiki.miraheze.org/wiki/Trigonometrie_an_rechtwinkligen_Dreiecken

Thanks in advance!
Jan12

Related Objects

Event Timeline

Jan12 created this task.Sat, Sep 5, 10:44
Reception123 triaged this task as Normal priority.Sat, Sep 5, 12:51
Reception123 added a project: Extensions.
Reception123 added a subscriber: Paladox.
RhinosF1 raised the priority of this task from Normal to High.Mon, Sep 7, 17:09

Moving to high, a lot of wikis use this.

RhinosF1 added a subscriber: Southparkfan.

@Southparkfan, @Paladox: Could this be the rate limiter kicking into play?

RhinosF1 renamed this task from Extension:Math formulas return 429 errors to Varnish/Nginx? returning 429 when rendering Math Images in some cases.Mon, Sep 7, 17:13
RhinosF1 renamed this task from Varnish/Nginx? returning 429 when rendering Math Images in some cases to Varnish/Nginx? returning 429 due to DDoS/SQLi mitigations when rendering Math Images in some cases.

Yes, because the browser is downloading the images.

Nomalias added a subscriber: Nomalias.Mon, Sep 7, 17:23

Assigning to @Southparkfan since he added the rate limit to nginx.

@Southparkfan I think we should move the rate limit to varnish where you can also do it base on url (or to exclude it based on the url).

Also wasn't this suppose to be temporary? Do we really want to rate limit like this (rate limiting everything)?

If we do we should probably increase how many requests a browser can make.

Nomalias rescinded a token.Fri, Sep 18, 02:28

Waiting for response from @Paladox regarding nginx changes.

I've had to revert my change as it was causing icinga to warn because nginx access logs were filled with 429s. Though I did raise the limit to 50r/s still not enough.

Since the client can make external requests which counts toward the rate limiting.

Nomalias added a comment.EditedThu, Sep 24, 08:59

It seems this problem will prolong indefinitely or should we expect a solution soon? Thank you (ps: some days ago I was able to load my page without problem but then went back to the same, was the rate limiter lifted at some point?)

Edit: @RhinosF1 thanks,no pressure intended I know you all are working hard but I did need to ask since there seems to be no solution in sight from what I understand.

It seems this problem will prolong indefinitely or should we expect a solution soon?

@Paladox and @Southparkfan are working to deal with this. The rate limiter will probably exist for the foreseeable future unless we are happy are resources can cope with traffic that was causing issues (DDoS/SQLis) but we are working to improve the solution so it doesn't affect legitimate traffic.

causing icinga to warn because nginx access logs were filled with 429s.

It was doing this with the old system anyway on occasion. I assume you mean more frequently. Are 429s something we can exclude from the check? Would that be better?