IABot is hitting a rate limit, inhibiting it from working on the several wikis on Miraheze and is also holding up a process that runs through wikis sequentially. I would like to request it be exempt from the rate limit.
Description
Description
Event Timeline
Comment Actions
@Cyberpower678 I can confirm on cp9 (cache proxy) traffic from IABot is being rate limited. However, sending over 40 requests per second (on average) to api.php (and thus requests that bypass the cache, and have to be served by the MediaWiki cluster) is in most cases not acceptable at all. We only send about 70 requests per second to the MediaWiki cluster.
Is there a reason you cannot slow the bot down? What is your use case for IABot, scraping the wikis using api.php? If so, what is the POST data you're including in your request? Can we help you by generating your content server-side, so we can lessen the load on the php-fpm cluster?
Comment Actions
The bot has to run on multiple wikis, with Miraheze being remote from the VM it runs on. Due to technical limitations, it can only run one wiki at a time. Which mean in order to run through all of the wikis in a timely fashion, to keep up with the workload, it operates at that speed. If it helps, that was just for initial indexing which is what the bot was doing. Afterwards, it does delta indexing which is significantly less burdensome on the server, but the rate cannot be slowed down.
Comment Actions
@Cyberpower678 sending 45 requests per second is still a lot. Remember this is a multi-tenant environment with few spare resources. However, if you can give me a combination of source IP and User-Agent to exempt for the rate limit temporarily, I can see how much issues are actually caused in production by the bot. I'm curious.
Comment Actions
No response in over a week, so declining for now. If this is still needed please feel free to reopen the task and respond.
Comment Actions
I somehow missed the responses to this ticket.
IABot queries with a UA starting with IABot/2 and the IP is 185.15.56.22