Page MenuHomeMiraheze

Collect Statistics for API Requests (Including Module Type)
Closed, ResolvedPublic

Description

It would be useful for us to get a full picture of the number of API requests we have on average in a minute/sec and then the types of modules and requests being executed. Would allow us to more quickly and easily pin point increased resource usages due to bots suddenly misusing API requests as well as guide potential caching attempts.

Event Timeline

John triaged this task as Normal priority.Mar 15 2021, 17:18
John created this task.

For this, looping over access logs may be of use (Graylog dashboards could also help?) on a minute-minute basis and reporting usages directly to either Grafana or via Prometheus may be the best option rather than setting up specific data storage services currently.

In T6979#138313, @John wrote:

https://grafana.com/docs/grafana/latest/datasources/elasticsearch/ data could alternatively be collected directly via ES potentially

I told you on IRC yesterday, but it's good practice to document this on Phabricator: my suggestion is to use https://github.com/braedon/prometheus-es-exporter, a tool that runs on the graylog hosts. The tool takes an elasticsearch query, performs the search and returns the result in prometheus format. Prometheus collects the metrics, after which we can use the metrics in Grafana dashboards.

ACLs in elasticsearch are hard (but aren't needed so far, since elasticsearch only listens on 127.0.0.1), but the prometheus-es-exporter tool is a nice solution.
Wikimedia queries: https://github.com/wikimedia/puppet/tree/f9bdcb97b5d1a6154bfa033f0cac292ede3710a1/modules/prometheus/files/es_exporter
Puppet classes: https://github.com/wikimedia/puppet/blob/f9bdcb97b5d1a6154bfa033f0cac292ede3710a1/modules/prometheus/manifests/es_exporter.pp

Unknown Object (User) unsubscribed.Apr 3 2021, 19:56
Unknown Object (User) subscribed.Aug 6 2021, 17:24
Reception123 removed a subscriber: Unknown Object (User).Aug 24 2021, 19:05
Unknown Object (User) claimed this task.Oct 16 2021, 23:31
Unknown Object (User) moved this task from Unsorted to Goals on the Universal Omega board.Oct 16 2021, 23:31
Unknown Object (User) added a commit: R9:59d959adb0a8: Enable `api-request` logging (#4156).
Unknown Object (User) added a comment.Oct 16 2021, 23:47
log_mediawiki_mediawiki_channels_doc_count{instance="graylog2.miraheze.org:9206", job="elasticsearch", mediawiki_channels="api-request"}

Works to get the total amount of API requests now, but not module type. I'm not exactly sure how to continue now.

Unknown Object (User) added a comment.Oct 17 2021, 19:09

https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki now has total API requests, will still attempt to do module specific.