Page MenuHomeMiraheze
Create Task
Maniphest T6858

Messages take a while to be sent to graylog
Closed, ResolvedPublic
Actions

Description

For example if there's a spammy message such as this one https://graylog.miraheze.org/messages/graylog_92/fb119430-702b-11eb-b57d-0200001a24a4 then it appears that when we fix it, it takes a while before it stops.

Is there a way we can speed up messages being sent/processed without using too much resources (io/cpu)?

Related Objects

Duplicates Merged Here
T6872: Process messages instantly

Event Timeline

Paladox triaged this task as Normal priority.Feb 16 2021, 16:55
Paladox created this task.
Herald added subscribers: Unknown Object (User), RhinosF1. · View Herald TranscriptFeb 16 2021, 16:55
Paladox added a comment.Feb 16 2021, 16:56

the only problem is that this causes things to be out of sync, for example a message could have been sent an hour ago but we would have only just saw it now because of this. This has happened with exceptions when investigating. This impedes investigations.

Southparkfan added a comment.Feb 17 2021, 22:31

Do you know what the bottleneck is? Is it MediaWiki -> syslog-ng or syslog-ng -> graylog?

Paladox added a comment.Feb 17 2021, 22:37

I think it may be syslog-ng -> graylog. Has to be otherwise messages would be lost going from mw -> syslog-ng.

John moved this task from Incoming to Short Term on the Infrastructure (SRE) board.Feb 18 2021, 15:03
John merged a task: T6872: Process messages instantly.Feb 19 2021, 23:17
John added a subscriber: Reception123.
Paladox raised the priority of this task from Normal to High.Feb 19 2021, 23:17
Paladox closed this task as Resolved.Feb 26 2021, 02:07
Paladox claimed this task.

Think we've resolved this by using pipeline rules to use timestamp in the json and setting the timestamp.