Page MenuHomeMiraheze
Create Task
Maniphest T7599

Graylog search not working
Closed, ResolvedPublic
Actions

Description

It's not possible to search anything on Graylog search. Relevant errors seem to be:

a few seconds ago	graylog_231	8691ce60-de29-11eb-8767-0200001a24a4	ElasticsearchException[Elasticsearch exception [type=cluster_block_exception, reason=index [graylog_231] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];]]
Elasticsearch nodes disk usage above flood stage watermark (triggered 4 days ago)
Elasticsearch nodes disk usage above high watermark

Event Timeline

Reception123 triaged this task as High priority.Jul 6 2021, 07:13
Reception123 created this task.
Herald added subscribers: Bukkit, Unknown Object (User), RhinosF1. · View Herald TranscriptJul 6 2021, 07:13
Void added a comment.EditedJul 7 2021, 20:54

Graylog has not ingested any new logs in roughly the past four days. I'm looking to clear some disk space, but we need to immediately find a way to reduce our logging input or expand our storage capability for Graylog.

As a temporary solution, I may cut our retention down to 20 days instead of 30 to reduce the storage requirements.

Or possibly disable the MediaWiki deprecation warnings entirely, as they are causing undue load, and are put in the errors stream instead of the deprecations stream. We don't need these types of errors every time someone loads a page with an extension that hasn't yet figured out how to migrate from an unsupported hook with no replacement.

Void claimed this task.Jul 7 2021, 20:54
Void added a comment.Jul 7 2021, 23:18

Graylog is working now, but I'm leaving this open so we can reduce our input.

Void added a comment.Jul 13 2021, 01:20

Created https://github.com/miraheze/mw-config/pull/3997, but not yet willing to merge. Thoughts?

Void moved this task from Incoming to Short Term on the Infrastructure (SRE) board.Jul 13 2021, 01:21
Reception123 added a comment.Jul 13 2021, 07:32
In T7599#153307, @Void wrote:

Created https://github.com/miraheze/mw-config/pull/3997, but not yet willing to merge. Thoughts?

Seems fine with me. We know what the deprecation errors are and there's no need to have them spam our logs.

Reception123 added a comment.Jul 13 2021, 07:38

@Void It seems that it's not working again.

Unknown Object (User) raised the priority of this task from High to Unbreak Now!.Jul 13 2021, 07:39
In T7599#153341, @Reception123 wrote:

@Void It seems that it's not working again.

Lack of logging should be UBN I think.

Redmin subscribed.Jul 13 2021, 09:02
Void lowered the priority of this task from Unbreak Now! to High.Jul 13 2021, 14:15

Cleared some disk space, will need to monitor if our recent changes are sufficient to prevent any further issues.

Void added a comment.Jul 13 2021, 14:17

FYI if anyone needs to get graylog working again, go to System > Indices > Default index set, and delete the oldest indexes (oldest at bottom) until there is at least 15% disk space available.

Void lowered the priority of this task from High to Low.Jul 17 2021, 21:40

Still monitoring this, but our storage usage is down to 10GB per day of logs from of 30GB per day. I think we can sustain this without difficulty.

Void closed this task as Resolved.Jul 20 2021, 23:30

Operating at 77% disk usage, so looks good. Feel free to reopen if icinga reports a disk usage warning, or you see any disk usage warning in graylog.