Page MenuHomeMiraheze
Create Task
Maniphest T10458

puppet141 disk full and files not deleting / failed TCP connection puppet
Closed, ResolvedPublic
Actions

Description

/var/log/ssl/miraheze-renewal.log was 25GB and contained a lot of "Lock 140340809382544 not acquired on /tmp/tmp_file.lock, waiting 0.05 seconds" entries. I deleted the file but disk space is still full for some reason.

Related Objects

Event Timeline

Reception123 triaged this task as Unbreak Now! priority.Feb 8 2023, 11:27
Reception123 created this task.
Herald added subscribers: BrandonWM, Agent_Isai, Unknown Object (User), Void. · View Herald TranscriptFeb 8 2023, 11:27
Reception123 closed this task as Resolved.Feb 8 2023, 11:33
Reception123 claimed this task.

After reboot getting

Error: Connection to https://puppet141.miraheze.org:8140/puppet/v3 failed, trying next route: Request to https://puppet141.miraheze.org:8140/puppet/v3 failed after 0.001 seconds: Failed to open TCP connection to puppet141.miraheze.org:8140 (Connection refused - connect(2) for "puppet141.miraheze.org" port 8140)

when trying to run puppet

Reception123 reopened this task as Open.Feb 8 2023, 11:38

whoops, didn't mean to resolve

Reception123 renamed this task from puppet141 disk full and files not deleting to puppet141 disk full and files not deleting / failed TCP connection puppet.EditedFeb 8 2023, 11:38

This could be unrelated but could there possibly be an issue with the new private key being discussed in T10441 that was set? It seems things started failing not long after and I don't see what else could've caused it.

Reception123 closed this task as Resolved.Feb 8 2023, 12:42

Fixed by manually restarting puppetdb and puppetserver.

Restricted Repository Identity mentioned this in rPUPC7f04665a95c9: mirahezerenewssl.py: raise logging level to info.Feb 8 2023, 18:53
Void added a comment.Feb 8 2023, 18:54

Not sure what caused the space to still be full after deleting the file, but see https://github.com/miraheze/puppet/pull/3145 for what I believe to have been the cause.

Restricted Repository Identity mentioned this in rPUPC10ce09668f73: mirahezerenewssl.py: raise logging level to info (#3145).Feb 8 2023, 19:12