puppet3 almost ran out of storage today:

DISK CRITICAL - free space: / 1424 MB (5% inode=92%);

The cause: the file /var/log/puppetlabs/puppetserver/puppetserver-access.log.json.1, containing a whopping 13 GB of logs. Removing the file and restarting puppetserver has fixed the issue for now: puppet3 is back to 56% disk usage. If we do not fix the issue, we'll encounter the same issue within 30 days:

For the non-json variant of this log file, we have a rolling policy, but that is not the case for the json log file. The logs are automatically ingested into Graylog via syslog-ng. Therefore, we should reconsider deleting the access logs sooner (max retention date of three days?). You may need to reload syslog-ng after rotating a file: T5044#136990.