Page MenuHomeMiraheze

unattended-upgrades does not monitor third party repos
Closed, ResolvedPublic

Description

RamNode recently suffered from an exploit where SaltStack tools were used to mine bitcoin on their nodes, because SaltStack did not properly check input from unauthorised users. Miraheze was patched after two days of the fix being released, but only because @Paladox performed an upgrade due to unrelated reasons.

The reason for the delay? unattended-upgrades (which periodically upgrades packages using Miraheze's provided configuration - in our case, only debian-security updates are eligible for automatic installation) does not check third party repos, which we use multiple times.

Event Timeline

Southparkfan assigned this task to Paladox.EditedMay 5 2020, 17:19

As discussed in the staff channel, paladox and I will work on the investigation of the SaltStack issue (why did it happen, what could we have done better, etc) and this task. Since I have IRL constraints (need to work on large projects for college), and I do not want to be the constraining party here, I have asked paladox to work on the first part of the investigation.

paladox is an important party since he installed and maintained Salt, I am involved because it is a security related task.

Questions to answer, specific to this issue:

  1. Why did unattended-upgrades not monitor the Salt packages and install their updates in time? (hint: cause known, but needs a proper write-up)
  2. What is the root cause? Was this a technical malfunction or an oversight? Was there a lack of training or awareness?
  3. How can we get unattended-upgrades to monitor packages in third party repositories as well?

(small note: in the case you have added monitoring for all third party repos in use, this task can be made public, since in that case there is nothing here to warranty the security sensitive status of this task)

Eventually, if done right, we can answer the following question: how can we fix unattended-upgrades' handling of security fixes?

Paladox added a comment.May 5 2020, 22:10

To summerise, the problem is here https://github.com/miraheze/puppet/blob/master/modules/base/files/apt/50unattended-upgrades#L1 . We need to add the salt repo to here so that upgrades happen, only downfall is the repo is not just for security updates like debian's so new features could be pulled in.

Paladox added a comment.May 5 2020, 22:41

Well even debian didn't publish the security update in the security repo so we still wouldn't have got the update even if we were using debian.

Paladox added a comment.May 6 2020, 16:26

So here are the changes that have been done:

  • Monitoring switched on for packages updates (icinga) done in T5545.

So overall this task is mostly complete now i'll leave this up to @Southparkfan in case he does not find this satisfactory and rather we installed packages automatically from upstream third party repos.

Paladox added a comment.May 6 2020, 16:33
This comment was removed by Paladox.

For review.

Thanks for looking, paladox. From what I have seen here you have not added automatic installs for third party repositories; instead, you have blacklisted even more packages from automatically installing, thus delaying security patch installs for those packages. What is the rationale for this? Are these packages likely to contain faulty updates? Don't we at the very least need to subscribe ourselves to security related announcements for most of these services?

It is not easy to say what the best method for automatic installation from third party repositories would be. Unlike Debian (where the 'security repository' is separate, and thus you can tell unattended-upgrade to only apply security updates very easily), third party repositories do not allow clients to distinguish between security and normal updates. An option is to allow all updates from third party repositories, at the cost of stability (which is even more of an concern in said repositories - Debian hammers on stability, others might not do so).

Paladox added a comment.May 12 2020, 17:53

Thanks for looking, paladox. From what I have seen here you have not added automatic installs for third party repositories; instead, you have blacklisted even more packages from automatically installing, thus delaying security patch installs for those packages. What is the rationale for this? Are these packages likely to contain faulty updates? Don't we at the very least need to subscribe ourselves to security related announcements for most of these services?

It is not easy to say what the best method for automatic installation from third party repositories would be. Unlike Debian (where the 'security repository' is separate, and thus you can tell unattended-upgrade to only apply security updates very easily), third party repositories do not allow clients to distinguish between security and normal updates. An option is to allow all updates from third party repositories, at the cost of stability (which is even more of an concern in said repositories - Debian hammers on stability, others might not do so).

Well if critical services are upgraded automatically and it goes wrong then we have an outage till one of the SRE are around. There's always a big risk doing automatic upgrades for critical services. I nginx it went from 1.16 -> 1.18, what if 1.18 had a bug that broke nginx?

We now monitor package upgrades, also because third party repos don't have security repos, we are always pulling the latest unless it's packaged like salt (versioned e.g 2xxxx or 3xxxx repos). Gluster is latest 7.x version, or pinning it to a minor release like 7.4. Nginx it's the latest stable release.

If we don't mind outages then we can do automatic installs for all packages.

Well if critical services are upgraded automatically and it goes wrong then we have an outage till one of the SRE are around. There's always a big risk doing automatic upgrades for critical services. I nginx it went from 1.16 -> 1.18, what if 1.18 had a bug that broke nginx?

Correct, automatic updates always carry a risk, especially those jumping major versions. That is the reason why I prefer the Debian policy, focused on reliability and security, instead of new functionality.

We now monitor package upgrades, also because third party repos don't have security repos, we are always pulling the latest unless it's packaged like salt (versioned e.g 2xxxx or 3xxxx repos). Gluster is latest 7.x version, or pinning it to a minor release like 7.4. Nginx it's the latest stable release.

Got that. See T5542 for follow up.

On top of that: ideally we get as many security announcements (or incorporate a regular check, automated or not) as possible. Though, for now, start with announcements for all repositories and whitelisted packages.

On top of that: ideally we get as many security announcements (or incorporate a regular check, automated or not) as possible. Though, for now, start with announcements for all repositories and whitelisted packages.

For security announcements unfortunately for most (if not all) there was no updated security announcements mail list, so I've subscribed us to the main release emails and then main checks will have to be done by the SRE member on duty.

Southparkfan changed the visibility from "Custom Policy" to "Public (No Login Required)".May 22 2020, 20:36
Southparkfan changed the edit policy from "Custom Policy" to "All Users".