Page MenuHomeMiraheze

Revise usage of third party repositories
Closed, ResolvedPublic

Description

In T5536, we discovered wildly used exploits for SaltStack were used to attack dozens of Salt masters with a certain vulnerability. We have found no proof Miraheze was affected, but we did discover that was only due to luck on our side (manual upgrade of packages due to unrelated reasons); not because unattended-upgrades worked as intended. Even while we are not vulnerable to the Salt exploits (thus immediate danger taken away), I want to take this opportunity to revise our usage of third party repositories, as it was a very contributing factor regardless.

Despite the fact both paladox and I are responsible for the investigation process, I will be inactive for the rest of this week. I am asking paladox to provide his answers and pov on the questions below, after which I will give input and my pov to his input.

  1. Why do we use third party repositories in general? What is the reason SaltStack depends on one of them?
  2. If the depedency for the third party repository was only temporarily, shouldn't it have been removed from puppet immediately?
  3. What is the security impact of third party repositories? Are there risks we have never considered before?
  4. Should we consider decreasing (or even prohibiting) these repositories?
    1. If the answer is 'yes' on either question: for all packages relying on them, can we safely remove the repositories? Why?
    2. If the answer is 'no; to both, should we discuss this further and make a policy for it? What are important points for such a policy?
  5. The fix was released multiple days before we found reports of this exploit, could we have been notified sooner? Are we not subscribed to important mailing lists?

Event Timeline

Southparkfan triaged this task as High priority.May 5 2020, 17:32
Southparkfan created this task.
Southparkfan created this object with visibility "Custom Policy".
Southparkfan created this object with edit policy "Custom Policy".

After the immediate issue in T5536 is resolved (fixed monitoring in unattended-upgrades and/or proper set up of notifcations for security advisories), this task must become public, in the light of transparency. It is only security sensitive because T5536 presents an on-going high risk to our infrastructure, and these two tasks are too related and disclosing to make them public.

Paladox added a comment.EditedMay 5 2020, 19:44

To answer the points:

  1. When i tried to use the debian repo version of salt (during the migration to the new infra), it didn't work for me. From what i remember it was permission errors on the disk? So i went with using the third party repo (again) to get it to work.
  1. We were suppose to stop using the third party repo, but due to errors that never really happened (possibly i needed to change some configs?).
  1. There is no security impact of using a third party repo, in matter of fact the developer are more likley to be secure than debian. Answering point 2, no.
  1. We should have a policies that make exception in certain cases, but in general the recommendation should be to use debian repo (or a trust worthy source like wikimedia). So the answer is yes and no. We should decrease usage where possible and make exception where we know security cannot be compromised. Now according to https://groups.google.com/forum/#!topic/salt-users/zjwt44a919U the releases in debian were vulnerable too so not using the third party repo wouldn't stop us being vulnerable just risk are higher due to new features.
  1. Yes, https://groups.google.com/forum/#!forum/salt-users No we have not subscribed to it, though i have now (i don't think i get email updates?).
Paladox added a comment.May 5 2020, 22:31

John suggests we add monitoring of packages (e.g icinga tells us when there is package updates).

John suggests we add monitoring of packages (e.g icinga tells us when there is package updates).

I think that's a great idea if we can configure that, and that way it will be easy to see when there's an update and install it right after.

Paladox added a comment.May 6 2020, 17:51

My recommendation is that we keep third party use as it is now because debian didn't upload the security package for salt either to the security repo (thus it wouldn't be updated either).

So we've improved us attending to security updates by monitoring packages for updates T5545.

For review.

Southparkfan added a comment.EditedMay 12 2020, 17:16

To answer the points:

  1. When i tried to use the debian repo version of salt (during the migration to the new infra), it didn't work for me. From what i remember it was permission errors on the disk? So i went with using the third party repo (again) to get it to work.

Permission errors...?

  1. We were suppose to stop using the third party repo, but due to errors that never really happened (possibly i needed to change some configs?).

'Errors' suggests you weren't able to change the configuration afterwards. Do you know why?

  1. There is no security impact of using a third party repo, in matter of fact the developer are more likley to be secure than debian. Answering point 2, no.

I do not agree with this statement. Debian commits to providing security updates for the packages they host, whereas I haven't seen a commitment from third party repository providers. Besides that, about all third party repositories are not as much scrutinised by the FOSS/Debian community as the official repositories.

  1. We should have a policies that make exception in certain cases, but in general the recommendation should be to use debian repo (or a trust worthy source like wikimedia). So the answer is yes and no. We should decrease usage where possible and make exception where we know security cannot be compromised. Now according to https://groups.google.com/forum/#!topic/salt-users/zjwt44a919U the releases in debian were vulnerable too so not using the third party repo wouldn't stop us being vulnerable just risk are higher due to new features.
  1. I agree with the recommendation.
  2. Unfortunately, there is no such thing as 100% secure in IT, not even for the official repositories. Good risk management determines which route (include or exclude repository X) should be chosen in our environment.
  3. I never said we wouldn't have been vulnerable if we used the official Debian repository. This was an vulnerability in many versions of Salt, the download route did not have impact on the presence. However, what is important here too is the speed of patching. It looks like the patches in debian-security were distributed on May 6th, which is awfully late. Is that date correct?
  1. Yes, https://groups.google.com/forum/#!forum/salt-users No we have not subscribed to it, though i have now (i don't think i get email updates?).

Cool, which email address? Our sre one?

Paladox added a comment.May 12 2020, 23:12

@Southparkfan i forgot what the error was exactly, it could have been the config is different under the debian packaging then under salts repo.

I thought doing https://github.com/miraheze/puppet/commit/e6e4a18381bd259ba15555e6bb697e193a89fcaf#diff-ce0006a6fb896e893e50ae83249001aa would fix it but it didn't so reverted.

The official source is always going to be faster at patching, unless the company/developer tells debian and debian patches faster (which i think is unlikely as we can see with salt).

I've signed up to https://groups.google.com/forum/#!forum/salt-users using my @yahoo.com account (though seeing as possibly we're going with cumin i guess singing up to that is not really needed now as the package has been removed).

Southparkfan added a comment.EditedMay 19 2020, 18:27

The official source is always going to be faster at patching, unless the company/developer tells debian and debian patches faster (which i think is unlikely as we can see with salt).

That might be correct, but Debian comes with a security policy and commitment. Third party repositories don't, we have no control over them. That is potentially more dangerous than waiting a few days.

I'd like to resolve this task very quickly. With the help of @Reception123 we discovered Miraheze uses the following third party repositories:

  1. https://packages.sury.org/php/ (php packages)
  2. https://packages.grafana.com/oss/deb (grafana)
  3. https://download.gluster.org/pub/gluster/glusterfs/7/LATEST/Debian/${::lsbdistcodename}/amd64/apt (gluster)
  4. http://packages.icinga.com/debian (icinga)
  5. http://ams2.mirrors.digitalocean.com/mariadb/repo/stretch/debian (mariadb)
  6. http://nginx.org/packages/debian (nginx)
  7. https://deb.nodesource.com/node_10.x (nodejs)
  8. http://apt.puppetlabs.com (puppet)
  9. http://repo.saltstack.com/py3/debian/10/amd64/2019.2 (salt)

Nine repositories, do we really need all of those? I doubt it. What I'd like to propose:

  • The need for each repository must be explained here. If we cannot reasonably understand why it is absolutely necessary to keep a repository, said repository will be removed and their GPG keys must be removed from the trust database.
  • For the leftover repositories, the security contact will subscribe SRE to security announcements (if available). If there is no announcement mailing list, they will advise on other measures.
  • And related to this task, announcements for packages: T5536#109739
Paladox added a comment.EditedMay 19 2020, 19:45

Here's my response in order:

  1. We can get rid of it I think, unless we want to use php7.4/8 (when it comes out). Not really any good reasons to keep it as we're on debian buster. We could downgrade php on misc1 though.
  1. There is no grafana debian repository in debian. (As in debian is not supplying the package through their apt repo).
  1. Gluster cannot be downgraded (debian buster have version 5 in the main one, and 6 in the backports repo. One of the 7.x releases fixed a memory leak which has helped to keep the memory low! Before it was using much more memory than it is currently. If we want to move to using debian, we could look at doing this under bullseye (11).
  1. We're currently using icinga version 2.11 (version 2.10 is in the debian repo). We can just reinstall the db if we want to downgrade.
  1. SPF said it him self, this is a critical service, so we're not downgrading this.
  1. NGINX could be downgraded. I’m not sure if the version in buster supports TlS 1.3?
  1. When I last looked at npm, it was very old, seems the package was added in stretch (a more up to date one). Version 5 is in buster.
  1. We migrated to PuppetServers years ago which involved migrating some files inside the puppet repo. We had to make the switch as at some point we would have been forced to due to the puppet passenger thing not being maintained anymore (puppetserver being their new thing).
  1. We're likely getting rid of salt.
Southparkfan added a comment.EditedMay 20 2020, 18:59

Sury repo removed. @Paladox now switching over roundcube so we can remove the role (and delete the php 7.4 packages as well) from misc1. Repositories to remove:

  • nodejs
  • icinga
  • nginx

Edit: rest comes tomorrow.

Southparkfan closed this task as Resolved.May 22 2020, 20:08

Two repositories have been removed and we have incorporated new processes regarding security announcements.

Southparkfan changed the visibility from "Custom Policy" to "Public (No Login Required)".May 22 2020, 20:08
Southparkfan changed the edit policy from "Custom Policy" to "All Users".