@John would simply following https://piwik.org/docs/update/#the-manual-three-step-update be enough to update to 3.x?

Yes

Reception123> well downloading the file gave me piwik.zip, extracted that gave me a piwik folder (so it was /home/reception/piwik/piwik)
I moved that, so now the actual files are stored in /srv/piwik/piwik/piwik 
so I'm not sure if I'm supposed to remove the current contents of /srv/piwik and replace it with the one in /srv/piwik/piwik

Note: I did that, and when I go to Piwik it prompts me for a completely new installation. (When I replaced the current /srv/piwik with the new files)

@John Do I proceed with that? (Does the database stay intact on new installation?)

Piwik is managed by puppet 100%, why are you downloading anything at all?

@John You did say "Yes" when I asked :P

How should I proceed to update to 3.x?

Change the branch in puppet to get the files.

Oh, so Puppet fetches it. Will do

@John

I changed the branch but then realized that the actual repo (https://github.com/piwik/piwik) doesn't have actual branches for each version (don't know how it worked for the previous one) but only has tags.

Should I use https://github.com/piwik/piwik/tree/3.x-dev or somehow clone the tag for the exact release?

@John
The 2.17.1 tag works set as branch, but the 3.2.1 doesn't for some reason. Can't figure out why..