Page MenuHomeMiraheze

Help with migration
Closed, ResolvedPublic

Description

I have recently signed up for a wiki and would like to migrate an existing wiki.

The URL is https://wiki.wikimedia.cat/ and it's currently running Mediawiki 1.31.6.

I have access to the server that's running it, and on IRC, RhinosF1 told me that the best way would be to generate a full sql dump and send it to you. I was also warned that I had to make sure that users updated their recovery email, because their access password would have to be regenerated.

Users have been warned, so now I would like to move on with the migration.

My plan is to:

  • set wiki to Read-only
  • dump database and send it to you (not sure if mail will be able to handle the size, is there an alternate way to send you files?)
  • cross fingers and hope there are no user collisions.
  • you import the wiki
  • I ask for the DNS to be changed.

The DNS guys only work on weekdays. I would like to set a date that suits everybody, send one last warning to the users, and go ahead with it.

Is the plan all right?

The command to dump the full database would be something like "mysql -u <user> -p --database=<databasename> dump > bigfile.sql"?
I have one more question: the current bureaucrats, admins, and so on, will keep their privileges inside the wiki?

Thanks in advance

Event Timeline

RhinosF1 triaged this task as Normal priority.May 10 2021, 21:18
RhinosF1 added projects: Database, MediaWiki.
RhinosF1 added a subscriber: Reception123.

@Reception123: ping

This will likely take a few days once we start to complete.

The plan sounds fine. My suggestion would be to start as soon as possible.

OK, we'll get back to you.

{F1437857}What's the best way to send the dump. Can we just upload it to phabricator?
I am uploading a list of the user names, in case you need to check for conflicts.

You can upload it to Phabricator yes.

Sysadmin plan (for us)

  1. Create blank wiki with our stock stuff.
  2. Import your dump
  3. Drop the blank wiki's tables that are in the dump
  4. Run renameDBPrefix.php
  5. Run update.php to make sure tables are fully in sync
  6. Run the CentralAuth migrate scripts (pester me to do this)

Well, the blank wiki is already there: https://amical.miraheze.org/

Otherwise, that's fine. I'll try to create the dump tomorrow evening, we will leave an additional day for users to tidy things up.

Can't we import an XML dump instead and create user accounts (with the 'send temporary password via email' option) beforehand? Importing the SQL dump comes with multiple challenges (need to check schema, character sets, sync CentralAuth and ManageWiki tables with local users and potential exotic config).

Can't we import an XML dump instead and create user accounts (with the 'send temporary password via email' option) beforehand? Importing the SQL dump comes with multiple challenges (need to check schema, character sets, sync CentralAuth and ManageWiki tables with local users and potential exotic config).

In theory we can but then they'd loose old logs, and any access accounts had would have to be resynced.

Can't we import an XML dump instead and create user accounts (with the 'send temporary password via email' option) beforehand? Importing the SQL dump comes with multiple challenges (need to check schema, character sets, sync CentralAuth and ManageWiki tables with local users and potential exotic config).

In theory we can but then they'd loose old logs, and any access accounts had would have to be resynced.

Well I personally at least have only ever done SQLs by importing them into an empty Miraheze wiki and then exporting them in XML version to import into the desired wiki. I think importing a pure SQL and then syncing/fixing is way too complex and not really feasible, as well as bound to break something

It shouldn't be too complex. Update.php should simply work. We're at a huge advantage that they have a prefix on the tables so we can sync data post import. It'll cause the wiki to go down for a bit but I don't see any reason why it won't work post us setting it up. I should be able to do most of the work myself.

It shouldn't be too complex. Update.php should simply work. We're at a huge advantage that they have a prefix on the tables so we can sync data post import. It'll cause the wiki to go down for a bit but I don't see any reason why it won't work post us setting it up. I should be able to do most of the work myself.

Okay, if you're confident about it, I think we can give it a try at least.

@RhinosF1 It would probably be good if you would make a list of steps to do (like CentralAuth scripts, what would happen if there's character set mismatches, etc.) so we don't get stuck in the middle of it and have to go looking then.

@RhinosF1 It would probably be good if you would make a list of steps to do (like CentralAuth scripts, what would happen if there's character set mismatches, etc.) so we don't get stuck in the middle of it and have to go looking then.

CA is migrateAccount.php

If a charecter set mismatches, I doubt there'd be much user facing impact but it should be a case of just checking the tables. We will know more once we have the actual dump which I intend on looking through as much as I can.

I can do every step but the actual import the sql.

I do have one concern though. Their privacy policy says no data will be shared with Third Parties. That needs to be looked into to ensure it can be modified to allow sharing data with us.

Quick note: when uploading the dump to Phab @Amical: please make sure only acl*security can see

I have concerns about this. Is the requestor requesting user accounts be imported? If so, I disagree that we should be doing this. We can just an interwiki prefix and specify --username-prefix="interwikiprefix" to link to and reference the user's contributions.

Can't we import an XML dump instead and create user accounts (with the 'send temporary password via email' option) beforehand? Importing the SQL dump comes with multiple challenges (need to check schema, character sets, sync CentralAuth and ManageWiki tables with local users and potential exotic config).

In theory we can but then they'd loose old logs, and any access accounts had would have to be resynced.

This is quite minor. This is one of the downsides to migrating to Miraheze. They wouldn't lose their logs, actually; they just wouldn't have them on Miraheze.

Quick note: when uploading the dump to Phab @Amical: please make sure only acl*security can see

Sorry, I can't find the option to do that.

Can't we import an XML dump instead and create user accounts (with the 'send temporary password via email' option) beforehand? Importing the SQL dump comes with multiple challenges (need to check schema, character sets, sync CentralAuth and ManageWiki tables with local users and potential exotic config).

In theory we can but then they'd loose old logs, and any access accounts had would have to be resynced.

This is quite minor. This is one of the downsides to migrating to Miraheze. They wouldn't lose their logs, actually; they just wouldn't have them on Miraheze.

But we are migrating for good, if we don't have them on Miraheze, we don't have them anywhere. Same with accounts. There won't be an interwiki to go back to.

This is the dump. Thanks in advance!{F1438957}

This is the dump. Thanks in advance!{F1438957}

We'll review it as soon as we can to confirm everything.

How's it going? Do you have an estimation on when it can be done?

Hello, do you have an estimation about when the migration will be ready?

Hello, do you have an estimation about when the migration will be ready?

Not yet. I'll try and chase it up soon.

Update?

I was going to ask you. Can we import this to a blank db to review charsets and stuff?

Update?

I was going to ask you. Can we import this to a blank db to review charsets and stuff?

Sure, that's not an issue

@Dmehus didn't you have contact regarding an XML import?

Hello, do you have an estimation about when the migration will be ready?

@Amical We'd prefer t

@Dmehus didn't you have contact regarding an XML import?

@Southparkfan I replied to Amical and Cc"ed you and @Reception123.

Dmehus moved this task from Backlog to Short Term on the MediaWiki (SRE) board.
Dmehus removed a project: Database.

I hope XML works. We just asked our users to create a user in Miraheze, and we are having problems with mail confirmation. They can't access the page because it's limited to members. By the moment, I am moving them to the "users" group and I will ask them to retry. Is there a better way to handle that?

I plan to take the XML snapshot tomorrow. Do you have some kind of guide on what exactly I need to extract and how? Please, I would like to get it right on the first try.

I would like to have wiki.wikimedia.cat as the main name for the wiki. How do we go about the certificate?

This comment was removed by Amical.

Hello, do you have an estimation on when the import could be done?

I'm temporarily claiming this pending a response from @Amical or Joan, the other contact with which I've been liaising, to decide on the interwiki prefix to use.

Apologies for the delay, @Amical. I've now added the interwiki prefix for this wiki's XML import from above (permalink). I see it looks like all or most of your wiki's users have created user accounts on Miraheze, and you've assigned each of them the member user group. Please have a look over everything and confirm if we are good to proceed.

@Reception123, when you have the 👍 from @Amical to proceed with the import, please specify --username-prefix="amical" in your importDump.php command and, additionally, if possible, add #[[phab:T7278|T7278]] to the end of your server admin log log entry.

If there are any images to import, you will also need to specify -r Miraheze maintenance script. Thank you.

Yes, everything is ready, please proceed. The dump was made with the following command:

php dumpBackup.php --full --include-files --uploads --output=gzip:../../../dumpamical.xml.gz

So the images should already be included, I expect.

Is something wrong, now? You're blowing all the deadlines you set yourselves. If you can't do it, please say so and I will look for an alternative, I'm running out of time.

Is something wrong, now? You're blowing all the deadlines you set yourselves. If you can't do it, please say so and I will look for an alternative, I'm running out of time.

Apologies for this. @Reception123 just saw this now, and will proceed immediately, following the instructions at https://phabricator.miraheze.org/T7278#150045.

I'm so sorry about this, while this is no excuse we have a lot of open tasks currently and very few system administrators who are able to deal with them. The import is now running but will take a long time as it contains a lot of revisions. I hope you can forgive us for how long this task took.

Is it working? Because I don't really see any changes, twelve hours later. Do you have an estimation on how long it will take?

It's in progress. You should see pages start to appear. We've seen large imports take days before.

Is it working? Because I don't really see any changes, twelve hours later. Do you have an estimation on how long it will take?

If you look at https://amical.miraheze.org/wiki/Especial:Llista_de_p%C3%A0gines you will definitely see that it is working as RhinosF1 said. Imports do take very long.

Is it still going? I haven't seen the list move for a while now.

Yes, I just checked and it's still going.

It's taking way longer than I expected. Can you tell what percentage is done by now?

One question. If we edit the wiki now, can we cause some kind of conflict?

It's taking way longer than I expected. Can you tell what percentage is done by now?

I've just checked and it says that the import is completed now.

The File: namespace has quite a few pages, but no file. Do you know how I can extract them and upload them to Miraheze?

And the Main Page has a problem, because {{#widget}} isn't working. Could you please install the widgets extension?

The File: namespace has quite a few pages, but no file. Do you know how I can extract them and upload them to Miraheze?

If you want to upload a large number of files we can do that for you. You would just need to get the raw files from wherever you store them now and send them to us via a Dropbox, Google Drive, etc. or even upload them here with a Zip.

And the Main Page has a problem, because {{#widget}} isn't working. Could you please install the widgets extension?

Unfortunately that won't be possible as Widgets extension has been removed due to past security concerns. You can see alternatives here and you can ask at our Community Noticeboard if you need help with that. I'm sorry if this is an inconvenience to you.

This comment was removed by Amical.

I have created a compressed tar file of the images in the directory. Can you please grab it and delete it, if this phab is public?{F1459452}

Done. It's downloaded now so you can delete it from Phab.

Oops, I have removed my comment, but the file can still be downloaded from yours.

Everything seems fine now. I think we can close this ticket. Just remember to delete the tar.gz file.

Thank you very much!

Everything seems fine now. I think we can close this ticket. Just remember to delete the tar.gz file.

Thank you very much!

Yes the images are uploading, though if you wanted to delete them from Phabricator you haven't done so yet (just noting that since my understanding is you don't want them on Phab)

The images are done now. @Amical I now realise that you wanted us to delete the file, while technically possible Phabricator makes that difficult for admins and requires us to do it via the server, so it would be easier if you could do it with the click of a button.

Your understanding is correct, but I don't know how to delete the files in Phabricator. I see that even the first dump is still there. Can you please do it for me?

The images are done now. @Amical I now realise that you wanted us to delete the file, while technically possible Phabricator makes that difficult for admins and requires us to do it via the server, so it would be easier if you could do it with the click of a button.

I think I found it now. Done!