Page MenuHomeMiraheze

Import Request for dragalialost.miraheze.org
Closed, ResolvedPublic

Description

Requesting an XML and image import for the above Wiki. Sending link of the XML (646 MB) and image ZIP (>2GB) dumps to tech@miraheze.org.

Event Timeline

A lot has been imported but it seems an error relating to Cargo has been encountered (cc @Paladox )

#0 /srv/mediawiki/w/extensions/Cargo/includes/CargoSQLQuery.php(69): CargoUtils::getTableSchemas(Array)
#1 /srv/mediawiki/w/extensions/Cargo/includes/parserfunctions/CargoCompoundQuery.php(87): CargoSQLQuery::newFromValues('Events', 'Name=name,Start...', NULL, NULL, NULL, NULL, NULL, NULL, NULL)
#2 /srv/mediawiki/w/includes/parser/Parser.php(3340): CargoCompoundQuery::run(Object(Parser), '', 'tables=Events;f...', 'format=calendar')
#3 /srv/mediawiki/w/includes/parser/Parser.php(3047): Parser->callParserFunction(Object(PPFrame_Hash), 'cargo_compound_...', Array)
#4 /srv/mediawiki/w/includes/parser/PPFrame_Hash.php(253): Parser->braceSubstitution(Array, Object(PPFrame_Hash))
#5 /srv/mediawiki/w/includes/parser/Parser.php(2887): PPFrame_Hash->expand(Object(PPNode_Hash_Tree), 0)
#6 /srv/mediawiki/w/includes/parser/Parser.php(1556): Parser->replaceVariables('{{Note|Return t...')
#7 /srv/mediawiki/w/includes/parser/Parser.php(651): Parser->internalParse('{{Note|Return t...')
#8 /srv/mediawiki/w/includes/content/WikitextContent.php(374): Parser->parse('{{Note|Return t...', Object(Title), Object(ParserOptions), true, true, 56864)
#9 /srv/mediawiki/w/includes/content/AbstractContent.php(590): WikitextContent->fillParserOutput(Object(Title), 56864, Object(ParserOptions), true, Object(ParserOutput))
#10 /srv/mediawiki/w/includes/Revision/RenderedRevision.php(263): AbstractContent->getParserOutput(Object(Title), 56864, Object(ParserOptions), true)
#11 /srv/mediawiki/w/includes/Revision/RenderedRevision.php(235): MediaWiki\Revision\RenderedRevision->getSlotParserOutputUncached(Object(WikitextContent), true)
#12 /srv/mediawiki/w/includes/Revision/RevisionRenderer.php(215): MediaWiki\Revision\RenderedRevision->getSlotParserOutput('main')
#13 /srv/mediawiki/w/includes/Revision/RevisionRenderer.php(152): MediaWiki\Revision\RevisionRenderer->combineSlotOutput(Object(MediaWiki\Revision\RenderedRevision), Array)
#14 [internal function]: MediaWiki\Revision\RevisionRenderer->MediaWiki\Revision\{closure}(Object(MediaWiki\Revision\RenderedRevision), Array)
#15 /srv/mediawiki/w/includes/Revision/RenderedRevision.php(197): call_user_func(Object(Closure), Object(MediaWiki\Revision\RenderedRevision), Array)
#16 /srv/mediawiki/w/includes/Storage/DerivedPageDataUpdater.php(1315): MediaWiki\Revision\RenderedRevision->getRevisionParserOutput()
#17 /srv/mediawiki/w/includes/Storage/DerivedPageDataUpdater.php(1665): MediaWiki\Storage\DerivedPageDataUpdater->getCanonicalParserOutput()
#18 /srv/mediawiki/w/includes/Storage/DerivedPageDataUpdater.php(1452): MediaWiki\Storage\DerivedPageDataUpdater->doParserCacheUpdate()
#19 /srv/mediawiki/w/includes/page/WikiPage.php(2172): MediaWiki\Storage\DerivedPageDataUpdater->doUpdates()
#20 /srv/mediawiki/w/includes/import/ImportableOldRevisionImporter.php(218): WikiPage->doEditUpdates(Object(MediaWiki\Revision\RevisionStoreRecord), Object(User), Array)
#21 /srv/mediawiki/w/includes/import/WikiRevision.php(669): ImportableOldRevisionImporter->import(Object(WikiRevision))
#22 /srv/mediawiki/w/includes/import/WikiImporter.php(371): WikiRevision->importOldRevision()
#23 /srv/mediawiki/w/maintenance/importDump.php(201): WikiImporter->importRevision(Object(WikiRevision))
#24 /srv/mediawiki/w/includes/import/WikiImporter.php(507): BackupReader->handleRevision(Object(WikiRevision), Object(WikiImporter))
#25 /srv/mediawiki/w/includes/import/WikiImporter.php(999): WikiImporter->revisionCallback(Object(WikiRevision))
#26 /srv/mediawiki/w/includes/import/WikiImporter.php(864): WikiImporter->processRevision(Array, Array)
#27 /srv/mediawiki/w/includes/import/WikiImporter.php(802): WikiImporter->handleRevision(Array)
#28 /srv/mediawiki/w/includes/import/WikiImporter.php(612): WikiImporter->handlePage()
#29 /srv/mediawiki/w/maintenance/importDump.php(353): WikiImporter->doImport()
#30 /srv/mediawiki/w/maintenance/importDump.php(286): BackupReader->importFromHandle(Resource id #1007)
#31 /srv/mediawiki/w/maintenance/importDump.php(130): BackupReader->importFromFile('/home/reception...')
#32 /srv/mediawiki/w/maintenance/doMaintenance.php(107): BackupReader->execute()
#33 /srv/mediawiki/w/maintenance/importDump.php(358): require_once('/srv/mediawiki/...')
#34 {main}

I'm not sure what the error is @Reception123 you just pasted the traceback without it telling me what the error is.

And also no exception id either.

Apologies if any of this is completely off the mark as this is mostly me making assumptions about the traceback, but the query looks pretty close to the ones on these two pages:

Currently, I have Cargo turned off, but with Cargo on, visiting these pages throw Internal errors, e.g.

[4c67999318a1d8f2dd3e0612] 2020-11-06 21:03:51: Fatal exception of type "MWException"

My guess it that it's because the Cargo query is looking for Cargo datatables that haven't been instantiated yet, as creating these datatables is reliant on several Templates that are also part of the import. We (the Dragalia Lost Wiki team) were basically expecting most of the pages with Cargo queries to be broken because of this, but since all of us are unfamiliar with the import process, we're unsure if that's something that would impact it.

While neither are direct fixes, two ideas I thought of were maybe either:

  • Importing the templates and instantiating the Cargo datatables first. I'm not sure if this would cause conflicts with the original import XML though.
  • Or alternatively, we could do the import with the Cargo extension off on the Wiki, since I would assume that would negate the Cargo parser functions activating and erroring out. Afterwards, the other editors and I can iron out the Cargo problems after on a page-by-page basis, since we were expecting to do this anyway. This is assuming the traceback error isn't something that would bork the entire Wiki if we turned Cargo on post-import though.
2020-11-06 21:03:51 mw5 dragalialostwiki: [4c67999318a1d8f2dd3e0612] /wiki/Events/Calendar   MWException from line 265 of /srv/mediawiki/w/extensions/Cargo/includes/CargoUtils.php: Error: Table Events not found.

So looks like you need to create the table before using:

{{#cargo_compound_query:|
tables=Events;fields=Name=name,StartDate=start,EndDate=end
|format=calendar
}}

Sorry if I wasn't clear, but yes, the error thrown on that page is because the table hasn't been created yet. This is because the Cargo table it's expecting is created from a Template, which the import hasn't done yet due to hitting the Cargo error.

I think that upstream issue is a non-issue since even with Cargo on, I believe it's still possible to edit the page. Even then, usually missing tables can be resolved by creating the needed table from a template on a different page.

My thoughts from my last post were that, if the import error is along the same lines as the error on the linked pages, then to look for a solution to bypass Cargo so that the rest of the import could be completed. The rest of the Cargo issues I'm sure can be resolved after everything gets imported.

Sorry again if my initial post wasn't clear, but hopefully it makes more sense this time.

The import seems to have gone fine now

Elaeagnifolia reopened this task as Open.EditedSat, Nov 7, 18:17

Re-opening since image import still not done.

Also checking, it seems that there are several pages missing from the import. For example, it doesn't look like any of the pages not in the Main namespace were imported (Templates, Modules, etc.)

https://dragalialost.miraheze.org/wiki/Template:Adventurer
https://dragalialost.miraheze.org/wiki/Module:CargoQuery

Checking the pages in the Main namespace, it looks like it goes up to The Great Destroyer. In the XML, the page right after this is The Great Geysers, but this page does not exist.

I'm not sure if the page statistics have been updated to reflect all the pages on the Miraheze Wiki yet, but to compare:

https://dragalialost.miraheze.org/wiki/Special:Statistics
https://dragalialost.gamepedia.com/Special:Statistics

Elaeagnifolia changed the task status from Open to Stalled.Sun, Nov 8, 05:10

After checking, it looks like the XML might be corrupt. Going to double-check and send an updated version, sorry for the trouble.

I have sent the updated XML to tech@miraheze.org.

Elaeagnifolia changed the task status from Stalled to Open.Sun, Nov 8, 17:44

Looking through the image dump again, it looks like since Gamepedia's image files were moved to Wikia's image servers, the original image dump I sent was also not useable b/c of a bug in the Wikiteam dump script. I sent an updated image dump to tech@miraheze.org.

Also opened up a pull request to get that fixed in WikiTeam's dumpgenerator.py here: https://github.com/WikiTeam/wikiteam/pull/396

Sorry for the delay, will begin the import for the new XML dump soon

The new XML dump has now been completed. I will now look at the image dump.

@Elaeagnifolia Looking at the files, it doesn't seem like they're image files? Could you perhaps double check the zip you sent?

The original image ZIP as I mentioned wasn't dumped correctly, so I sent an updated Google Drive link on Nov. 9 to tech@miraheze.org. The updated ZIP should be named dragalialost-20201108-images.zip. I double-checked, and it does look like it has images files in it. If there is still some trouble with the newer ZIP though, please let me know!

Looking through the image dump again, it looks like since Gamepedia's image files were moved to Wikia's image servers, the original image dump I sent was also not useable b/c of a bug in the Wikiteam dump script. I sent an updated image dump to tech@miraheze.org.

Also opened up a pull request to get that fixed in WikiTeam's dumpgenerator.py here: https://github.com/WikiTeam/wikiteam/pull/396

@Elaeagnifolia Hi yes, the second zip has valid images. I will begin the import.