It seems this has been investigated but the source hasn't been found. @John would you have any ideas about why this is happening?

I do not - I haven't touched the code in about 2 years and the fact this started to happen immediately after an upgrade suggests something changed and CreateWiki wasn't tested correctly.

So far, it appears to be that the cache files aren't being updated properly. The wiki will be created, but databases.json and <wiki>.json won't reflect this change. The databases.json file won't list it, and the later simply doesn't exist.

I'm seeing the following stack traces in graylog which don't seem possible:

PHP Warning: rename(/srv/mediawiki/cache/databases.json.tmp,/srv/mediawiki/cache/databases.json): No such file or directory
from /srv/mediawiki/w/extensions/CreateWiki/includes/CreateWikiJson.php(151)
#0 [internal function]: MWExceptionHandler::handleError(integer, string, string, integer, array)
#1 /srv/mediawiki/w/extensions/CreateWiki/includes/CreateWikiJson.php(151): rename(string, string)
#2 /srv/mediawiki/w/extensions/CreateWiki/includes/CreateWikiJson.php(83): Miraheze\CreateWiki\CreateWikiJson->generateDatabaseList()
#3 /srv/mediawiki/w/extensions/CreateWiki/includes/Hooks.php(54): Miraheze\CreateWiki\CreateWikiJson->update()
#4 /srv/mediawiki/w/includes/HookContainer/HookContainer.php(160): Miraheze\CreateWiki\Hooks->onSetupAfterCache()
#5 /srv/mediawiki/w/includes/HookContainer/HookRunner.php(3372): MediaWiki\HookContainer\HookContainer->run(string, array)
#6 /srv/mediawiki/w/includes/Setup.php(410): MediaWiki\HookContainer\HookRunner->onSetupAfterCache()
#7 /srv/mediawiki/w/includes/WebStart.php(86): require_once(string)
#8 /srv/mediawiki/w/index.php(44): require(string)
#9 {main}

Given that line 150 of CreateWikiJson.php checks that the file exists before renaming, I'm wondering how a file could possibly disappear in that timespan.

In T10351#208612, @Void wrote:

So far, it appears to be that the cache files aren't being updated properly. The wiki will be created, but databases.json and <wiki>.json won't reflect this change. The databases.json file won't list it, and the later simply doesn't exist.

I'm seeing the following stack traces in graylog which don't seem possible:

PHP Warning: rename(/srv/mediawiki/cache/databases.json.tmp,/srv/mediawiki/cache/databases.json): No such file or directory
from /srv/mediawiki/w/extensions/CreateWiki/includes/CreateWikiJson.php(151)
#0 [internal function]: MWExceptionHandler::handleError(integer, string, string, integer, array)
#1 /srv/mediawiki/w/extensions/CreateWiki/includes/CreateWikiJson.php(151): rename(string, string)
#2 /srv/mediawiki/w/extensions/CreateWiki/includes/CreateWikiJson.php(83): Miraheze\CreateWiki\CreateWikiJson->generateDatabaseList()
#3 /srv/mediawiki/w/extensions/CreateWiki/includes/Hooks.php(54): Miraheze\CreateWiki\CreateWikiJson->update()
#4 /srv/mediawiki/w/includes/HookContainer/HookContainer.php(160): Miraheze\CreateWiki\Hooks->onSetupAfterCache()
#5 /srv/mediawiki/w/includes/HookContainer/HookRunner.php(3372): MediaWiki\HookContainer\HookContainer->run(string, array)
#6 /srv/mediawiki/w/includes/Setup.php(410): MediaWiki\HookContainer\HookRunner->onSetupAfterCache()
#7 /srv/mediawiki/w/includes/WebStart.php(86): require_once(string)
#8 /srv/mediawiki/w/index.php(44): require(string)
#9 {main}

Given that line 150 of CreateWikiJson.php checks that the file exists before renaming, I'm wondering how a file could possibly disappear in that timespan.

That can happen if there is a race condition. By the looks of it, the update() method in SetupAfterCache, is always reading database changes, so attempting to reset the database as well. And the same exact time from when called from WikiManager, leading to a race condition where it gets deleted before the rename is executed but after the check.

Just an idea though. Can't be sure.

The odd thing is that absolutely 0 issues seem to be associated with the CreateWikiJob itself (including the trace I've found) other than the fact that the maintenance scripts can't run due to the wiki not being found.

I've also just tested creating wikis from eval.php, and it worked both times. Not really sure what it means for our bug, but it is sure annoying that it can't be replicated in a testing environment. However, once I tested with CreateWiki, it failed the first time.

Some details that may further help debug the situation:

• When this bug occurs, databases.json has a timestamp equal to the wiki_creation field in cw_wikis.
• The entry for the wiki is missing from databases.json.
• The wiki.json file is missing.
• Forcing the wiki cache to update results in the wiki redirecting to miraheze.org/wiki/Main_Page temporarily (likely a result of missing wgServer)

I'm wondering either if something earlier in the CreateWiki process (before WikiManager::create) causes the databases.json file to update early with the same timestamp that the later refresh would cause, or if something in the process causes lag between WikiManager's $dbw and CreateWikiJson's $dbr (i.e. that the update to cw_wikis isn't picked up by CreateWikiJson for some reason).

I was unable to reproduce this bug in a local testing instance.

@Void Could it be clarified to wiki creators if current wiki requests should be approved or not? We had a slight altercation recently on the Miraheze Discord/IRC channels about the topic, and wikis have continued to be approved (though likely no permissions granted or anything) as if nothing has gone wrong. Would appreciate clarification on that part as there seems to be a great deal of confusion surrounding it.

In T10351#208982, @BrandonWM wrote:

@Void Could it be clarified to wiki creators if current wiki requests should be approved or not?

Ideally, due to the nature of the cleanup effort required for fixing affected wikis, I would prefer if no new wikis were created at this time.

I've done some extra work on attempting debugging this issue, and I'm going to update the task description with a full list of my findings.

In T10351#208983, @Void wrote:

Ideally, due to the nature of the cleanup effort required for fixing affected wikis, I would prefer if no new wikis were created at this time.

Gotcha, thanks for the clarification. We just had 30+ wikis created over the past 2 hours or so, so that will have to eventually be fixed. Hopefully, it can all be rectified soon. I also believe @Universal_Omega just disabled Special:RequestWiki on Meta, so that should at least cease all requests. And if it's relevant/helpful at all: two requests are currently in the queue.

I didn't. I had a PR but after discussions we want to maybe do it differently. But I'm not around much right now, so someone else will have to do it however it is decided.

One of the wiki creators lacked full context to what was happening and accidentally created several wikis thinking that we could just manually assign permissions for the time being, from what I can tell this was wrong and caused some problems, because I am pretty sure that this is not only a permissions error.

This doesn't appear to be a Miraheze isolated incident, but definitely seems to be something that changed between 1.39 and 1.39.1. I'm running CW and can confirm the same behaviour happens. When a wiki is created, I get a success message, and everything seems to be alright. The database file <database>.json is created for the wiki, but the entry isn't added to the databases.json file, so $wgServer is just being set as the domain name and not the wiki domain. Just thought I'd add a bit more to the debugging of sorts :)

In T10351#209050, @Original_Authority wrote:

This doesn't appear to be a Miraheze isolated incident, but definitely seems to be something that changed between 1.39 and 1.39.1. I'm running CW and can confirm the same behaviour happens. When a wiki is created, I get a success message, and everything seems to be alright. The database file <database>.json is created for the wiki, but the entry isn't added to the databases.json file, so $wgServer is just being set as the domain name and not the wiki domain. Just thought I'd add a bit more to the debugging of sorts :)

Could you clarify whether you'd confirm for sure that it worked on 1.39 (not the RC)? Since we can't seem to remember whether we tested it on 1.39 RC or on the actual 1.39 version. Since that would narrow things down a little. But thanks for your help either way!

In T10351#209059, @Reception123 wrote:

In T10351#209050, @Original_Authority wrote:

This doesn't appear to be a Miraheze isolated incident, but definitely seems to be something that changed between 1.39 and 1.39.1. I'm running CW and can confirm the same behaviour happens. When a wiki is created, I get a success message, and everything seems to be alright. The database file <database>.json is created for the wiki, but the entry isn't added to the databases.json file, so $wgServer is just being set as the domain name and not the wiki domain. Just thought I'd add a bit more to the debugging of sorts :)

Could you clarify whether you'd confirm for sure that it worked on 1.39 (not the RC)? Since we can't seem to remember whether we tested it on 1.39 RC or on the actual 1.39 version. Since that would narrow things down a little. But thanks for your help either way!

Yeah, it definitely worked on the official released version of 1.39; I didn't test on the RC version, but it was definitely—and has been for the months since the release—working on 1.39. I'm not sure if there were any breaking changes between RC and the official release, and didn't test it so couldn't provide any information on that, but it definitely works on 1.39 official release.

I've identified the cause. OAUTHAuth appears to be starting a database transaction (in MediaWiki\Extension\OATHAuth\OATHUserRepository::findByUser) that "consumes" the CreateWiki database interactions. This means that until that transaction (which we can't control) is committed, the changes to cw_wikis won't be visible to other database readers that are not in this transaction.

Interesting — I'm not using OAUTHAuth and the issue persists for me. Does that definitely look to be the root cause?

In T10351#209162, @Original_Authority wrote:

Interesting — I'm not using OAUTHAuth and the issue persists for me. Does that definitely look to be the root cause?

No, I've done some more digging, and it appears to instead be related to how MediaWiki handles database transactions. Basically everything gets included into the same database transaction, which is seems really shoddy and likely to cause more issues than it will solve. I'm also not finding any way to create my own transaction or force the transaction to commit.

I've gotten it working on test131 with https://github.com/miraheze/CreateWiki/pull/376, waiting for CI before I merge.

In T10351#209186, @Void wrote:

I've gotten it working on test131 with https://github.com/miraheze/CreateWiki/pull/376, waiting for CI before I merge.

Interesting, this seems to work for me too — although the URL of the wiki isn't being passed to $wgServer, but instead just the domain. Perhaps this is an issue with my setup? But nonetheless, it seems that patch did indeed fix the config-dbname.json and dbname.json issue.

Edit: scratch that, seems as though that patch works perfectly!

Hmm, curiously, this doesn't seem to have fixed every problem. For whatever reason, databases.json seems to still get stuck with the wrong content. I suspect this could be due to the fact that other processes could be resetting the database list before we get to it.