Hello Paladox thank you for claiming this task! I was the bureaucrat on the project's previous location and Delta has extended those rights here as well. Wikia/Fandom appears to have closed the project now. Do you still have the XML file? Pages_full zipped in a 7z was 17MB so very easy to download but I noticed when I decompressed it was 461 MB.

Do you know if there is a way to track the % completion we are of incorporating the XML data? When it wouldn't let me upload the 461MB file (I figured due to 250MB cap on file uploads) I tried to do pages_current since it was smaller (8MB compressed, 90MB decompressed) but it only added about 5 pages before giving an error.

What confuses me is that when I view https://animatedfeet.miraheze.org/wiki/Special:Log?type=import I can see my 5 adds but I can also see that a lot of history has been imported but it just doesn't show up in the log.

Am I complicating matters by doing this? I don't want to get in the way. I didn't know there was already a Phabricator Task open for this until today when I saw Void's reply at https://meta.miraheze.org/wiki/Community_noticeboard#ways_to_check_import_logs and only just set up Phabricator due to activations getting sent to junk mail.

I'm not sure how to monitor the progress of the XML import (is it at 100% now that you've moved on to help with AB's XML now?) but for whenever Phabricator has the GB available I believe archive.org has a .torrent and even a direct download of the images?

It was very big though so if you only have 70 left we can wait however many months it takes to expand storage.

https://archive.org/download/wiki-animefeetfandomcom shows the dump's .7z file to be 60.9 GB, no idea how much bigger it would get decompressed. My guess is more than the 70 you have available, and given that AnimeBaths was shut down first it only seems fair to prioritize it's recovery first, which is more feasible since there is less data needed for its image files (about 25%)

the overview at https://archive.org/details/wiki-animefeetfandomcom also links to https://archive.org/download/wiki-animefeetfandomcom/wiki-animefeetfandomcom_archive.torrent if that might be an easier way to acquire it.

I didn't set this up (it was the mysterious "Feetlord 3000" from /a/ in February) so I'm not really sure how they did this. I've never known how to back up more than individual webpages on archive.org so entire sites are beyond me. I'm not sure how hard it would be to get the format to match here.

Hello, i still have the xml file (and have restarted the import). Sorry for the late response as it seems i missed your reply :(.

I would be good to hold of on the 60g files for now.

Also even though the file is smaller (the processing of the file is what counts, which can take longer then the timeout we have configured).

Any updates on this? No comments since November yet I can't identify a clear status.

We have more storage now, how much do you need to upload?

Have just seen the download link, 60gb compressed.

So uncompressed would mean hundreds of gb i guess? So we would need to increase capacity by alot.

Stallng for now.

I'm not sure that this will be able to be imported ever, it's such a large wiki....

Please disregard that message.

@Tycio You can just mention it in this task if you'd like

60.9g wikidump from 24 feb 2019 at https://archive.org/download/wiki-animefeetfandomcom
I figure this must include the images.
Basically 2fold:

1. is it feasible to recover and import?
2. does miraheze have storage capacity for that?
3. after such an import, are there tools we could do to purge low-quality / bloated / duplicate images to help lessen Miraheze's storage burden?

I know in a lot of cases when images get deleted they're still actually there in storage (mods can view them, just not normal users) which doesn't help lessen Miraheze's storage burden
so I can only figure this would be possible via some kind of "hard delete" option on files

Prior to such a massive import, with the intent being prepared to give tools to userbase to purge undesired files to lessen our footprint, would like to know what tools could be used to help remove them quickly and FOREVER

ie for example to be able to pare down 60g to 30g as a start, maybe create some kind of nomination system where any user could nominate images for perma-purge for admin approval, to allow collaboration on such an effort

will readily admit that because of Wikia/Fandom having seemingly unlimited storage this was nver a concern and pretty much wasn't picky about people adding dupe images, bloated scale-ups, uncropped, frame-by-frame spam, etc

regret not being pickier then but can be picky now, but will need help

We have more storage now, could this be imported over the coming weeks?

I haven't forgotten this task, it just turned out more complicated than originally anticipated as all file names are in the form of revision grabber and not direct filenames - so I have had to create a script to not only rename them, but also categorise them by author to ensure attribution is retained.

while IFS=$'\t' read -r -a fileArray
do
        if [ ! -d ${fileArray[2]} ]
        then
                mkdir ${fileArray[2]}
        fi

        shortTitle=${fileArray[1]::-34}

        mv ${fileArray[0]} ${fileArray[2]}/${shortTitle:67}
done < fileData

About 50% of files should have been imported by now, this should go up to 60-65% overnight.

Tomorrow I will start the last 25% import I have prepped already, to bring it to 85-90%. The remaining percentage are classified as 'hash files' which contained hashes and special characters that the MediaWiki importImages script is not able to render for one reason or another. Tomorrow I will attempt to sort as many of these files as possible to make them importable.

I expect this task will be completed tomorrow evening or Thursday morning at the latest.

Yes it seems like it, thanks for your help!
I just wish there had been a similar archive.org download of animebaths.
The MEGA.nz also appears to be down...
Now I'm trying to remember if I had downloaded it or not before my old PC died, need to check externals.