Page MenuHomeMiraheze

File Storage is running low on storage
Closed, ResolvedPublic

Description

The file storage has ~889MB as of when this task was filed.

Event Timeline

Paladox triaged this task as High priority.Mar 13 2019, 18:03
Paladox created this task.
Paladox raised the priority of this task from High to Unbreak Now!.Mar 14 2019, 16:41
Paladox added a comment.Mar 14 2019, 16:42

We now only have 126 MiB with users reporting errors like "Could not store file "/tmp/phpaFjjIY" at "mwstore://local-backend/local-public/0/0e/'file name'"

How do I determine how much storage is left on each device?

Paladox added a comment.Mar 16 2019, 12:13

It’s unlimited for all users (ie we doint put a limit per user) but we currently have 119MB left for users to use.

I'm looking for baseline information about servers, volume sizes and free space by day.

NDKilla added a subscriber: NDKilla.Mar 16 2019, 21:26

Robert I previously linked you to the storage space for db4 on Facebook. The same website shows various monitoring information for all of our servers.

Would you provide that link in this task for future reference?

Thanks!

Void added a subscriber: Void.Mar 17 2019, 02:26

That would likely be icinga.miraheze.org (login as guest/guest), however it's odd, because the servers (lizardfs1 and lizardfs2) both seem to still have 17 GB free. Is this just not allocated to the file system? Thing number two, can we actually do something about this, even if it winds up being a temporary thing?

NDKilla added a comment.Mar 17 2019, 04:13

@Void / (root partition) has 17GB free on lizardfs1, but /mnt/mediawiki-static has 100% disk usage (rounded / 100 MB free)

can you add space to /mnt/mediawiki-static ?

@Rsterbal I'm not familiar with mediawiki, but I think that's good too.

Paladox added a comment.Mar 17 2019, 13:56

They have 17gb left per @John recommendation.

John added a comment.EditedMar 17 2019, 16:17

They have 17gb left per @John recommendation.

It was a soft recommendation to prevent using more than 90% disk space. Genuinely do not let “good practice” block UBN service affecting tasks. It’s never been the case before

Paladox lowered the priority of this task from Unbreak Now! to High.Mar 17 2019, 19:55

Now there is 14gb left.

John, do you have a remediation and monitoring plan in place for this defect?

Thanks!

Paladox added a comment.Mar 18 2019, 01:27

Hi, we currently have adequate monitoring in place. We will be looking at seeing if it has a api that tells us correctly how much storage is left.

Also what remediation are you looking for?

The only thing we would need to do is extend the storage which can easily be done but we are lacking funds to do it.

So maybe asking for donation through centralnotice might be the solution for the time being. Some users have already suggested it on our discord, and perhaps it's indeed high time we did so.

labster added a subscriber: labster.Mar 24 2019, 09:28

I'm not sure at what level funds are lacking -- last I heard we were trying to buy a CVZ and RN NL didn't have one available. What sort of server are we looking at to resolve this?

I'd like to see some budgeting.

I haven't given a lot to Miraheze over the years, have you?

Sometimes the solution to a technical problem is a financial one.

I'd like to see some budgeting.
I haven't given a lot to Miraheze over the years, have you?
Sometimes the solution to a technical problem is a financial one.

Yes, I agree. We are working on a fundraiser.

I'm not sure at what level funds are lacking -- last I heard we were trying to buy a CVZ and RN NL didn't have one available. What sort of server are we looking at to resolve this?

@John @Southparkfan ^

I'm not sure at what level funds are lacking -- last I heard we were trying to buy a CVZ and RN NL didn't have one available. What sort of server are we looking at to resolve this?

@Paladox told me LizardFS allows extending the available volume size by buying an extra server with lots of disk space (as an addition to our current servers), I'll assume this does not need to be yet another CVZ in NL, although that does have our preference.

CVZs won't make their way back to NL anytime soon.

Paladox added a comment.Mar 25 2019, 23:00

So we either have to get a CVZ in another region, or have to get one of the other vps with a smaller disk in NL.

Void raised the priority of this task from High to Unbreak Now!.Mar 31 2019, 15:25

If I'm not mistaken, we're out of space again.

Paladox added a comment.Apr 5 2019, 15:29

We should get a vps in hetzner for 3.01 euros that will allow us to use block chain storage (which would cost like 4.89 euros for 100gb of block chain ssd).

This would allow us to future proof miraheze file storage infrastructure.

labster added a comment.Apr 6 2019, 04:23

I'm pretty okay with this idea. If this works out, we may want to move all of the file storage to another DC, so we don't get too much replication traffic. But static data would be fine to keep in another DC.

We should definitely try to get through with this ASAP since uploads have been affected for a while now.

labster added a comment.Apr 6 2019, 18:44

The placement of erotic, pornographic, extremist material or material not deemed in good taste is not permitted.

If we went with Hetzner, we'd have to change our Content Policy, as well as our Privacy Policy.

Paladox added a comment.Apr 6 2019, 18:47

Ah thanks @labster, i guess Hetzner should be out of the question then :(.

We should try getting another vps at ramnode (but the only available data centres for the type of vps we need is in america)

Paladox added a comment.Apr 6 2019, 19:20

@labster does that mean that user won't be able to have pornographic material if we went with hetzner?

labster added a comment.Apr 6 2019, 20:00

Yes. Not permitted is the set of content ("erotic", "pornographic", extremist", "not deemed in good taste"). Therefore pornographic material is expressly prohibited, as are erotic writing and tasteless jokes.

Yes. Not permitted is the set of content ("erotic", "pornographic", extremist", "not deemed in good taste"). Therefore pornographic material is expressly prohibited, as are erotic writing and tasteless jokes.

That would be problematic as 1) it would take a while to get the changes through 2) there would still be remaining content that would be on wikis that would be nearly impossible for us to purge, and therefore we would be in breach of Hezner's policies.

Therefore I'm afraid we're probably going to have to look for an alternative, and fast.

Paladox added a comment.Apr 8 2019, 19:10

@labster would you be able to generate a budget please?

Wedhro added a subscriber: Wedhro.Apr 9 2019, 05:11
Wedhro added a comment.Apr 9 2019, 05:30

@labster does that mean that user won't be able to have pornographic material if we went with hetzner?

Blocking controversial material is one of the main reasons why we and other Uncyclopedias moved from Fandom to Miraheze in the first place, because you can't do satire without offending someone. Such a restrictive host would kill us, and we just moved here.

In T4197#81135, @Wedhro wrote:

@labster does that mean that user won't be able to have pornographic material if we went with hetzner?

Blocking controversial material is one of the main reasons why we and other Uncyclopedias moved from Fandom to Miraheze in the first place, because you can't do satire without offending someone. Such a restrictive host would kill us, and we just moved here.

Yes, as I said above that makes the option not possible.

Paladox added a comment.Apr 9 2019, 14:10

Per chat on irc, we propose that we buy $5 CVZ in NY (as they have none in NL).

Need @labster approval for ^^.

(The RamNode one)

John added a comment.Apr 9 2019, 14:14

Per chat on irc, we propose that we buy $5 CVZ in NY (as they have none in NL).
Need @labster approval for ^^.

Too long an issue, using own approval.

Paladox closed this task as Resolved.Apr 9 2019, 15:15
Paladox claimed this task.

Done.

John lowered the priority of this task from Unbreak Now! to Normal.Apr 9 2019, 15:19
John raised the priority of this task from Normal to Unbreak Now!.