Forum Outage

[Calum]

How odd, forum seems to be playing up.

Could be the HDD failing so I will back up the database and monitor it.

We may lose data if it fails.

[NINJA]

@Calum Is everything fixed now? As it seems awfully quiet in here???

[Calum]

@NINJA May do, stats say the site is drawing in over 400,000 hits a month with over a 4000 unique visitors a month.

Think of all that money in ad revenue...that I don't make

[NINJA]

@Calum Missed opportunity there bud!!! :smiling_face_with_open_mouth_cold_sweat: :face_with_tears_of_joy:

[NINJA]

@Calum The forum is going down a lot lately, as I keep getting the 'Down for maintenance message'. This is happening at different times of the day, so I'm not sure if it's a hardware, server or hoster issue? Can you look into it please bud.

[Calum]

@NINJA Oh really? I've not seen this my end.

Is it the 503 Message? With the Blue Screen?

It's pay day this weekend, maybe I ought to upgrade a few things.

[NINJA]

@Calum No just the 'down for maintenance message' - I'll screenshot it for you next time.

[NINJA]

@Calum The site was just down with the blue 503 message.

[Calum]

@NINJA Yeah I know the one. I guess the poor server is just tasked to capacity. I don't think there really is much I can do about that. A faster server maybe.

[NINJA]

@Calum I've not had the blue 503 message for a long while, but the other message is more frequent. You should ask the server company to put more 50p's in the meter slot more regularly!!! :grinning_face_with_smiling_eyes: :smiling_face_with_open_mouth_closed_eyes: :smiling_face_with_open_mouth_cold_sweat: :face_with_tears_of_joy:

[Calum]

@NINJA Yeah okay bud.

It looks like the server has been crashing and rebooting quite a fair bit by the looks of it.

Everytime you seen that error message, that means the server has just died.

There are three threads of this forum running at any one time. These are proxied to a single thread of the Database. It looks like when one thread dies, the proxy load balances to another thread. If you kill all three of them, you'll be facing the blue message whilst you wait for the processes to come back alive.

I think I'm going to have to get some new hardware over the next few weeks.

I will investigate other options in the meantime.

We may lose some data...

[NINJA]

@Calum OK good_2_know that you know what's going on. Hopefully you can get it all sorted and not lose to much???

I just got this message;

ERROR
MISCONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk. Commands that may modify the data set are disabled. Please check Redis logs for details about the error.

Followed by this when I refreshed the page;

[Calum]

@NINJA yeah that is just nginx x. Looks like all three processes died.

[Calum]

Okay this is getting ridiculous.

I've isolated the problem and it sounds like the forum is getting a little too big for its boots.

I don't think the hardware is failing, which is good news. But it does sound like it is bottlenecked. Unfortunately the hardware isn't upgradeable. Especially the parts that it is having problems with.

Ideally I think the solution would be to migrate the database to a different technology. This would result in a slower website, but would be infinitely more reliable.

[Mightyman]

Would you consider moving the forum to a faster machine altogether? Maybe a VPS?

Vultr.com $2.50 per month for a decent enough Linux server. I'm sure a few of us on here would be happy to donate a few quid to help the cause

[Calum]

@Mightyman Lol, that's worse specs than what we're already running.

[Calum]

You're talking spending $10 a month to beat the spec's of our current server.

Might not sound a lot, but it's going from $0 a year, to $120 a year, for five or six members lol.

I am going to keep a close on the situation. I noticed the machine only had 100Mb Swap, I've upped it to a Gb and the problem has gone away.

But the issue is with the software used, it will reoccur once the Database reaches a certain size again, by design.

I think long term, we need to migrate away from the current Database and move to a more reliable service.

The guys who wrote this forum do offer this as a service, at $50. Which is a better price to pay as a one off than pay for hosting which is no better than what we got.

But even then, it'll only be a script to migrate the database. Shouldn't be too hard to figure out.

[Calum]

So upping the Swap file has seemed to make a noticeable difference. Haven't encountered this database problem so far.

But as I say, it'll only be a matter of time.

[Calum]

Just quickly upgraded the operating system and forum software.

[Calum]

There is likely to be a forum outage tomorrow as the power is scheduled to be disconnected.

I don't know for how long, but obviously I will endeavour to put it back online as soon as power is resetablished.

Kind Regards,

Calum