Forum Outage

[Calum]

@NINJA Yeah okay bud.

It looks like the server has been crashing and rebooting quite a fair bit by the looks of it.

Everytime you seen that error message, that means the server has just died.

There are three threads of this forum running at any one time. These are proxied to a single thread of the Database. It looks like when one thread dies, the proxy load balances to another thread. If you kill all three of them, you'll be facing the blue message whilst you wait for the processes to come back alive.

I think I'm going to have to get some new hardware over the next few weeks.

I will investigate other options in the meantime.

We may lose some data...

[NINJA]

@Calum OK good_2_know that you know what's going on. Hopefully you can get it all sorted and not lose to much???

I just got this message;

ERROR
MISCONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk. Commands that may modify the data set are disabled. Please check Redis logs for details about the error.

Followed by this when I refreshed the page;

[Calum]

@NINJA yeah that is just nginx x. Looks like all three processes died.

[Calum]

Okay this is getting ridiculous.

I've isolated the problem and it sounds like the forum is getting a little too big for its boots.

I don't think the hardware is failing, which is good news. But it does sound like it is bottlenecked. Unfortunately the hardware isn't upgradeable. Especially the parts that it is having problems with.

Ideally I think the solution would be to migrate the database to a different technology. This would result in a slower website, but would be infinitely more reliable.

[Mightyman]

Would you consider moving the forum to a faster machine altogether? Maybe a VPS?

Vultr.com $2.50 per month for a decent enough Linux server. I'm sure a few of us on here would be happy to donate a few quid to help the cause

[Calum]

@Mightyman Lol, that's worse specs than what we're already running.

[Calum]

You're talking spending $10 a month to beat the spec's of our current server.

Might not sound a lot, but it's going from $0 a year, to $120 a year, for five or six members lol.

I am going to keep a close on the situation. I noticed the machine only had 100Mb Swap, I've upped it to a Gb and the problem has gone away.

But the issue is with the software used, it will reoccur once the Database reaches a certain size again, by design.

I think long term, we need to migrate away from the current Database and move to a more reliable service.

The guys who wrote this forum do offer this as a service, at $50. Which is a better price to pay as a one off than pay for hosting which is no better than what we got.

But even then, it'll only be a script to migrate the database. Shouldn't be too hard to figure out.

[Calum]

So upping the Swap file has seemed to make a noticeable difference. Haven't encountered this database problem so far.

But as I say, it'll only be a matter of time.

[Calum]

Just quickly upgraded the operating system and forum software.

[Calum]

There is likely to be a forum outage tomorrow as the power is scheduled to be disconnected.

I don't know for how long, but obviously I will endeavour to put it back online as soon as power is resetablished.

Kind Regards,

Calum

[Louis-DT-WR]

@Calum cheers for all the administration good someone's keeping the forum ticking over..

Any update on the wall of fame idea?

[NINJA]

@Calum OK - Good_to_know.

[Calum]

@Louis-James Yeah, the code is all written. I need to migrate the forum to redirect all requests from dt125r.co.uk/ to /forum.

So every single link, image, post and topic needs to be migrated.

I then need to migrate the database from Redis to Mongo, might as well throw in a new server to boot. Then deploy the code.

Probably won't be done this side of christmas.

[NINJA]

@Calum Forum keeps going down, connection lost and won't reload.

[declan]

@NINJA fine for me today

[Calum]

@NINJA Not seeing anything obvious in the logs.

[NINJA]

@Calum MMMMMMMM weird must have been something on my end then???

[Calum]

@NINJA Doubtful, I wouldn't discredit the forums stability. Just have to appreciate we're doing it on a shoe string budget.

[NINJA]

@Calum Nylon or Teflon laces??? :grinning_face_with_smiling_eyes: :smiling_face_with_open_mouth_closed_eyes: :smiling_face_with_open_mouth_cold_sweat: :face_with_tears_of_joy:

[Calum]

@NINJA Phwoar, if I was paying that much then I would have thrown in the towel long ago.