Chaos Issues

[Peaches]

EIHorizons
Horizons Management Team


Join Date: Jul 2006
Posts: 7 Chaos Shard Alert

--------------------------------------------------------------------------------

The Control Server for the Chaos Shard experienced a major fault/failure. The server is going to require an operating system rebuild, and perhaps the replacement of hardware, as we do not know the full extent of the damage to the server system. We will continue to work on the problem as quickly as we can and will have service restored on Chaos just as soon as humanly possible.

[Casyle]

My gosh... Almost sounds like we may be waiting days, not hours.

[Spartikus]

Yes days. They had the backup for Unitas and its been over 6 weeks and no sign of it up yet.. Wonder how they will bring Chaos up any sooner since they are now in a similar situation..

Sad day for chaos players. My chars are from Chaos but I wrote them off a long time ago.

Still at least you all know whats up now. Lets see Unitas backup was recieved right before christmas and its now Feb.. Could be a month or so for anymore news.

I wish ya all luck.

If you want to all play nice the Order people might want to organize an event to welcomethe Chaos and Blight peeps who have lost everything in the last 6 weeks.

[Theros Ironfeld]

I was thinking more along the line of
"We didnt pay the power bill and the generator out back has no fuel ... sorry"

[Casyle]

Tis a sad day for HZ players. Lord knows how many are going to leave over this.

Maybe HZ is just doomed, period... *Sighs and ponders*

[Steelclaw]

About time! Thanks for ... some kind of info.

[BKBanzai]

Glad we got some word, thanks very much.

Future updates could go along the lines of:

(EXAMPLES
We've determined that there is some hardware replacements required, we estimate that the server will not come up until late Friday evening - Saturday morning.

Have a nice restful weekend folks, we anticipate Chaos will be down until next Tuesday.

OR

We're done working for the evening and we'll be back at it early Friday morning, don't bother trying to log in until at least 12 PM PST or we'll update again before then.


(END EXAMPLES)

[MadDwarf]

Limelight is currently hosting the game servers.

These types of failures broadcast themselves well in advance. Hence chaos has been having issues for weeks. This usually reaks of a hardware failure somewhere. One has to ask why it was not corrected before a total meltdown happened ?

Why did Limelight not alert PME ? Could it be perhaps they were owed ???

Being a sys admin I just cannot see such a failure happening outright without the server communicating symptoms before hand . Servers are like that ya know they have internal components that let the admins know they are sick . Why just last week I got an email from mr Destiny our SQL server telling me that a raid controller was having issues. In the meantime he was switching to redundant measures. The next day I replaced it. What a nice little server . Me thinks MR chaos is cranky.

something isn't right.

But what do I know

[Foxfire]

Also having a sys admin background, I'll agree that you often get warnings about pending hardware failures. But sometimes, things just go belly up. I'm more struck that there isn't redundancy of some sort provided.

Well, if they do have to get new hardware, here's to hoping they buy newer faster stuff to replace the broken stuff. You can usually get twice the speed and capability (memory, etc.) for the same price every 2-3 years, sometimes less.

*chant* Upgrade! Upgrade! Upgrade! */endchant*

[Poppi]

We will continue to work on the problem as quickly as we can and will have service restored on Chaos just as soon as humanly possible.

Welcome to the "soon" waiting list Chaos!

[anjinson]

So after taking a sabbatical from the game, I come back to take a look around to see what new messes I can get myself into. Only to find, the only really new thing is new owners of the game. (which isn't really a new thing either) Both my characters on blight are under "maintenance" never to be played again, and my 4 characters on Chaos are under "maintenance". Yelp, same game, different day. The only question here is how much bubble gum will it take to patch this problem to get the shard back up? And yes, bovine dung happens. I think the real frustration here is, that after a long frustrating day at work your driving home thinking I'll just hop into an online role playing game and escape my real life issues for a bit, only to be frustrated again when you go to log on. Got to love it!!!!

[Steelclaw]

Hmm... "operating system rebuild" so like, reinstall Windows?

Computer electronics generally don't give warning signs of impending failure, they just stop working. This is especially true of server grade hardware that has been functioning for months and is kept in a controlled environment where it doesn't cook.

On the other hand, mechanical things like disk drives usually do give warnings and limp along before giving out completely. The sort of hide-and-seek db lag we were seeing and the necessity to reinstall the OS supports this type of failure.

Mitigating the damage from such a failure is pretty straightforward: redundancy and flexibility. Replicating the live data on a slave is a first step, however, any node should be able to pick up the role of any other node too (by changing the configuration and running different programs). That way, downtime is measured in minutes or hours, not days.

I know that what happened has already happened. The question is, will anything be done so that a failed node doesn't cause extended downtime again?

[Screech]

So from having uh... 4 shards when they took over we have...

Blight - deleted.
Unity - deleted.
Chaos - gone from "having issues" to "Eek the server died"

Everyone piles onto Order....

*stares at the server trembling in terror*

[Katscratch]

I wonder if this has any impact on Horizons http://www.gameintruder.com/SavageNews.htm Its an announcement of Major Content Upgrade to Savage Eden (Their words not mine)

[Kerech]

I don't understand why it took 36 hours for them to determine that it was a hardware failure... or if they knew that from the start, why didn't they tell us?

[AA0]

Katscratch, SE has been "to receive major update" for over a year now. They have strung out the players time and again making many many excuses blaming the original devs for the issues (even though all the other servers have had the patch). Look at the similarities between what happens, and you'll figure out the future pretty fast

[MadDwarf]

Computer electronics generally don't give warning signs of impending failure, they just stop working. This is especially true of server grade hardware that has been functioning for months and is kept in a controlled environment where it doesn't cook.
?

Blade servers ftw ? Even standard box servers

I get an email and a page when anything even remotely minor happens. From raid controllers to even a video subsystem to yes the ever telling hard drive failure. Computer electronics do indeed give fair warnings when something is wrong. From system crashes to bios resets to general strange behaivor of your computer. Most times they start fortelling their problems LONG before they actually fail. The ONLY time I have had systems fail on startup and I went duh its dead is when they suffered from power surges or some fool tech worked on it without practicing ESD procedures. But in the case of I press a button and magic blue smoke comes out the back issue the end user is probaby aware that their computer is fried. 15 years in the IT biz and I have never had a server just die because I always knew of any problems long before it got to the point of just dying.

thing is Choas has been limping along for weeks now with their "fixing" never fixing. That should have been the first sign IF the servers had no monitoring systems that something was drastically wrong.

However in this case... I wonder if it was improper maintenance procedures that caused the database to collapse in on itself and not an actual hardware failure. In that case you hope PME has a non corrupted DB to restore from.

Who knows really and well I can speculate we all can speculate till the cows come home. I hope for the folks of chaos the issue gets repaired and PME at least gives the community updates.

[Kerech]

Yeah, it's hard to say there weren't any warning signs. We've had problems for at least 6-8 weeks, getting progressively worse each week. My worry is that they won't be able to fix it. They can't even get a billing system set up, how are they going to build a new server and migrate all the data to it?

I'm cancelling today. I doubt we ever see Chaos again

[Steelclaw]

Computer electronics do indeed give fair warnings when something is wrong. From system crashes to bios resets to general strange behaivor of your computer. Most times they start fortelling their problems LONG before they actually fail.

When you start to see system crashes and strange behavior, the electronics have already failed. It's not a warning, its broken already! :P

Of course, if you continue to leave the system in operation and ignore the failed component, ("Oh, just keep rebooting it, it'll be fine.") you deserve every bit of pain and trouble that will result.

[Kerech]

"Oh, just keep rebooting it, it'll be fine."

Surely nobody running a commercial MMORPG would ever say something like that... oh, wait...