Server outage on Friday 13th
06-13-2008, 02:17 PM
Post: #1
Server outage on Friday 13th
We had some server problems last night. After 2:00 am CET the server hanged and we could not reach it anymore. It must have been because today is Friday 13th! Angry

Normally we have serial console access and a powerbar to powercycle the machine. Serial console didn't give me access since the machine hanged completely and the powerbar failed in shutting off the power.

Today someone from the hosting party manually restarted the server at 13:50 so it's back online and we only had just less than 12 hours of downtime! Hurray!
Thanks to Niels for going through the trouble! Thumbs up!

1982 - TA60 Carina 4dr sedan - fun cruiser
1983 - AE86 Sprinter Trueno - import project
2013 - Honda Civic sport - daily driver
2004 - AEU86 dot ORG - daily domain

Support our forum, buy from the AEU86 shop:
[Image: AEU86 AE86 - Server outage on Friday 13th]
Visit this user's website Find all posts by this user
06-13-2008, 02:24 PM
Post: #2
Server outage on Friday 13th
good work Smile

i'd suggest check the log files and see whether the linux kernel was killing random processes, and whether apache was potentially hogging all resources causing extreme loads.

I remember having a load of 29+ on one of our servers, luckily sshd wasn't killed (yet), it just took about 15 minutes for the shell prompt to arrive, and then some blindtyping and hoping things would work out with a simple apachectl (apache2ctl?) graceful to hotwire apache again Big Grin

Grin.

Oh hey banpei, bit offtopic, but i noticed you were on IRCnet too? Big Grin
Visit this user's website Find all posts by this user
06-13-2008, 04:23 PM
Post: #3
Server outage on Friday 13th
Same error as before: one of the harddisks failing (disappears) and then the RAID failing due to this. However it is set up as a mirror, so it should just continue like nothing is wrong but in reality it hanged the system.

Howerver there is one difference now: during startup the second harddisk failed again and now it's running raid on only one disk. Better check that out this weekend...

1982 - TA60 Carina 4dr sedan - fun cruiser
1983 - AE86 Sprinter Trueno - import project
2013 - Honda Civic sport - daily driver
2004 - AEU86 dot ORG - daily domain

Support our forum, buy from the AEU86 shop:
[Image: AEU86 AE86 - Server outage on Friday 13th]
Visit this user's website Find all posts by this user
06-13-2008, 04:29 PM
Post: #4
Server outage on Friday 13th
ouch, at least you guys are running a raid Hurray!

good luck with fixing! i/o troubles suck even more
Visit this user's website Find all posts by this user


Possibly Related Threads...
Thread Author Replies Views Last Post
  Server crashed last sunday banpei 17 11,975 10-25-2012 11:15 AM
Last Post: banpei
  Slight problem with the server banpei 4 4,273 05-31-2009 09:34 AM
Last Post: nmwisima
  Server problems banpei 4 4,124 06-10-2007 09:27 PM
Last Post: Lipinhu
  Server problems Mux213 10 6,301 01-04-2007 11:43 PM
Last Post: banpei
  Probably some outage banpei 1 2,237 10-09-2006 10:31 AM
Last Post: banpei

Forum Jump:


User(s) browsing this thread: 1 Guest(s)

Contact Us | AEU86 | Return to Top | Return to Content | Lite (Archive) Mode | RSS Syndication