LifanForum.com
General Category => Chit Chat => Topic started by: almonpoole on May 31, 2008, 10:21:23 PM
-
Saturday afternoon there was a fire that has knocked out electricity to our hosting provider’s data center.
We hope to have this back up and running soon, perhaps by Sunday night or midday Monday..
Thanks,
Almon
Below is a message from our provider,
From Doug Erwin:
This evening at 4:55pm CDT in our H1 data center, electrical gear shorted, creating an explosion and fire that knocked down three walls surrounding our electrical equipment room. Thankfully, no one was injured. In addition, no customer servers were damaged or lost.
We have just been allowed into the building to physically inspect the damage. Early indications are that the short was in a high-volume wire conduit. We were not allowed to activate our backup generator plan based on instructions from the fire department.
This is a significant outage, impacting approximately 9,000 servers and 7,500 customers. All members of our support team are in, and all vendors who supply us with data center equipment are on site. Our initial assessment, although early, points to being able to have some service restored by mid-afternoon on Sunday. Rest assured we are working around the clock.
We are in the process of communicating with all affected customers. we are planning to post updates every hour via our forum and in our customer portal. Our interactive voice response system is updating customers as well.
There is no impact in any of our other five data centers.
I am sorry that this accident has occurred and I apologize for the impact.
-
Got your email too.
Better hurry, I'm having withdrawls. ;D
-
I have gone ahead and started migrating lifan-moto to another provider. I have heard that H1 will not begin a test power up until 5pm central time today and then they will begin testing stuff out, if you include domain propagation this could put us out until late Monday and I at least need domain services back up by tonight if possible so I am moving everything, nothing personal as this could not be avoided by H1 but we all need to move on.
As I understand it there are about 9,000 servers down because of this outage, some of my favorite sites are parts of this outage,
http://www.mobilitysite.com
http://www.pezcyclingnews.com
This is really hard on Pez as the Giro ended today in Italy and there site cannot reap the advertising rewards as they are on site and doing interviews I'm sure.
It sucks and I plan to have some sort of mirror in the future for Lifan-moto so when these things happen I can basically change some name server info and be back up within hours and not days.
I'll keep everyone updated here in this thread until we come back online.
Thanks,
Almon
-
Here is the thread from the planet reporting every few hours on the status of this outage,
http://forums.theplanet.com/index.php?showtopic=90185&st=0
Almon
-
Hey Almon,
Actually, in my opinion, this could have easily have been avoided by H1. Their fault was relying on "local" backup scenerios. It only costs slightly more to have a redundant set of servers in an alternate location to avoid these types of problems.
I owned an ISP for several years and I had multiple locations in Indiana as far apart as Sellersburg and Fort Wayne. All the systems were 100% redundant and in the 15 years I owned the site, I never once had a failure that knocked it out. In fact, I had a car run into the server room at one time and take it out completly and there wasn't so much as a single complaint from any of my customers.
I also personally use two different hosting sites myself for my mission critical sites and I control the DNS settings for each. I periodically swap my name servers to make sure both sites are working correctly, as well.
Just a thought.
Bill
-
Well this is not an isp issue. The planet is simply reselling bandwidth to isp's. But I agree anything this critical that has such an effect on so many sites should have redundancy backups in place. I would think a lot of people with active site like lifan-moto are pulling their hair out by now.
I'm hoping I can have us back up by tonight at some point.
I'm currently uploading the May, 28th backup to my other provider and this will take several more hours to finish and then I'll let the domain propagate over.
Thanks,
Almon
-
I have uploaded the May 28th backup to the new provider and I am awaiting two things,
1: I am installing a newer version of Yabb forum and then I will overwrite the templates, messages, members and other stuff. This will take several hours to upload again
2: I am waiting for the domain to propagate. The DNS has been changed and hopefully this network outage will not kick us from changing the name servers.
So I wait.
I am hoping to have the site up at least minus the forum in around four hours and then in another hour or two after than it is my hope to have the forum back up where we left off on May 28th.
I'll continue to keep everyone updated.
Thanks,
Almon
-
Why would someone post this info on another person's forum. Doesn't sound very professional or ethical to me. Just my opinion though.
-
The domain has propagated and Lifan-moto.com is back up and running minus the forum.
I am working on this now and I hope to have it resolved before midnight.
Thanks,
Almon
-
Why would someone post this info on another person's forum. Doesn't sound very professional or ethical to me. Just my opinion though.
Why would you make such a comment in a forum that has nothing to do with you other than to attack someone else. Stryder please ban this guy once and for all.
It has always been my intention for Lifanforum.com to be a sister site to Lifan-moto.com but this is why I have not moved in this direction because of these type of users who just go around kicking people who do not follow their so called 'values'. This user CC has always tried to force his values down other users throats.
Almon
-
Now that most of our services are back up I will continue to keep everyone updated on the Lifan-moto.com homepage.
Thanks,
Almon
-
Also a big thank you to Stryder of Lifanforum.com for allowing us to post updates here while we were down.
Almon
-
Arg, dns has gone down so I am waiting again. I will continue to make posts here on Lifanforum.com until things are back up for good.
Sorry and thanks,
Almon
-
"Lifan Commandments LOL", possibly?
;D
Oh well. :P
Could make posts on AL if you want..... LOL
Sorry, but I couldn't resist.
I responded on the AL site also.
Hope lifan-moto comes back up soon. I'm loosing my mind.
Not really.
Really, the wife is in there cooking so almost time to eat.
See y'all later.....
-
Hey Tejasmud,
Well I hope we will be back up soon. DNS took a dive so I am basically just waiting now and I hope those guys will have everything back up soon. The site should start working properly again once we get the dns up.
I'll send out another email by tomorrow with the latest and I'll also post here in this thread. I don't think this will go on much longer than midday Monday, at least that is my hope..... This seems to be a major outage in the United States.
Thanks,
Almon
-
I figure it will all work out, and sure it will.
Didn't read much into the links you posted, but hope no one was hurt in the ordeal.
Have a good one,
Eric
-
Looks like no one was hurt but the internal transformer shorted and blew up knocking three walls down in the building and started a big fire.
Why would anyone stick a freaking transformer inside a building? Aren't those things designed to fail with massive power fluctuations?
I guess I do not know enough about the situation, but I found out in 2003 a transformer also blew up in the same building but it did cause as much damage then.
I have heard 400,000 sites are down and the first floor is melted, Monday morning is going to be rough on us web folks......
Almon
-
Some good news!
It looks like power is being restored as we speak to the second floor of H1. I believe our server is located there based on the phase1 and phase2 time line.
This means DNS will be coming up if it is not already and we should have our rack back up by morning, they only have 6,000 servers to bring up so it will be time consuming.
Hopefully Lifan-moto will breath again before lunch tomorrow as well as many other sites :D
Almon
-
Man, what a nightmare. Hope everything comes together OK. Outages are a huge pain when you can't do much about it.
-
I was wondering what happened to the forum,
Hopefully get it going real soon, I like Lifan-Moto, haha... even someone like me who isn't very computer literate can use and learn!
-
Thanks guys, I'm still waiting on dns to come up, The Planet says that they have 90% completion in Phase 2 and 0% in Phase 1, I'm thinking we are in Phase 1, but I thought phase 2 last night.
But until then here is a temporary url until the domain comes up,
http://edtweb24dev250.edthosting.com/
Thanks,
Almon
-
Thank god, I have the forum up and running along with the site, but I am waiting for the Lifan-moto.com domain to propagate.
Notice that most of the links will not work until the domain comes back up.
Here is a forum url,
http://edtweb24dev250.edthosting.com/yabb/YaBB.pl
Thanks,
Almon
-
We are officially back up now at lifan-moto.com.
Thanks,
Almon
-
Almond,
WOW, that's great news for you I'm sure. I'm happy for you. Glad everything worked out well for you.
-
Country Boy,
Hey thanks good buddy!
Almon