Status on Problems With Chrysler (67.19.170.34)

We are having some problems with the server Chrysler (67.19.170.34) and are working on fixing this issue as quickly as possible. The problem involves an assertion failure in the Kernel and this should be fixed soon.

[chopper]

Can anyone give any idea how long these type problems take to resolve?

[GatorJustin]

I am being told that all CHrysler issues have now been resolved. If you are still having problems, please reply here with details.

-Justin

[GatorJustin]

Scratch that. Just had another failure so the techs will continue to work on this until it is resolved. Thank you for your continued patience.

-Justin

Hi, I just wanted to update everyone and let you know it looks like this is a hardware issue concerning memory that went bad this afternoon, we are having it replaced as we speak and should have another update shortly.

[karz10]

Since you created the thread...I first reported problems at 12:30 PM ET after waiting 15 minutes of continued downtime. I've been checking them fairly frequently ever since, and they did come up a few times while it was being reported that they were back online, but it was only for a minute or two at a time. Sometimes Web came up, but Mail was down, and vice versa. Even when it was said to be fixed there were at a minimum or 2-3 errors/failures in cPanel, with frequent excessive server load.

At the moment, even though I can synch with mail, after everything came back up at about 5:45 PM ET, I still see errors in cPanel, and I don't think I'm actually getting mail from the outside, to confirm this, I did some tests, and according to several sources, my mailserver cannot be reached.

Please post here when all is resolved, and if you have time contact me directly in case my mail is still not working, I had submitted a trouble ticket earlier, so you should have my alternate email on file too.

Regards,

Karsten

[chopper]

We all appreciate the updates i am sure, Any play by play you can give us?
Any time estimate please would be greatley greatley appreciated.

Thanks

[karz10]

Well, ever since I posted that, it went down again...so I give up, my whole day is shot...hopefully it will come up some time tonight and we won't have any further problems tomorrow/rest of month, or longer...

[chopper]

Originally Posted by GatorBen

we are having it replaced as we speak and should have another update shortly.

How is that other update coming along?
I for one would love to be in the know.
Mike

[GatorJustin]

I have been told that the problem has been traced to a hardware failure. All data is safe and is currently being transferred to a new machine.

-Justin

[karz10]

me too, i'm getting nervous now, 7 hrs and counting...

[chopper]

Originally Posted by GatorJustin

I have been told that the problem has been traced to a hardware failure. All data is safe and is currently being transferred to a new machine.

-Justin

Have you got an ETA on the situation.
Thanks

[chopper]

well it appears to be online again, Still Failures though but i was able to recieve a test email i sent to myself.


Here is some small constructive critisism:
We could have done with more updates, Particularly exactley what the problem seemed to be and an expected timeframe to have it fixed.

I am still a happy costumer but that may be something to work on.
Please keep us updated regularly, even if it is uneventful. please keep us in the loop so we can do the same for our customers
Mike

[GatorJustin]

Everything seems to be ok now. The problem was narrowed down between a faulty ram module, or motherboard, so the data was transferred to a new chassis and the ram was replaced.

-Justin

We discovered the server was reporting 256mb ram when it should have 2 gigs. We always confirm specs when server goes live so either the ram has gone bad, the mother board, or something else.

We've concluded it ended up being a faulty mother board. A chassis swap was done and the server is behaving perfectly, as well as reporting the correct amount of ram again.

The planet techs said...
"(rbrooks-09/13/04-17:58):
The chassis swap and memory swap appear to have corrected this. Sorry for the delay.
(rvera-09/13/04-19:21):
We do not know for sure, it was most likely a bad motherboard. "

This bad motherboard would explain the past freezes we had on this server which was most likely amplified by the extreme usage a reseller was using that we banned.


There shoudln't be any more problems from this server.

[chopper]

Originally Posted by GatorBen

We discovered the server was reporting 256mb ram when it should have 2 gigs. We always confirm specs when server goes live so either the ram has gone bad, the mother board, or something else.

We've concluded it ended up being a faulty mother board. A chassis swap was done and the server is behaving perfectly, as well as reporting the correct amount of ram again.

The planet techs said...
"(rbrooks-09/13/04-17:58):
The chassis swap and memory swap appear to have corrected this. Sorry for the delay.
(rvera-09/13/04-19:21):
We do not know for sure, it was most likely a bad motherboard. "

This bad motherboard would explain the past freezes we had on this server which was most likely amplified by the extreme usage a reseller was using that we banned.


There shoudln't be any more problems from this server.

Thanks for the update, Lets hope you are right.
Mike

[RyanM]

Does this thread apply to the Escape server?

[Stef]

Ryan,

as stated in the thread subject, this is about the Chrysler server, not Escape.

Stef.

[digitalb]

Question for my fellow resellers on chrysler:

I am new to hostgator. I was a reseller before with a certain undisclosed-acore company. The whole reason I left was because of problems like we had with chrysler yesterday. Of course this was on a regular basis at my previous provider. Since I have only minimal experience with hostgator (so far so good), I wanted to get some feed back from my fellow resellers as far as the reliability you have experienced.

I certainly understand the nature of this business would require an understanding that server down time is just to be expected on occasion. Of course the less downtime the better, and the shorter and during off-peak hours the better. Also realizing the very competitive nature of this business, we all understand the need to make decisions that give us the edge. Just incase you worked hard all weekend to be prepared for a Monday morning presentation to one of the biggest clients of the year who you went out of your way to sell on you wonderful, new and improved hosting services. Then just incase you had to try and explain "page cannot be found" for 7+ hours.

Any feedback would I'm sure help us all keep our competitive edge as re-sellers.
Thanks.

[GatorJustin]

digitalb, what happened yesterday was one of those things that cannot be foreseen. I hate to use the phrase "it wasnt our fault", but that is exactly the case.

There was a hardware failure, and we had all of our resources focused on correcting it as quickly as possible. If we hadn't put so much effort into this, you would be looking at maybe double the downtime.

Every host on the planet experiences occasional issues such as this, and no matter which provider you select, there is no guarantee that the same thing won't happen.

You rely on us to provide you with the best service, fair prices, and great support. We rely on the hardware that we purchase to provide us with error-free operation. Unfortunately, this time the hardware failed.

We do apologize for the inconvenience, but this was impossible to detect until after it had already failed.

-Justin

[digitalb]

Justin,

Thanks so much for the reply and for caring. That certainly makes a customer feel better. you guys won me over when you answered the phone AND emails/forums.

I realize you guys were on top of the situation, thanks for that.

I guess my previous provider has tainted my view, and made me slightly uneasy since I am new to hostgator.

I guess what I would like to know (without predicting the future), is what can be expected as far as downtime/what downtime has been like in the past, and if yesterdays situation was indeed.. extremely rare.

Secondly, is there a connection between the server you are placed on and the package you are paying for? In terms of the number of other resellers on a given box etc.? I would certainly be happy to upgrade my package if it meant having some sort of advantage/upgrade that would increase the reliability factor. I couldn't help but notice other server names like Ferrari and Porsche... while server Chrysler was having issues.. lol

[David]

digitalb,

We've been with HG since mid June in one form or another. First had a shared account, and then reseller. During that time I have only received one notice of downtime that was 30 min. or less (don't know exactly since siteuptime only goes to 30 min. increments for free customers). I believe that had to do with a PHP rebuild or reboot situation, but we have not seen any other issues since or during that period.

David

[GatorJustin]

Originally Posted by digitalb

I guess what I would like to know (without predicting the future), is what can be expected as far as downtime/what downtime has been like in the past, and if yesterdays situation was indeed.. extremely rare.

Unfortunately, this is one of those questions that simply cannot be answered with any degree of accuracy. I could tell you that you'll never experience such a lengthy downtime again and you may find your site down for 5 hours tomorrow. Conversely, I could tell you that you should expect regular outages, only to have your server perform flawlessly for 12 months straight. I wish I could provide a better answer, but it's just not possible. All I can say is that every server, whether it is ours or another hosts, will experience downtime. They have yet to build a server that did not require an occasional reboot.

On the bright side, virtually all downtime we experience is less than 10 minutes. Only on the rarest of occasions do we experience an issue like we had with Chrysler where we have a hardware failure.

Hopefully this will help to alleviate your fears a bit.

Secondly, is there a connection between the server you are placed on and the package you are paying for? In terms of the number of other resellers on a given box etc.? I would certainly be happy to upgrade my package if it meant having some sort of advantage/upgrade that would increase the reliability factor. I couldn't help but notice other server names like Ferrari and Porsche... while server Chrysler was having issues.. lol

Resellers are grouped together on the same machines, and shared customers are grouped on others. ANy server named after a vehcile (Jeep, Supra, Hummer, etc) are reseller servers and only house reseller accounts. Servers with names such as gator1, gator2, gator3, etc are the shared customer boxes.

We always prep and install a new server once it reaches about 75% capacaity. Capacity being the load average for the machine, and not the number of domains on it. We do not oversell on any of our servers.

Once we are at that 75% level, we begin to stagger the new accounts. Right now for example, Chrysler has just past the 75% mark, so we recently added Mustang to the network. We will add an account to Mustang, and the next to Chrysler, etc. so that if a probelm develops on the newest machine, we still have a place to put the sites on the other one, thus minimizing downtime.

Also, as great as we are we still have people cancel from time to time, and in those cases, we add new accounts to the empty spots that are available.

Does this help clear things up a bit?

-Justin

[digitalb]

Thanks for the comprehensive reply. That all sounds great.
I have to plead ignorance on some of these topics, but I have to ask. Then I'm no longer ignorant!

I understand you guys house your servers at the planets data center. I was reading here: http://www.theplanet.com/datacenter.html

There is a lot of talk about redundancy. It seems redundancy is in place for certain aspects, network, power etc. My question is: Do you combine your data storage with your web servers? Is there real-time redundancy of some sort for data on a given server in the case of a major failure?

Secondly, in case of one of these major failures like what happened yesterday with Chrysler, is there some sort of policy in place to keep the downtime significantly less than yesterdays downtime? Based on your "99.9% uptime guarantee", there would be about 8-9 hours of downtime a year. Yesterdays 7 or so hours took a sizeable bite out of that.

Thanks again.

[karz10]

Interesting, after letting things settle down about this, I emailed to inquire status of a few issues at one time...regarding this outage I wrote:

*****************
... 2 - I hate to even ask about this, but I hate the downtime even more, I have to ask about the several hours of downtime I experienced recently when Chrysler went down. I'm not sure what your *official* downtime count was, but I became aware of it at around 12:15 PM ET, and I know it wasn't formally fixed until after 8 PM ET that evening, so I'm assuming there is a credit due as a result of this. I don't care how you want to handle it, whether a refund is issued, or if you just don't charge me next month. Just so long as I don't have to manually do anything to repeat my request, I'm happy.

I want to give HG a chance, I have had a lot of issues with vendors in general the last 2 yrs getting my business off the ground, but you guys seem to respond to people, and although it concerned me that I was unlucky enough to be on Chrysler, and there was a Jeep issue recently, I know you have a lot of other servers, so I'm gonna hope there are no more unpredictable failures, and I never have to ask for a credit, I would be happy to pay every month to have minimal issues...
*****************

To which I got this response:
Chrysler had no such outages. Below is every outage chrysler has had since the 29th. (it can take 5 minutes for it to note a service backup)

If your site was down for what you say it was your isp.

What is your username with us?



Return

Fri 17, 2004 @ 06:14:46

PORT 443 - HTTP SECURE WEB SERVER

chrysler.websitewelcome.com

Failure

Fri 17, 2004 @ 05:59:59

PORT 80 - HTTP WEB SERVER

chrysler.websitewelcome.com

Failure

Fri 17, 2004 @ 05:58:17

PORT 443 - HTTP SECURE WEB SERVER

chrysler.websitewelcome.com

Return

Fri 17, 2004 @ 05:36:47

PORT 80 - HTTP WEB SERVER

chrysler.websitewelcome.com

Return

Fri 17, 2004 @ 05:36:27

PORT 443 - HTTP SECURE WEB SERVER

chrysler.websitewelcome.com

Failure

Fri 17, 2004 @ 05:30:16

PORT 80 - HTTP WEB SERVER

chrysler.websitewelcome.com

Failure

Fri 17, 2004 @ 05:30:11

PORT 443 - HTTP SECURE WEB SERVER

chrysler.websitewelcome.com

Return

Mon 06, 2004 @ 17:16:39

PORT 80 - HTTP WEB SERVER

chrysler.websitewelcome.com

Failure

Mon 06, 2004 @ 17:14:57

PORT 25 - SMTP SERVER

chrysler.websitewelcome.com

Return

Mon 06, 2004 @ 17:13:14

PORT 143 - IMAP MAIL SERVER

chrysler.websitewelcome.com

Failure

Mon 06, 2004 @ 17:02:44

PORT 80 - HTTP WEB SERVER

chrysler.websitewelcome.com

Return

Sun 29, 2004 @ 00:16:56

PORT 443 - HTTP SECURE WEB SERVER

chrysler.websitewelcome.com

Failure

Sun 29, 2004 @ 00:11:51

PORT 80 - HTTP WEB SERVER

chrysler.websitewelcome.com

Failure

Sun 29, 2004 @ 00:11:46

PORT 443 - HTTP SECURE WEB SERVER

chrysler.websitewelcome.com

Thank you for your business!
Hostgator.com
********************

Hmmmm... I'll respond with this and see what happens...

Please read the following thread http://forums.hostgator.com/showthread.php?t=1272, where I made some comments about the outages, but HG finally makes an announcement starting 3 hours after the outage was made known to me, and I had already places several phone calls, as well as issued email trouble ticket back-and-forth, including this ticket number:

[HGSupport #IJX-77907-615]: Logging site problems


This happened on 9/13, which I don't see that date listed at all on your below response, I suggest you check with some other staff members before you claim their has been no significant outage in the last month, and start blaming it on the user, or the user's ISP, because as you can clearly see by the thread, there was a VERY significant outage.

Please advise ASAP, and may I ask what your name is for future reference?

Regards,
******************

I'll let you know...

Karsten