MUSTANG MySQL data loss

[VPC]

To those on MUSTANG....check your MySQL DB's if you do daily transactions on it. Between Thursday night and Friday morning (I would say around 5 a.m. EST) HG claims they had to swap a drive due to corruption to which all of my clients MySQL DB's were missing nearly 30 days of transaction data. How does that make sense? It doesn't, unless the "good" drive(s) had old data.

Not sure what the real story is, but I am pretty sure 98% of the HG techs do not as well. I haven't seen any missing file/folder data, just missing MySQL data across all of my clients using different apps which is even more crazy.

On top of that, I notified them Friday morning of this....have gotten replies that state I should not be missing data (how stupid because I am) and no resolution yet from HG.

Does HG think people can just blow off a work day? What are my clients supposed to do when they are missing a months worth of work? The clients use the web based app all day everyday to enter data...their business functions off of it. Should they work all day...then HG decides to restore DB's which overwrites their work day? Does HG actually have working backups of the MySQL DB's? When is the last one?

Well, I am sure some of you are saying "well you should have backups" - ok, I will let you in on the secret. I do. I have MySQL backups daily and they are full backups running at night. I had all of my clients restored in about 1 hour by re-importing their SQL files from Thursday night.

My real point is HG lack of response and responsibility. We are approaching 24 hrs with no answers, no resolution, no admission, no anything. How do you run such a soup sandwich operation HG? A free for all server farm of no responsibility? What will you do to prevent this in the future? The answer is nothing since you cannot tell me why it happened. Will you now run a DB restore and wipe out my restore and any work they have done since the loss of data? WHo knows, you don't communicate!!!

The reason this is an issue is once you have lost data...it is imperative you make a decision quickly to either restore and move on fresh or not restore. If you choose to wait, as a hosting co, you will either overwrite more responsible business owners restores (like me) who get my clients up and running within an hour of your foul-up, or you, as a hosting co, attack the problem, find the root of the issue, communicate with your customers, and get the data restored or have a plan to restore so everyone is on the same page.

I basically told my clients that even though I restored their data, the hosting co may, at some point, restore data overwriting my restore and screwing it up again. My clients ask me....well would they communicate that with you first to coordinate an effort like that? I laughed out loud....smacked them across the face, and said "this is HG...they do not notify the pearly gates if they are coming."

Anyways, here is to techs that take care of their clients http://forums.hostgator.com/images/icons/icon7.gif...and here is to hosting companies that don't give a damn http://forums.hostgator.com/images/icons/icon13.gif about their clients.

[ufopsi.com]

Do you make backups?

[GatorDaveC]

Hello,
I believe that it would be almost impossible that the information in your database was 30 days old. The initial rsync from one drive to another ran two days prior to the drive swap. Hours before we did the drive swap, we re-rsynced the data to ensure all of the data was current, in fact I still have the while loop script on the server which constantly updated the MySQL databases every 60 seconds.


I do agree that the customer communication level should be improved, especially on tasks that would require downtime on the server, or any types of upgrades which we do, just to give a heads up for the clients.


Having said that, what kind of customer communication would you find most effective? A Forum post? A link on the Hostgator site to server issues / updates? An E-Mail list for major updates only (Using .contactemail in cpanel)?


If you would like, I think we still have the old hard drive from the Mustang server available, so its possible that I could fetch your database from there, or from our weekly backups which run every Sunday.


Sorry for any inconvenience caused.

[ufopsi.com]

How about an RSS feed? Fasthosts did so and it was useful.

[Serra]

Originally Posted by GatorDaveC

I do agree that the customer communication level should be improved, especially on tasks that would require downtime on the server, or any types of upgrades which we do, just to give a heads up for the clients.

Anything that was listed on the prior list of ways that HG was going to improve communication with customers would be fine. No need to make a new list, just do one of the ones from the OLD LIST!!!

[Sam]

I find it strange why VPC didnt have a backup of his database anyway and his customers accounts. (For anybody who dosnt backup there stuff regurarly i recommend you do back it up.. you never know what will hapen next).

[Serra]

Originally Posted by _Sam_

I find it strange why VPC didnt have a backup of his database anyway and his customers accounts. (For anybody who dosnt backup there stuff regurarly i recommend you do back it up.. you never know what will hapen next).

You really have to read what was said. VPC did backup the database. He also restored it back once the problem was discovered. In fact, there really was no problem with the database.

VPC's complaint was that he wasn't notified. (That gets lost in all his rambling)

[Sam]

Ahh, my bad

Since this really isn't a network status issue, im going to move it over to suggestions. Feel free to continue discussion.

[VPC]

whewww....thank goodness for detailed readers. Yes, of course I had backups. Does that negate the responsibility of the hosting company when it comes to data retention and the importance of immediate decision making for restoration of data or not?

My rambling is not just about getting notified, it is about the "I find it almost impossible that you lost data" statements that are argumentative in nature. I do not need to argue about it, I have proof and I need HG to figure out what happened.

What can I provide to help you find out what happened?

GatorDaveC, I have DB backups of Wed night (good backup), Thurs night (good backup), Friday morning (missing data) and Friday night (good backup).

03/21 DB size = 33013
03/22 DB size = 33047
03/23 DB size = 32836 - missing data
03/23 DB size = 33048

There is no ability for client to delete data, DB always grows everyday for past 2 years as I have all 700+ SQL files to compare against. Also, clients notified me of missing data and how far back it was missing. I am pretty confident that multiple clients can tell how far they are missing data in an app they use everyday for the past few years. I personally confirmed data that had been written on March 02nd that was not in the DB's after the drive swap. That is all I needed.

I cannot tell you why this happened, that is your job. I can tell you that what I am telling you is accurate at least back to the beginning of March.
Who knows if more than that, I chose to stop looking at that point.

Using .contactemail in cpanel is the best way in my opinion. People can argue that not all will get it for one reason or another. It is simple....make it policy that the email listed there MUST be valid in order to receive notifications and throw up a big disclaimer that HG cannot be responsible for undeliverable notifications. I am not asking you to guarantee my brain receives notification...I am asking just for the attempt from HG and I will worry about how it gets to me.

One other thing GatorDaveC, please do NOT restore anything for my accounts. I would prefer that MY restores stay valid. Thanks!

[VPC]

Originally Posted by GatorJay

Since this really isn't a network status issue, im going to move it over to suggestions. Feel free to continue discussion.

GatorJay, so if I remove my suggestions...does that make it an open issue? Would other MUSTANG clients come to the suggestions forum to read about how they should check their MySQL DB's for lost data? I don't think so.

[Serra]

Why move you? No one is going to take you seriously with your attitude. If you want to have a serious discussion, thats fine, if you want to rant, that is fine too, but if you are ranting, don't expect a serious discussion to spring from that rant.

One other thing GatorDaveC, please do NOT restore anything for my accounts. I would prefer that MY restores stay valid. Thanks!

That isn't your choice to make. You are on shared hosting, you will need to live with the rules for shared hosting. If you want to run your own system, then I suggest you move to a system that allows that type of management.

My rambling is not just about getting notified, it is about the "I find it almost impossible that you lost data" statements that are argumentative in nature. I do not need to argue about it, I have proof and I need HG to figure out what happened.

What can I provide to help you find out what happened?

What do you what and why do you want it? The hard drive had a problem, they restored it. The restore wasn't the best data available. You fixed the problem by restoring a better backup than they had of your date. What exactly is the issue? They did the best they could, it wasn't good enough for you, you fixed it. End of story. Get over it and stop demanding answers that they don't have.

[Serra]

Originally Posted by VPC

GatorJay, so if I remove my suggestions...does that make it an open issue? Would other MUSTANG clients come to the suggestions forum to read about how they should check their MySQL DB's for lost data? I don't think so.

No one is going to read this rant and come to the conclusion that they need to check their data. Heck, MOST of the people who read your rand were unable to determine exactly what you were ranting about!

[VPC]

To be more specific, no one can take it seriously because my "ranting" involves all pertinent questions that you nor HG can answer. Therefore, it is called ranting in your eyes because any intelligent answer would take too much investigation and time.

It isn't my choice to make for a restore of data? They did not do a restore, they swapped a drive due to corruption. Today I was emailed and even in this post was asked which DB's specifically I wanted restored if possible. That seems like a choice to me in a shared environment. Get your facts straight before you open your holster.

Yeah, your right, I should not demand answers they do not have on how they lost 30 days of MySQL data. That is just crazy. - just because I fixed it doen't mean they should not come up with a "why it happened and how to prevent it in the future" if it was a mistake of some sort.

rand? rant? who cares what it is, the subject would prompt me to go look. for those who could not "determine" what was actually wrong in that post might have more problems in life than the simple ability to read and comprehend. Get some common sense.

[Serra]

I'm not trying to pull your chain here, I just think you could do better if you started a thread to rant and a thread to discuss.

I'm sorry you didn't take my advice and continued to rant. I feel for you, but there is little I can do to help your cause as you are doomed by your ranting.

[kmaw]

I like rants... it makes things for the rest of us to laugh at....

[VPC]

It is all rhetorical. I never expected a real answer as I have already stated but you must be stuck on a word you overuse.