It all started well, then...... [resolved]

[DigitalAviator]

I have a long story to tell and it may be a while in sifting through various emails from tech support and sales to get all of my experiences with hostgator of late out into the open, so please bear with me until I get all the information out into the open before any responses as there's a lot of serious problems that have developed...

First of all I want to say that for the first year of my relationship with HostGator it was the best customer service and support I've ever experienced...

You see I originally signed up for the Swamp account which offered 900GB of disk space and 9TB of transfer on the account per month... More than enough to cover my needs for a small start up business that provides scenery enhancements for use with Microsoft Flight Simulator... I also set up a separate baby account in preparation for a geospatial data store that I would set up at a later time...

My number of visitors per day to my site is an average of between 30-50 customers per day and sever load at my site is on average 0.5% CPU load and 0.2% RAM load. The most bandwidth ever used in a month was about 2.5TB with inodes only being between 15,000-20,000.

For the first year everything was going swimmingly until one day while trying to set up a second store for launch I was transferring in files from another server (public domain geospatial data), and suddenly I hit a 300GB cap on my account...

I enquired via tech chat about the limit and that my account and cpanel was suspended and they unfroze it... Subsequently to that they forwarded my account to 'security' for further review...

Now at this point I was pulling in data for sale at the new store and had only just hit 1/3 of what I had paid for in terms of my original hosting account which later got upgraded to unlimited disk space with unlimited bandwidth.

That's when the wheels started to fall off the relationship with HostGator...

Even though I made very clear to the technician that I was setting up my new store and was bringing data in the response I received was accusatory and rude.

I do apologize, but we will not be able to host this account on a shared server in its current form. The amount of disk space being utilized for the /home/fsdreams/public_html/developer to store raw image files is not something we can allow. These files exceed what apache can serve in size, and are not being utilized in the hosting of your web page. This type of activity will require a dedicated server and is forbidden in our Terms of Service under under using a Shared account as a backups/storage device.

From our Terms of Service viewable at: http://www.hostgator.com/tos.shtml

Using a shared account as a backup/storage device is not permitted, with the exception of one cPanel backup of the same account. Please do not take backups of your backups.

My response immediately was:

I'm in the process of setting up my geospatial data store at www.sitename.com and the files are in process of being bumped over to the store and they will be split into less than 2GB chunks, I only started work on getting the store set up yesterday and the files will be referenced once I get them all organized which I am doing at the moment, I've been talking with Hostgator about this for several days now and they have said it is fine and i won't be breaking the TOS, I'm not using it for data storage it is to be used for my geospatial data store...

I'm supposed to have unlimited storage on the account, why is it only 300GB? It was supposed to be 900GB from memory when I set up the account...

Please help... I am looking at moving to a dedicated server within a month or two but I don't have the revenue stream as yet to support it...

Also the developer folder is being used by my partner business at (name witheld). but I'll be switching the ftp access over to the store/download folder for that, and my beta team has been using the betateam folder for testing out our products... I can't put that folder into the live store...

I'm moving all of the <2GB files into the correct store directory and am hooking them into the storefront as quickly as possible, I am in NO WAY using my account as a backup/storage service and I am quite upset at being accused of that, especially when I've been repeatedly in contact with Hostgator to ensure that I'm doing everything correctly...

Note in a subsequent email i made clear that the beta testing being done was with regards to testing the uploaded files for the store, as I was in the Philippines at the time and internet access from a 3rd world country where I've been doing humanitarian work in the 3rd world (which my store supports via sales) often leads to broken and corrupted uploads...

The responses back continued with the accusations of data backup and server mirroring:

I did attempt to call you prior to my email. I was unable to reach you at the 808 number we have on file. The images you are mirroring from (site witheld) would take 866G to fully download. Are these files processed on our servers or is this done via your PC at home?

My response was that I was not mirroring a site, I was using wget to pull in the data for the store for direct sale on the geospatial store site, i wasn't mirroring a site and it was in no way anywhere near 866GB, it was only about 30GB of data... I also noted that it was the first time I'd used the wget command and didn't know how to format it correctly to dump the whole pathway information from the server I was grabbing the files from, I basically explained that I was a noob when it came to using shell access...

The conversation continues with my response...

I'm actually operating out of the Philippines at the moment while I get my business set up... To get all that data into the store would require downloading it here to the Philippines on a 384kbps line and then re-upoloading it to the store at hostgator... It would take years at that speed to get the data into the store...

The (site name withheld) data is something I can wait to do until I can afford to make the switch to a dedicated server, but the fact is I'm supposed to have at least 900GB of file space according to when I first signed up to my account, that's since been upgraded to unlimited...

In terms of splitting files, that's something I'd process off site, but I thought Apache 2.2 was supposed to be able to handle up to 4GB files?

As I said I'm really try to do everything right here and I don't want to violate any terms of service and I'm trying to be very careful not to and have been asking and checking every step along the way to see if what I'm doing is ok in setting up the store...

I'm certainly not my account for backup/storage, and I have been talking with Hostgator and trying to make plans to upgrade to a dedicated server as soon as possible...

I'm a little surprised though that I'm trying to use the account as per:
Our Most Popular Hosting Plan

UNLIMITED Disk Space
UNLIMITED Bandwidth
Host UNLIMITED Websites
FREE Instant Setup
99.9% Uptime Guarantee
45 Day Money Back Guarantee
CGI, PHP 5, RoR, Perl
MySQL, SSH, SSL, IMAP, POP
FREE Site Builder
cPanel Hosting ( View Demo )
24/7/365 Upgraded Support
No Contracts or Hidden Fees
View Web Hosting Plans

and I feel like I'm getting grief over trying to use the account as it was advertised to be... The geospatial data store at www.sitename.com was just moved over from a baby account and rolled into my business account a few days ago and I'm still trying to get it all set up properly, there really won't be a whole lot of traffic at that site and I'll be on a dedicated server way before traffic or bandwidth becomes an issue...

David I'm really trying to do everything right here, I really would appreciate your guidance on getting everything squared away...

to be continued...

[DigitalAviator]

continued...

Also David I wasn't mirroring the images from sitename.edu, I only have about 1GB of data left to obtain from that site, I was only grabbing specific files and certainly wasn't mirroring the site, I was just grabbing the files needed for the store. It certainly wasn't going to be 866G, I have the california data here at my office and that only barely uses 150GB...

I'm sorry I really only am just learning about shell access and I didn't know how to direct the files properly and somehow ended up with the full path from the originating site I was grabbing the files from and they were only transferring in at 1MB/s so it should have negligible impact upon the shared hosting and I verified that with tech support yesterday...

Again I'm trying to do this right and I'm certainly not pulling in 866GB of data from *any* site...

I'm still lost though with the 300GB limit that my account seems to have with disk space when it should not be that limited...

I did a quick calculation and the Arizona NAIP data should not total more than about 80GB total, somewhere more like the 60GB range though...

This is where the immediate push from being on a shared host to move me onto a dedicated server started...

I appreciate your willingness to work to work within the limits of our TOS and understand that for your needs a dedicated server would be best. I do know that sales sometimes are able to find a cost effective method in situations like this. I also noticed you did remove some of the extra images/directories and thank you for that.

I do sympathize over trying to retrieve these images via 384k line. Please do try and only use wget for the images/directories you are working with actively.

My reply...

I know I'm bumping my ticket down in the queue but I also wanted to mention that the /developer/betatester folder is only a temporary folder, I upload my scenery for the store and because of the bad internet connection in the Philippines sometimes the files get corrupted during upload, the files are usually only in that folder for about 24 hours or less before they either get deleted or bumped over to the /store/download/folder/folder...

Another reply from me...

Roger that on the wget, i'm only new to the shell process but I'll make sure to learn the proper commands to put everything in the right places. I'll clean up the Arizona stuff as soon as I get back from town, I'm actually doing humanitarian work in the Philippines and my business goes to support that...
I do want to move to a dedicated server as soon as possible, I just need to get the other store live and working... I'll go back to having a dedicated account for the geostore as soon as I get the chance to set it all up again...

I'm still curious though about the 300GB limit on the account, that's still an issue for me, as when I established the account it was 900GB of disk space, so is there a remedy for that per chance?

Thanks for your working with me David, Hostgator is an awesome company and the customer service is excellent, is the 300GB limit due to having established the account as a Hatchling account at the start?

Again the technicians start going back to accusing me of data backup and storage:

The issue is not exactly one of too much diskspace used, rather it is one of the massive amount of files not actively used as part of an active website. Per our ToS: Using a shared account as a backup/storage device is not permitted, with the exception of one cPanel backup of the same account. Please do not take backups of your backups.

Data stored on the account should be limited to only what is currently needed to run the sites served. Large amounts of files will degrade a servers performance, and since server are shared among a number of users, we must take care to provide optimal performance for everyone.

This is where I start getting frustrated and where things start going downhill...

Michael, as i've been saying the files are part of my store, and were being placed on the server ready to be used in the stores as part of the active website, they were only in a temporary directory, the issue I have is that my storage space is only limited to 300GB which is the issue...

I am NOT storing any data that is not part of an active website, I only started pulling the files in yesterday and am working on getting my geospatial store online!!!

I've never had any problems with Hostgator before but being accused of using it for data backup/storage is FALSE and I am about to file a formal complaint with management if the responses from the security team continue in this vain...

How the heck is a customer supposed to set up an active website if you don't allow them to put the files onto the server in the first place???

Please explain why I don't have access to the 900GB that I was supposed to have when I signed up to my account and which I signed up to Hostgator specifically because I was setting up both a Flight Simulator Scenery Store as well as a Geospatial Data Store...

The files are active and were only in a temporary directory until I could get them into the store's download directory which I have mostly acomplished today...

Please fix the issue of me not having access to what I have paid for in terms of getting my store set up!!! I'm starting to get very upset with the way this issue is being handled and I'm tired of being accused of data backup/storage when I AM DOING NOTHING OF THE SORT!!! It's all data that is being placed into the store at sitename.com and was only pulled into the server yesterday!

As you will see in continuing dialogue I will post below the situation gets worse and worse. At one point I end up talking on the phone with one of the security team members who made out and out false comments and made claims that completely contradict statements made by Brent on his blog and in interviews...

to be continued.....

[DigitalAviator]

The reply trying to push me onto a dedicated server...

As I discussed with you on the phone, we noticed your account when it started to use a big chunk of the specific server partition you are on. This is happening because you are selling a pretty rare type of digital data that is quite large. At that point our security team needed to act to protect the server stability for the other users on that partition.

Our CEO Brent has discussed the realities of our hosting plans before on his blog. Let me give you two links that discuss this further:

http://blog.hostgator.com/2008/10/20...n-eat-hosting/ - which announced the unlimited plans we now advertise.

http://blog.hostgator.com/2007/10/06/selling-out/ - which announced the major upgrades we were offering when you signed up for your plan.

Sidebar: My conversation on the phone with NateB was one in which he claimed that users don't go into a buffet and stay there for weeks and eat, yet in his next email to me he gave me the direct link to Brent's blog that clearly states:

It’s like going into an all you can eat buffet and being told you can eat more.

<snip>

It’s an easy concept really. Every web host has a terms of service with CPU and memory limits. If your website consumed too much of its share of CPU or memory then most web hosts will require you to upgrade. When you purchase a dedicated server you can’t get shut down for CPU or memory abuse so they have to sell you a plan based on what your site could use with less restrictions. At HostGator, we pretty much get an unlimited amount of bandwidth from our provider thanks to having thousands of servers. We also have 10,000’s of gigs disk space not being used. Giving away unlimited disk space and bandwidth doesn’t really cost us anything. It’s the CPU and memory that has a real value. A server will usually become extremely overloaded way before you ever get close to using the amount of disk quota and bandwidth it comes with.

NateB's email continues with the push to a dedicated server...

For a bit of further context, I wanted you to see the pricing offered by companies that specialize in hosting very large data-sets (1TB and up) for download. Amazon s3 is a leader int the field, using the massive datacenters developed to host one of the largest sites on the internet to sell extra diskspace:

http://calculator.s3.amazonaws.com/calc5.html

You can use their price calculator to estimate what it would cost to use a service like that.

We have taken some steps to clear space on the partition and it appears you have as well. As we continue to dialog about this situation I would ask that you make a concerted effort to not use any unneeded space so we can work towards. I looked at your site and it appears that the major disk space using files are all compressed heavily already. I tried using bzip2 which has one of the most aggressive compression engines around and it was only able to save 2-3 percent.

I am going to leave this ticket open so that our senior security admins can review it and see if they can offer some suggestions. However let me propose a few possible ways to continue to operate your store without growing the disk space as much as you suggest you made need:

1) Bring your site live with a sub-set of all of the globe's data. Then as your sales start to come in use the funds to cover moving to a dedicated server with an increased selection.

2) Begin by offering to mail customers DVDs or USB Key cards with the specific data they order. Just like a retailer selling tshirts post pictures and previews online and then ship the product once it is ordered. This could help you build the clientele required to move to a dedicated server.

3) Use a dedicated file hosting location to help host your accounts or at least look at splitting your two large disk usage sites into different accounts on different server partitions with Hostgator.

4) We can look at getting a specialized dedicated server setup for you to use, one with a lower cpu and large harddrives in a RAID array that will give you some data protection and at the same time allow your sites to grow at a rapid pace.

Just some suggestions to help the dialog move forward.

Now understand I had paid for 900GB of server space and 9TB of bandwidth per month (the other choices were 300GB/3TB, 600GB/6TB from memory)...

What had transpired over the phone was that I was told by Nate that I couldn't use the server in terms of what I had paid for and that he asserted my site as not a normal site... I repeatedly asked him, how can a normal site ever get near the limits on the account as advertised, he never gave a direct answer but twisted Brent's statements around and claimed that 'all you can eat wasn't all you can eat.'

My response:

I'll reply in-line...

Dean,

As I discussed with you on the phone, we noticed your account when it started to use a big chunk of the specific server partition you are on. This is happening because you are selling a pretty rare type of digital data that is quite large. At that point our security team needed to act to protect the server stability for the other users on that partition.

Shouldn't my partion be 900GB as per the original contract? It seems it is now but maybe there was a technical reason a partition was only set at 300GB?

Our CEO Brent has discussed the realities of our hosting plans before on his blog. Let me give you two links that discuss this further:

http://blog.hostgator.com/2008/10/20...n-eat-hosting/ - which announced the unlimited plans we now advertise.

http://blog.hostgator.com/2007/10/06/selling-out/ - which announced the major upgrades we were offering when you signed up for your plan.

Contrary to what you mentioned on the phone about being at an all you can eat buffet and staying there for 3 weeks the CEO Brent said "It’s like going into an all you can eat buffet and being told you can eat more." when it comes to the unlimited plan... Further to this he also said

"What exactly is unlimited? Well, I just got done going to the top 5 largest web hosting companies I could think of and the lowest plan I could find was for a few 100 gigs of disk space with thousands of gigs of bandwidth for less then $10 a month. These are all scam plans because like most hosts we have something called a Terms of Service that mentions CPU and memory limits. It is close to impossible to use what we are all selling you without being suspended for going over CPU and memory."

The terms of service mentioned above talk about CPU and memory, my store has been using 0.3-0.7% on both CPU and memory use, even at the highest level of operation of the site... In terms of the TOS when it comes to the issue I have been having is that I'm being accused of breaking TOS #2 Content...

I've been accused of using the account as a backup/storage device and that's not the case... I was transferring in files to get my source files to the server for inclusion into my store and as I'm new to using shell access to snag the specific public domain files from another server it grabbed the whole path name off the originating computer (being government public domain servers), I was not mirroring the sites but because I didn't know how to get the wget command to dump the path data and how to properly place the files in the store download directory (which I was on the first day of getting set up as I transferred my sitename.com domain over from my old baby account to the business account), I had them in a temporary directory until I could organize the files properly for the correct directories in the download store... It seems that the security people have ignored the fact that it was only a temporary directory and was being used to help set up the store and the files were there in process of being brought into my hosting account for use in my store as a value added product.

Using a shared account as a backup/storage device is not permitted, with the exception of one cPanel backup of the same account. Please do not take backups of your backups.
Examples of unacceptable material on all Shared and Reseller servers include:

• Topsites
• IRC Scripts/Bots
• Proxy Scripts/Anonymizers
• Pirated Software/Warez
• Image Hosting Scripts (similar to Photobucket or Tinypic)
• AutoSurf/PTC/PTS/PPC sites
• IP Scanners
• Bruteforce Programs/Scripts/Applications
• Mail Bombers/Spam Scripts
• Banner-Ad services (commercial banner ad rotation)
• File Dump/Mirror Scripts (similar to rapidshare)
• Commercial Audio Streaming (more than one or two streams)
• Escrow/Bank Debentures
• High-Yield Interest Programs (HYIP) or Related Sites
• Investment Sites (FOREX, E-Gold Exchange, Second Life/Linden Exchange, Ponzi, MLM/Pyramid Scheme)
• Sale of any controlled substance without prior proof of appropriate permit(s)
• Prime Banks Programs
• Lottery Sites
• MUDs/RPGs/PPBGs
• Hateful/Racist/Harassment oriented sites
• Hacker focused sites/archives/programs
• Sites promoting illegal activities
• Forums and/or websites that distribute or link to warez/pirated/illegal content
• Bank Debentures/Bank Debenture Trading Programs
• Fraudulent Sites (Including, but not limited to sites listed at aa419.org & escrow-fraud.com)

Again in no way have I been doing any of the above, I was simply transferring data into the digitalearthdata.com store and placing them in a temporary directory and then moving them into the store download folder for osCommerce from there.

"Of course many of you reading this will be shocked and rush off to one of our competitors to ask them if you can really use all the disk space and bandwidth without being suspended, and I’m positive they’ll tell you yes. In fact, my employees would have told you the same thing if they didn’t read this blog post."

Just ask yourself one thing…
How do all us shared web hosting companies sell more disk space and bandwidth for ten bucks then the dedicated server providers sell for hundreds?
It’s an easy concept really. Every web host has a terms of service with CPU and memory limits. If your website consumed too much of its share of CPU or memory then most web hosts will require you to upgrade. When you purchase a dedicated server you can’t get shut down for CPU or memory abuse so they have to sell you a plan based on what your site could use with less restrictions. At HostGator, we pretty much get an unlimited amount of bandwidth from our provider thanks to having thousands of servers. We also have 10,000’s of gigs disk space not being used. Giving away unlimited disk space and bandwidth doesn’t really cost us anything. It’s the CPU and memory that has a real value. A server will usually become extremely overloaded way before you ever get close to using the amount of disk quota and bandwidth it comes with.

Here Brent is clearly stating that you can really use all the disk space and bandwidth without being suspended or it causing a problem, the issue is with CPU and memory of which my two sites use less than 1% of the server in terms of processes... The CEO is backing up my very statements that I'm being treated wrongly in this situation and I would love to talk with Brent about this situation as I'm being told completely different from the very thing the CEO himself has been saying...

For a bit of further context, I wanted you to see the pricing offered by companies that specialize in hosting very large data-sets (1TB and up) for download. Amazon s3 is a leader int the field, using the massive datacenters developed to host one of the largest sites on the internet to sell extra diskspace:

http://calculator.s3.amazonaws.com/calc5.html

You can use their price calculator to estimate what it would cost to use a service like that.

Thanks for that link, I've been looking at moving to a cloud farm eventually anyway and to be honest I'm a bit tempted to give them my business instead of Hostgator due to the way my situation has been handled and continually being accused of using my account for back up/storage which is not the case...

We have taken some steps to clear space on the partition and it appears you have as well. As we continue to dialog about this situation I would ask that you make a concerted effort to not use any unneeded space so we can work towards. I looked at your site and it appears that the major disk space using files are all compressed heavily already. I tried using bzip2 which has one of the most aggressive compression engines around and it was only able to save 2-3 percent.

Again please refer to the CEO's statements about offering unlimited bandwidth and space on shared accounts... According to the CEO's own words I am not in violation of any TOS when it comes to operating a geospatial datastore....

I am going to leave this ticket open so that our senior security admins can review it and see if they can offer some suggestions. However let me propose a few possible ways to continue to operate your store without growing the disk space as much as you suggest you made need:

1) Bring your site live with a sub-set of all of the globe's data. Then as your sales start to come in use the funds to cover moving to a dedicated server with an increased selection.

2) Begin by offering to mail customers DVDs or USB Key cards with the specific data they order. Just like a retailer selling tshirts post pictures and previews online and then ship the product once it is ordered. This could help you build the clientele required to move to a dedicated server.

3) Use a dedicated file hosting location to help host your accounts or at least look at splitting your two large disk usage sites into different accounts on different server partitions with Hostgator.

4) We can look at getting a specialized dedicated server setup for you to use, one with a lower cpu and large harddrives in a RAID array that will give you some data protection and at the same time allow your sites to grow at a rapid pace.

Just some suggestions to help the dialog move forward.

In response to #1, that is a possiblilty but according to the CEO's statements disk space and bandwidth is not something I should have to worry about...

In response to #2, I'm looking at DVD distribution but as mentioned before I am in the Philippines doing voluntary humanitarian work, my store income supports helping out projects in this nation that 40% of the nations population live below poverty, DVD-R's are impossible to get in spindles here and dupe services do not exist, the mail service is also 3rd world which means shipping things to customers from here is like gambling...

In response to #3, Not an option for data security and anti piracy reasons...

In response to #4, This is what I am looking at within the next couple of months and I'd like to get a quote on that...

Nate, the issue that seems to be in question here is that security has assumed that A) I was mirroring a site and B) that I was using the account for data backup and storage, this is what appears to be at the crux of the issue... I was not mirroring the site but getting the individual files for inclusion in the store but required me to pull them to a directory that was not 'live' until I could standardize the file names before I dropped them into the store, and also I was not in any way shape or form using the account for data backup/storage, and the security team should have seen that the files were there for less than 24 hours....

I totally understand if the partition was blocked due to assuming that a mirring was taking place, and yet in every communication I've had with the security team they keep saying it's because I was using my account for "back up/storage" in violation of the TOS #2, but I have not being doing that, I was simply transferring files into a temporary directory while I get the new store set up. The issue is not about bandwidth or size of the downloads in my store, and that is backed up by the statements so clearly made by Brent, the CEO of Hostgator...

Please, I have been wronged by the security team by their claims that I've been violating the TOS, I have in no way shape or form violated it and just a heads up, I work professionally in film and television production and worked for CBS, CNN and ABC in (withheld), some of my friends are primetime news anchors and one is now at MSNBC as an anchor, if this situation continues with being falsely accused of violating the TOS I do have the option of going to my friends in the media and I'm sure they would be very interested in running a story about companies that misrepresent their services. Also due to my time in the News business I do have the ability to get in contact with high ranking government officials including Senate (name withheld) (D-Nevada), as well as other senators and congressmen in Washington... I'd really prefer not to go down that path but would like to see Hostgator honor it's committments and claims as well as not even apologizing for accusing me of using the account for data backup/storage, I just want that line of thinking to be corrected with the truth of the situation and the situation to be resolved in terms of a disk quota being engaged at 300GB which is what happened earlier today.

Guys, I'm honestly coming to you with integrity and please lets get these issues properly cleared up and please stop with the wrong accusations that my account has been used for data backup/storage... Also please ensure that the account is allowing me access to at least the 900GB that was supposed to be my quota when I started my account with Hostgator... All of my data is being used for my site, the public_html/developer/folder folder is a temporary upload folder to test the integrity of the files uploaded to the server before i bump them into the download directory of www.sitename.com/store so that the files are intact before being placed live in the store, that particular folder is an "upload/check/bump to store" folder and is not being used for data backup... If there are history logs for that folder you will see that clearly...

NB from http://hostjury.com/blog/view/86/can...o-of-hostgator

"Brent Oxley:

"I'm pretty hands on with all aspects of the business. A few things I do are advertising, paying the bills, firing, hiring, monitoring servers, tickets, and the hundred+ emails a day I receive. The majority of my time goes to helping customers. I ask all our customers to request for me personally if regular support channels ever fail.""

I'm personally requesting Brent's assistance on this matter as of this point in time. I believe this issue has been handled totally inappropriately and I think ultimately we can find a solution by working together that is mutually beneficial...

In Hopes that this will be satisfactorily resolved sooner rather than later,

to be continued...

[GatorDHanna]

Thanks for your post. I've sent you a PM requesting some additional information so I can look into this further. I appreciate your patience.

[GatorDHanna]

Dean,

I just spoke with Brent and he told me he is looking into your ticket now and will be in touch shortly regarding it. Thanks for your patience!

[GatorDHanna]

Hi Dean,

I see you were able to work with Brent and Patrick and get everything resolved to your satisfaction. Thank so much for your patience and understanding.