Emergency Drive Maintenance :: VPS349

[GatorJVess]

Hello Gators,

Unfortunately, we will be needing to perform unscheduled emergency disk maintenance on VPS349.

We do not have an ETA to the completion of this procedure, however we will update this forum post as we progress.

We appreciate your patience.
Thank you.

[GatorNathon]

Hello,

At this time it does appear that we have major file system corruption on VPS349 due to RAID card issues. The level of data loss is still unknown and we are preparing to restore from NAS backups if needed.

We are still looking into this issue now and will provide more information as it becomes available. We will be attempting to update this thread on the hour until this issue is fully resolved.

Thank you for your understanding and I assure you that we are working on getting everything back online as soon as possible.

[GatorNathon]

Hello,

As of now we are in the process of performing a FSCK in an attempt to restore data locally. This is the preferred option as it will have the most up to date files. While this is happening we are concurrently preparing files on the NAS server so if the FSCK fails we can immediately start syncing data.

Again, thank you for your patience and we will continue to keep you updated every hour.

[GatorNathon]

Hello,

It now appears as if the FSCK will not be a viable option and we will have to go with backups. There is still a possibility that some data on the local system can be restored but until the FSCK completes we will be unable to see exactly what we have.

The preparation of backups is still ongoing and as soon as the FSCK completes we should be able to start restoring accounts. We are still unsure of how long before this can be done. Also, we will be restoring backups as soon as their preparation is completed even if other backups are not ready. This will help to get some users online as soon as possible.

[GatorNathon]

Hello,

At this time we have no further updates. The FSCK is still ongoing as well as NAS backups being prepared.

Thank you for your patience.

[GatorNathon]

Hello,

Good news and bad news.

The Good- The FSCK is still ongoing but we are seeing data coming back. This is a great sign that we will not have to restore from backups. This is still up in the air though.

The Bad- We still have no ETA on when the FSCK will complete and when sites will be back online.

We are still keeping our fingers crossed for the time being but we should know more for our next update.

[GatorNathon]

Hello,

The FSCK is still ongoing. It should hopefully be completed shortly and we can get everyone back online.

[GatorJacob]

Hello,

At this time, the fsck has finished and containers are starting to come online. If there are issues with bringing up specific containers, we will attempt to fix and contact the customer individually if needed.

Thank you for your patience during this time.