or, How to Destroy an Entire Data Center with a Typo
First, MDD Hosting is awesome. I’ve been with them for about a year and a half, and will likely stay for decades if they stay awesome. Uptime – with the one big exception described below, has been good. Speed is great. Price is reasonable. They check all my must-have and nice-to-have features: LiteSpeed server; cPanel; free Let’s Encrypt SSL; CloudLinux account isolation technology; SSD storage; Softaculous Autoinstaller; PHP version selector; Not associated with Endurance International Group (EIG); No obnoxious up-selling or other BS marketing games. Most impressive is help service – always prompt, courteous, and knowledgeable.
But last September, MDD and its clients endured a terrible, horrible, no good, very bad day – several days actually.
Throughout the crisis MDD CEO Michael Denney was transparent and communicative. Some highlights from his initial email …
A server administrator, the most experienced administrator we have, made a big mistake. During some routine maintenance where he was supposed to perform a file system trim he mistakenly performed a block discard.
What does this mean?
The server administrator essentially told our storage platform to drop all data rather than simply dropping data that had been marked as deleted.
Why is restoration taking so long?
Normally we would rely on snapshots in our storage platform – simply mounting a snapshot from prior to the incident and booting servers back up. It would have taken minutes. We are not sure as of yet, and will need to investigate, but snapshots were disabled. I wish I could tell you why – and I wish I knew why – but we don’t know yet and will have to look into it.
Our disaster recovery server is our last resort to restore client data and, as it stands, is the only copy we have remaining of all client data.
We are sorry – we don’t want you to be offline any more than you do.
Of course I have backups of my sites and could have restored them myself, but that would be a lot of work. Also, I had already been demonetized by Google, so my sites being down wasn’t costing me much. MDD restored service in order of account age – long-time customers first. Which means I was last in line. Somehow though my server got bumped up to the middle of the queue. Still, my sites were down for three full days.
These things happen, I guess, and MDD has been superb in all other respects, so they still have my loyalty. A quibble though – it would have been a nice gesture if MDD honored their 1000% Uptime Guarantee without it having to be specifically requested by each client. I did request, and received 30 days hosting credit.