Written by

Cloud com­put­ing is every­where, and is becom­ing more and more impor­tant as we’re slow­ly but sure­ly mov­ing towards Edge Architectures in an Io(T)T world.

Recently, the French Hosting Provider OVHCloud suf­fered a huge data loss after a mas­sive fire in one of its data cen­ters in Strasbourg, FR. The fire destroyed an entire unit and caused sub­stan­tial dam­age to anoth­er. Due to this inci­dent and the issues raised about the provider’s respon­si­bil­i­ty over the data stored on their servers, we’ve received a lot of ques­tions from cus­tomers ask­ing about their own data host­ed by always­da­ta. Here are our answers on how we are secur­ing our Cloud platform.

First off, we made a quick thread on Twitter and LinkedIn about this topic.This arti­cle is their expand­ed ver­sion from an oper­a­tional point of view at a host­ing provider com­pa­ny. The tech­ni­cal answers are avail­able in our doc­u­men­ta­tion.

We’ve known OVHCloud for a while now. They’ve been a big play­er in the French and Europe host­ing are­na for a long time, and they were our first host­ing ser­vice before we migrat­ed to our own infra­struc­ture years ago. We were super impressed by their reac­tion to the fire and we are relieved  the whole team and all the fire­fight­ers involved are safe.

DRP, a critical document

A Disaster Recovery Plan is a doc­u­ment that describes pro­ce­dures that acti­vate in case of emer­gen­cies. This doc­u­ment is unique to your enter­prise as it is close­ly relat­ed to your busi­ness and depends on the answer to two key ques­tions: What are your threat mod­els, and how do you plan to han­dle them to sus­tain your activ­i­ty, even when things go wrong?

As the OVHCloud’s CTO Octave Klaba tweet­ed dur­ing the fire:

We rec­om­mend to acti­vate your Disaster Recovery Plan.

For a lot of their cus­tomers, this came as  a sur­prise, as spot­ted in the respons­es to Octave’s tweet: What is a DRP, and how can we acti­vate it?

The truth is you do have to be pre­pared for the worst. As stat­ed by Murphy’s law:

Anything that can go wrong will go wrong.

Your  DRP is there to pre­vent your busi­ness going down, poten­tial­ly per­ma­nent­ly. According to some sur­veys, com­pa­nies that suf­fer data loss will often per­ma­nent­ly lose 93% of it. 53% of busi­ness­es that suf­fer dis­as­trous data loss will per­ma­nent­ly close with­in two years. Your DRP should be updat­ed and test­ed reg­u­lar­ly, which will allow you to face any  sit­u­a­tion, even the most desperate.

Like any oth­er com­pa­ny, we need our own DRP that helps us stay safe in this kind of sit­u­a­tion. We are no excep­tion, espe­cial­ly when we have cus­tomers like you who trust us to take care of the sen­si­tive assets that pow­er your busi­ness. As a Cloud provider, you rely on our infra­struc­ture to run your web­sites, appli­ca­tions, ser­vices, and so on, all relat­ed to your data.

The Cloud is now a cen­tral piece of every busi­ness, from SMB  to major cor­po­ra­tions, schools and uni­ver­si­ties, gov­ern­ment insti­tu­tions, etc. It hosts emails, files, data­bas­es and tools. We pow­er far more than just web­sites in this era of con­nect­ed devices and we do have to prepare.

When a plan comes together

always­da­ta, as a Cloud Platform Service, pow­ers hun­dreds of servers. We’re pre­pared to face the worst, includ­ing loss of pow­er, hard­ware, or (heav­en for­bid!) a whole data hall or cen­ter. To mit­i­gate these risks, we have made design deci­sions which are applied to our over­all infrastructure:

  1. Spread the hard­ware: Our servers are host­ed on sev­er­al racks, all locat­ed in dif­fer­ent places. A bunch of spare servers are ready to be start­ed in those racks. In case of mate­r­i­al fail­ure, we’re ready to switch the ser­vices to oth­er servers, even those locat­ed in anoth­er rack, to ensure the con­ti­nu­ity of service.
  2. Be every­where: Our racks are locat­ed in dif­fer­ent data halls: Even in a major inci­dent involv­ing a com­plete data hall, we can switch ser­vice to oth­er racks and restore it quickly.
  3. Handle back­ups: We snap­shot your over­all data every sin­gle day and retain these records for 30 rolling days. These snap­shot back­ups con­tain your files, data­bas­es, emails, con­fig­u­ra­tions, and so on. They’re avail­able read-only from your account at any time. By sav­ing your data by default, we ensure we can restore your con­tent quick­ly in case of a disaster.
  4. Keep the back­ups safe: To pre­vent any nat­ur­al dis­as­ters from destroy­ing every­thing, our back­ups are locat­ed in a com­plete­ly sep­a­rate data cen­ter. This back­up DC is geo­graph­i­cal­ly dis­tant from the pro­duc­tion units. It’s also oper­at­ed by anoth­er provider, to main­tain independence.
  5. Keep the name­servers: The DNS archi­tec­ture is as crit­i­cal as the data. In case of fail­ing hard­ware, we need to get it up and run­ning to route the incom­ing traf­fic to new servers. So, we host our name­servers in dif­fer­ent loca­tions and with dif­fer­ent providers so the DNS records will remain avail­able for updates and con­tin­u­al main­te­nance of traf­fic flow.
  6. Stay con­nect­ed: We’re com­mit­ted to stay­ing inde­pen­dent from any one provider, mit­i­gat­ing the risk of a SPoF. Our Internet access is pro­vid­ed by four dif­fer­ent ISPs in order to absorb any traf­fic issues. Our Cloud infra­struc­ture will remain reach­able by any­one, regard­less of the sta­tus of these ISPs.
  7. Plumb the Network: Inside our data cen­ters, we man­age the over­all net­work infra­struc­ture to pro­vide each serv­er with a fall­back net­work link. If the main net­work trunk is unavail­able for any rea­son, we can eas­i­ly switch the link to the fall­back inter­face and re-enable the net­work immediately.
  8. Stand on the shoul­ders of the best: We chose Equinix as our main data cen­ter host for our pro­duc­tion units. renowned for their qual­i­ty and expe­ri­ence in terms of pre­mi­um host­ing all around the world. Our data cen­ters ben­e­fit from the best infra­struc­ture and cer­ti­fi­ca­tions in terms of secu­ri­ty and protection.
  9. Offer more to the most demand­ing: You may have dif­fer­ent needs for redun­dan­cy and scal­a­bil­i­ty accord­ing to your busi­ness require­ments. For all of you who need such infra­struc­ture, we offer a Gold Plan in Catalyst. Your servers are dis­trib­uted across geo­graph­i­cal­ly dis­tant DCs in redun­dant synced states. In case of fail­ure of one node, all the traf­fic is instant­ly redi­rect­ed to the oth­er node(s) to keep every­thing up.

What could possibly go wrong?

When every­thing comes togeth­er, you may think to be ready to face any sit­u­a­tion. We have a strong archi­tec­ture, well-designed process­es, and we pick qual­i­fied part­ners to back us. We’re ready. And we are no exception.

The truth is, all providers that have expe­ri­enced out­ages in the past, from AWS to OVHCloud, thought they were as ready as we are. The real­i­ty is you can only real­ly be ready for the worst-case sce­nar­ios you can think of. And this is where we talk about your respon­si­bil­i­ty.

We’ve con­sid­ered  the worst case sce­nar­ios that might occur. It also falls to you to do the same on your side. You know your busi­ness bet­ter than any­one. During the last OVHCloud out­age and the fire at SBG2, OVHCloud was prob­a­bly con­vinced they were real­ly well pre­pared, but a lot of their cus­tomers still lost their data. Some weren’t aware they had to pay an extra fee to back up their data. Some chose not to pay the extra.

Trusting your provider is essen­tial. But ulti­mate­ly it’s about under­stand­ing the scope of the ser­vice you buy. To keep you from a major data dis­as­ter, we chose to embed all these secu­ri­ty fea­tures by design. This is exact­ly what makes us a Cloud provider (or a man­aged PaaS provider). Some cus­tomers will need it, some won’t. Your respon­si­bil­i­ty here is to pick an offer that match­es your secu­ri­ty needs.

Be Prepared. Be Safe.

We’ve seen how a DRP  is essen­tial to keep your busi­ness up and run­ning in the face of dis­as­ter. Picking a provider that fits your needs in terms of pro­tec­tions is sim­i­lar to a good DRP but should not pre­vent you from hav­ing one. As your data is crit­i­cal, you do need to have a plan in case of fail­ure and to choose your part­ners accordingly.

Because you know your busi­ness bet­ter than any­one, this choice of providers is up to you: You can choose to build a DRP that doesn’t depend on a sin­gle provider, let­ting you switch quick­ly to a fall­back in case of emer­gency. Or you can pick a part­ner who pro­vides you with the guar­an­tees you require, so you can stay focused on your busi­ness continuity.

There is no right choice to make. It’s your deci­sion on how to han­dle a poten­tial: Will you lead the charge, or do you pre­fer to put the respon­si­bil­i­ty in the hands of your provider? Only you have the answer.

The worst dis­as­ter is the one you don’t see com­ing. Be pre­pared. Your data is the heart of your busi­ness. Keep’em secured. Keep’em safe.