Even backups may not save you
When the Colonial Pipeline ransomware attack became public in the first half of 2021, many were surprised that the company paid a ransom of $ 4.4 million to recover its business systems. After all, it’s unthinkable that a company this large doesn’t have backups in place. According to a article in the Wall Street Journal, however, the CEO of Colonial “authorized the payment of a ransom of $ 4.4 million because the executives did not know how much the cyber attack had violated his systems and, therefore, how long it would take to re-establish the pipeline “.
Colonial didn’t say much more about why they decided to pay the ransom, but, assuming the company has backup systems in place, there are two possible scenarios. The first is that the backups, themselves, have been affected by the ransomware malware. This seems unlikely, as a large organization like Colonial is likely following the 3-2-1 maxim for backup: three copies of data on two different forms of storage media with one offsite. Certainly, it is possible that there was still a logical connection between all the backups and the production network, which would have allowed the attack to access the backups. But generally, offsite backups are well protected against this type of attack and are often stored in a read-only format that cannot be encrypted or overwritten by malware.
The importance of recovery time
Recovery time is much more likely to be the issue. Unfortunately, while almost all organizations have some sort of backup system in place, far too many do not take into account how long it will take to recover. This is especially true in the case of a large-scale ransomware attack or a disaster that destroys a significant portion of company data.
For a company like Colonial Pipeline, getting the business back up and running as quickly as possible was the # 1 priority. Not only was the company losing millions of dollars in revenue every day, the pipeline shutdown sparked panic that led to gas shortages across the Southeastern United States. There is no way to know exactly what Colonial’s recovery time (RTO) goals were for the data it absolutely needed to have to run its business, but in my experience I have seen systems even in very large organizations that would need several weeks to restore enough data for the business to function.
So when faced with the choice of paying a ransom to recover data faster than an organization could with its backup systems, the cost of additional downtime is a much larger loss. In this situation, financially, it is an obvious financial decision to pay the ransom. However, paying a ransom raises a number of legal and ethical questions that companies should discuss with their boards and ethics committees.
Different technologies for different workloads
To avoid having to pay a ransom, organizations should think about backup and disaster recovery (DR) from a recovery perspective. While everyone wants to be back online as quickly as possible, it is important to rank data and applications according to their importance; it is far too expensive to require an RTO of a few minutes or hours for every piece of data in the organization. So it’s critical to understand the business’s tolerance for downtime for each workload, and then match those tolerances to the right solution.
The options include:
- Immediate full recovery: This will require a synchronous hot site, which is by far the most expensive approach. But if ransomware hits, the company will experience, at most, a slight setback in its operations. A good example here would be a case where a museum’s HVAC control systems being offline results in the destruction of priceless works of art.
- Continuous data protection: Backup providers and Backup as a Service (BaaS) providers have solutions that deliver RTOs from seconds to minutes. They are less expensive than a synchronous dynamic site, but more expensive than a traditional backup system. For this category, think of a comprehensive just-in-time supply chain system. If it is offline, a truck or ship can wait a few minutes before leaving with a load without causing significant damage.
- Backup systems: There is a wide range of RTOs and pricing, depending on how the solution is architected and whether one is working with a managed service provider. They can range from less than an hour to a few days or weeks, if not well designed. It is essential to test the system to ensure that it will provide the required RTO for each level of workload. A business application that performs weekly batch processing of information can remain offline for hours, or even days, without significant impact.
- Ribbon: This is the cheapest route, but recovery can take hours for a relatively small amount of data, and if the backups are stored offsite, it can take days. While not useful for large-scale recovery, tape is good for archiving data for compliance and legal purposes. Many companies have data that they are required by law to keep for seven, 10, or even more than 20 years, when the only time target for recovery is “within a reasonable time”.
Finally, as businesses review their backup and disaster recovery strategies to ensure they can meet RTOs, they must continue to function in the face of ransomware or other disasters. They should therefore not neglect regular testing. The IT infrastructure is constantly evolving. Businesses need to make sure they’ve taken IT dependencies into account so that workloads get picked up in the right order and everything runs smoothly. And they need to make sure new applications and data are protected with the RTOs they need.
Doing this in-house will require disaster recovery experts who can recommend the right balance of protection for each workload and the corresponding expense. That’s a lot to be expected from IT staff who only deal with disaster issues on occasion. Additionally, when you factor in the time spent on ongoing management, organizations may find that the DIY approach can be more expensive than using a Managed Service Provider (MSP).
No matter how an organization approaches backup and disaster recovery, it can’t take the approach that if backup is available, we’ve got you covered. Backups that cannot be recovered in time are only slightly better than having no backup at all.
Bret Piatt is CEO of OffSiteDataSync, a global provider of highly available and secure cloud data protection solutions including infrastructure (IaaS), disaster recovery (DRaaS) and backup (BaaS). The company offers the best data protection and availability solutions, based on market-leading technology from Veeam, Zerto, VMware and Cisco.