The backup industry has always had a compelling hypothesis: first, integrate the “control plane” and the “data plane” into a media server and second, optimize the data plane with variable length de-dupe in the form of a backup appliance: presto, now you can greatly reduce both capital and operational expenses. This hypothesis has served the traditional vendors well: they are still selling in the billions of dollars!  However, the fact of the matter is that they are on the wrong side of technology trends: both in terms of the emergence of the cloud as well as cloud applications and workloads.

Let’s head back a couple of decades when tape was the king of secondary storage. The arguments brought forward against tape were the following: although much cheaper than disk, the operational complexity was very high. For example, tapes have high seek times compared to disk, cannot handle more than one stream reliably, and have high operational maintenance overheads (e.g, temperature controlled environments). As a result, tape never graduated out of the high end enterprise, and did not have an answer to the fundamental advantages of de-duplicated disk: the capital expenditure was comparable and the operational expense was much less. In short order, a combination of Veritas media servers and Data Domain backup appliances became the de-facto standard for backing up Oracle databases.

Fast forward to the present, the same technical trends that killed tape are once again front and center, but they have a new target. The same arguments that were made against tape are now being made against disk: seek times for disks have barely improved in the last decade even though disk throughputs have improved, disks have high operational cost in large data centers because a constantly spinning disk eats up energy even when there is no I/O, disk drive failures and silent data errors are the norm rather than exception.

So what is replacing disk? In short order, it is elastic cloud storage, where there are no storage quotas, bandwidths are sufficient and scalable for all but the most latency sensitive workloads and the capital and operational expense is unmatched. The backup industry has taken the “logical” next step: let us port the backup media server-based software, purpose-built backup filesystems, and the backup appliances to the cloud and the problem is solved! After all, the cloud is just another storage target.

But this “logical” next step is the wrong one as the backup industry is ignoring two important facts.

First, porting the media server to the cloud ignores the basic fact that a cloud is geo-distributed with data flowing in and across the entire continent, if not the world. Having all data flowing in through a single media server is not only a deployment challenge, but also a severe performance bottleneck. The cloud is architected for applications to have high bandwidth access to a scalable object storage backend, not to another server sitting in one geographic location. Some companies are trying to scale out the media server, but they are solving the wrong problem: why try to create a distributed “data plane” (aka, yet more backup / storage filesystems), when the cloud has created one for you?

The similar argument works against porting backup appliances to the cloud. The key value proposition is using variable length de-dupe to reduce storage costs and operational expenses. Not only is there no need to optimize the data plane, but more importantly the era of variable length de-dupe is over!  Workloads are changing and the entire spectrum of traditional to next-generation applications is becoming real-time and high throughput. This has led data to become compressed on disk to make load times quicker with the unfortunate side-effect that variable length de-dupe does not work any more.

At Datos IO, we firmly believe that there’s a need to build a cloud-first “control plane” that takes advantage of the scale-out “data plane” in the cloud. The role of the control plane is to mediate access to the data plane without stepping in the way but at the same time, delivering higher storage efficiencies by understanding the semantics of the application, built from the ground up for third platform and enterprise applications. With RecoverX 2.0, we demonstrate how the control plane can not only tackle the petabyte scale of HDFS but also the high transaction throughput world of relational databases. Just another step towards our vision, with so much more to come! For a technologist like me, we couldn’t have a better opportunity at hand, to build the “Tesla of the Backup Industry.”