The Future of Data Storage: Why It's ‘Flash & Disk' & Not ‘Flash Vs Disk'

"The debate ignores a critical variable that changes the whole tenor of the debate - rate of data growth, which makes case for both technologies"

Flash or Disk? Here in lies the rub. That question, whether the future of data storage would be defined by flash memory or hard drives, has been the looming battle royale in the storage industry for the last several years.

Hard drives, of course, remain at the heart of the storage industry. Manufacturers shipped 524 million disk drives in 2012, according to iSuppli, more than ten times the 39 million flash drives. Worldwide manufacturing capacity for drives at the end of 2012 came to around 125,000 petabytes per quarter, compared to around 10,000 petabytes per quarter for flash. By volume, hard drives are tough to beat.

But look at the growth rates, say flash advocates. Shipments will grow from 39 million units in 2013 to 83 million units. By 2016, 239 million flash drives will be shipping annually, propelled by sales of tablets, smart phones and notebooks. Enterprise arrays based on flash will also gain a stronger following among corporate buyers like banks with an excruciating need for speed. Flash might be in the minority, argue advocates, but it has time on its side.

Flash memory & hard drives: enemies or friends?

The debate, however, ignores a critical variable that changes the whole tenor of the debate. And that variable is the rate of data growth. The opportunity and demand is simply too large for either technology alone. When you start to look at it in that context, you can start to see that the future really isn't "Flash versus Disk." Instead, it's "Flash and Disk."

Online auction house eBay, for instance, recently said it installed 100 petabytes of storage last year. Two terabytes is the same amount of information contained in all of the U.S. research libraries. The company's Big Data servers come with twelve drives containing two terabytes each.

"It is the storage, stupid," said Dean Nelson, eBay's vice president of Global Foundation Services. "The volume of growth of storage is insane."

IDC estimates that the total amount of digital information in the world grew 48 percent to 2.7 zettabytes in 2012. And a huge portion of that new data is unstructured files like YouTube videos and audio files. And on top of that add in the data that will be generated by valves, security cameras, light bulbs and the other residents of the coming Internet of Things. The world will need both technologies because it will need as much storage capacity as it can get its hands on. Gartner reports that many of its clients expanded their storage requirements by 80 percent in 2010, 2011 and 2012.

Economics play a role too. It's simply far less expensive to manufacture drives. A new facility for producing hard drives costs about $35 to $50 million and can be operational in about four months. A similar NAND flash factory that can produce 100,000 wafers a month cost $4 billion and takes two years to build. It would take $210 billion in factory investments to displace 15% of the demand for drives in 2014, according to Gartner. Partly as result, drives will invariably cost less. By 2016, a one to two terabyte drive will cost OEMs $39 to $45, according to Gartner, while a 300GB SSD drive will cost $90 to $100. Doesn't make sense.

Displacing all of the demand for drives would take over a trillion dollars of new investment. Besides being impractical, a sudden surge in investment would likely plunge the semiconductor industry into a massive slump.

So does this mean flash won't play a significant role? Not at all. We've been investing in flash and shipping flash-based products for over four years and will continue to expand our flash portfolio.

Flash will be deployed strategically. Flash drives will likely take over markets where power consumption and size outweigh cost and capacity. Notebooks will come with flash drives or hybrid drives, which combine both technologies. Back-up drives, desktops and set-top boxes will likely veer toward traditional drives.

In enterprises, most servers will come with a traditional drive or a hybrid drive alone with a flash-based PCIe card to accelerate the flow of data. Flash-based storage arrays will handle the most time-sensitive jobs, but be backed by even larger arrays of disks. Another interesting applications: PCIe cards, flash-based devices that sit inside servers to accelerate performance. Is it a storage technology or a server technology? To storage providers, it's another opportunity to expand.

Some manufacturers have also begun to develop storage systems with server capabilities. With these systems, storage effectively becomes the focal point of the data center, not the computer. Drives will experience a quantum leap in performance with the introduction of heat-assisted magnetic recording (HAMR) that will increase density to over a terabit per square inch. Flash manufacturers, however, will follow up with memory devices made with three-dimensional transistors a few years later. Every reaction will lead to a complimentary reaction.

The technology might want to see a battle between standards and technologies. But instead, it's going to get an uneasy truce.