The End Of Storage As We Knew It

This morning, I came across this grim piece from Chris Mellor at The Register, detailing evidence of the violent restructuring in our familiar storage array market.

Call it confirmation bias, but many of us have seen this coming for a while, and have acted accordingly. The numbers have been mediocre for many quarters, but recently the pace seems to be accelerating.

It wasn't all that long ago that storage companies were almost semi-glamorous in the enterprise IT world: EMC, NetApp, HDS, Pure, Nimble, et. al. Great growth all around, and tons of VC flowing into the sector.

What happened? And should we think of this as a temporary aberration, or a permanent structural shift?

The Lure Of Storage

I decided to join a small storage startup in 1994, moving from Silicon Valley to suburban Massachussets. At the time, EMC was a ~$400 million company with something like 1,600 employees.

I became sold on the idea that -- in the world of computing, information mattered. Capturing, persisting and re-using data was going to be a big deal, so storage was going to be really important.

And I thought standalone storage vendors would have a better strategic positioning than server vendors in this regard.

So I packed up my young family, and off we went on our adventure. It turned out to be a good decision, but nothing lasts forever.

The Last Twenty Years

Generally speaking, the last twenty years have been good for dedicated storage vendors.

The need to store and protect corporate data has grown rapidly, and there has been no shortage of new technologies that make things better: performance, capacity or functionality.

We had year after year of rosy industry numbers: healthy growth, good margins and wildly optimistic forecasts for countless zettabytes! of future growth.

The growth in information still holds true, it's the consumption model that's changed.

Once again, nothing lasts forever. The familiar model is gone, and it's not coming back.

I think there are three primary reasons we're seeing a structural shift in the storage sector.

1. Opportunities for differentiation are gone.

Early on, storage was difficult enough technologically that there was plenty of room to invest heavily in unique IP and establish differentiation. In addition to secret sauce, things like integration and availability and customer support really mattered to enterprise IT teams.

And, overall, aggregate demand for capacity, performance and functionality exceeded supply, hence overall growth.

But storage technology has matured. The opportunities for secret sauce are now few and far between, and generally quite niche. And once differentiation slows, commoditization kicks in with a vengeance.

New, disruptive technologies aren't the answer because everyone has access to the exact same tech.

For example, how many flash arrays are in the marketplace today? Hyperconverged systems with integrated storage? Cheap, dense storage boxes that do dedupe? As another example, NVMe flash and fabrics are all the rage today, but it won't be long before it's just another checkbox feature.

Without meaningful differentiation, storage (like compute) becomes a race to the bottom. Unless demand for standalone storage skyrockets, the overall market will shrink.  And that demand isn't there.

FWIW, part of Oracle's differentiation in the storage space is unique feature integration with the Oracle Database. The two of them working together do a long list of things that others simply can't. To the extent that differentiation is meaningful to all those Oracle users, it's an advantage.

2. Storage has become part of something else.

There's been a fundamental shift in storage consumption models: storage has become a feature of something else.

I think the trend started in earnest with converged systems, picked up a notch with hyperconverged, and went into full turbo mode with public cloud.

Decades ago, when cars came with crappy sound systems, there was a huge market for aftermarket stuff. Now that most cars come with a decent sound system that's fully integrated, not so much.

This has not been lost on standalone storage vendors. They are in a desperate race to surround themselves with converged (and now hyperconverged) consumption models.

Public cloud is a different story entirely. Not a one of them have figured out to be a meaningful player in any public cloud. And I don't think it's for lack of trying.

At Oracle, we're in pretty good shape in this regard. Our ZFS storage appliance is an integral part of our popular on-premises engineered systems offering. And there's over 600 PB of ZFS storage running in our public Oracle Cloud.

3. Supply exceeds demand.

When it comes to the ingredients used to build storage arrays,  unit prices have been melting faster than an ice cream cone on a hot summer day.

4TB drives became 6TB and then quickly 8TB and now we're looking at 10TB and 12TB drives. Flash storage started with megabytes, quickly jumped into the gigabyte range and is now safely in terabyte territory.

Fast processing cores are getting cheaper at the same pace. As are wickedly fast interconnects between components, storage and servers. I would argue that supply has exceeded demand, as there's only so much storage capacity and performance a traditional enterprise can consume.

There are only so many flat panel TVs you can put in your house. Or consider the the automotive world, where 300HP+ cars are now commonplace. How many people can justify a 600HP+ car for their daily driver?

OK, myself excepted, but ... good enough is good enough for most folks.

Not only that, thanks to decent storage software, storage array vendors are starting to get cut out of the middle -- adventurous buyers are starting to roll their own.

Witness the success of VSAN, Scality, and others.

What Lies Ahead?

One thing is clear: the standalone storage array industry already has its best days behind it.

We are witnessing the end game of a structural industry shift. It will still continue exist as a category, but -- before too long -- we won't be seeing any more Gartner MQs as no one will care as much as they once did.

Unless storage vendors can offer meaningful differentiation, as well as being part of something architecturally bigger (like a public cloud), it's a grim scenario.

You might have noticed that the VC spigot has turned off for on-prem storage array vendors, and the IPO market has not been kind to the new entrants.

Storage *software* companies that have a decent public cloud angle might still be an opportunity though. Or not.

It was a nice run, but nothing lasts forever.

----------------------------------

Like this post?  Why not subscribe via email?

 

Comments

[Yaronhaviv]

Chuck,

While i 100% agree w your observations re the on prem storage market, cloud providers like AWS, GCP and Azure also have quite a different stack than on prem or Oracle with scale-out built-in to the Application/databases coupled with DAS

You praise Oracle ZFS when in fact it serves legacy workloads, to compete Oracle need to redesign for scale-out and cloud-native (12 factor) as well, or it won't manage to become relevant in the cloud wars, i suggest you look up the Architecture of AWS SQL DB Aurora or Google Spanner

Yaron

[Chuck Hollis]

Hi Yaron

I think your views might be too simplistic. When it comes to enterprise IT, the vast majority of workloads aren't cloud-native, and won't be anytime soon. Today, Oracle offers various cloud storage services that are similar to what you'd find in both AWS and Azure.

Cheers!

Chuck

[Yaronhaviv]

Chuck,

the $13B (and rapidly growing) AWS revenue is not from legacy workloads, its people that understand legacy Apps don't help them with the Digital transformation or the existential threats and they consume services/APIs to move faster vs run IT shops or VMs/vDisks.

yes, you are right about Enterprise IT having many legacy apps that wont transform easily but this $ is shrinking, some are just going to be ditched on the road in favor of Office365, SalesForce and the so many mobile first modern apps. if you are betting Oracle growth on Legacy Apps u wont be in a different shape than those storage companies, and AWS will pick the data in 100PB trucks.

believe me i know Oracle, AWS, GCP & Azure (& EMC) tech inside/out, helped to build it in my previous jobs, there is nothing similar. building S3 or some NoSQL doesn't come close to the amount of modern services and tight integration that AWS offers. i can go into details offline why Exadata/RDS is nice but not going to take you far in terms of scale or TP/s.

i work with many enterprises, haven't seen them considering Oracle seriously for cloud beside existing workloads or some that break under the huge pressure of Oracle sales, they need a cloud vendor that innovates and takes them through the digital transformation, not one that focus on the legacy, or one that is known for unfriendly licencing.

you may want to check my 1yr old post, they have progressed quite a bit since: https://sdsblog.com/cloud-data-services/

Yaron

[Chuck Hollis]

Hi Yaron, thanks for your comments. Best of luck with your new venture.

I think you've got a lot of your basic facts wrong.

Yes, Amazon AWS is growing nicely. But it's a small fraction of the >$1 trillion IT market. And significant enterprise adoption has yet to begin. I would argue this is for several reasons, one of the biggest is that it's completely incompatible with what most enterprise IT organizations need.

I remember a time when we all thought AOL would rule the internet, and Netscape would be the browser of choice. Early days in the cloud world, my friend.

Yes, SaaS is a factor. As I understand it, Oracle is the #2 SaaS vendor behind Salesforce, growing faster. And we also have nicely growing IaaS and PaaS businesses in addition to on-premises.

No one is betting on the growth of legacy apps. But one of the unique aspects of enterprise IT is is that old and new have to work together. No one gets the luxury of throwing everything away and starting over in some cloud. Tactical projects can be greenfield, but if they're successful they'll get woven into everything else an enterprise does.

I love how all the cloud fanboys talk about "scale", but never discuss use case or relevancy. "Scale" means one thing if I'm Facebook, something completely different if I'm a bank in the Middle East. The forms of scale relevant to most enterprise IT organizations is what Oracle tends to focus on.

Everyone in this industry is focused on digital transformation, including Oracle. To claim otherwise is insincere. Some are zealots, others are pragmatists.

And, to this day, none of the alternative data management approaches have even begun to recreate the rich functionality of the Oracle Database (I'm thinking Aurora, RedShift, MongoDB et. al.), nor have they enjoyed even a small fraction of the market success. This may change in the future, of course.

I do believe that new forms of applications will require new forms of data services. Scale-out IoT comes to mind, as do cutting-edge analytics, not to mention ML and blockchain. All nice shiny topics for startups to invest in -- as well as established, well-funded vendors such as Oracle.

Trying to pit an established, proven and credible IT vendor against newer tech ignores the fact that the same exact same tech is available to all players. The variability come in how different companies evolve as the market evolves.

Good luck with your startup. If you have the opportunity, I'd recommend you might spend more time in traditional IT settings -- it's a very different form of IT.

Best of luck

-- Chuck

[Sukh Grewal]

I'd never count Oracle out. There was a time when they were infants and competing with Ingress. Guess who won.
For storage, take a look at Peaxy. They are at the extreme high end for storage and the market need is still very far from being met for extreme in both speed and quantity at the same time. We certainly have quantity if that's all we want and we can speed with flash arrays and on board storage - but getting both at the sametime is still the challenge?

[Yaronhaviv]

Chuck,

thanks for the wishes, i hope Oracle succeed as well, we are far more likely to partner with you than with AWS :) already integrate with Oracle and most of our customers are T1 Enterprises, IoT, and SaaS.

yes, technology is available to everyone, but not everyone knows how to make the most out of it or execute at the same pace.
my point is that Block/POSIX abstractions are not optimal for scale-out data services (especially in the age of NVMe & NVM), Oracle actually pioneered that notion w Exadata Smart-scan & column compression. we just took that concept 2 steps further. can also see most new data platforms, not just cloud/hadoop (also Vertica, Splunk, ..) use DAS, this means trouble for NAS/SAN.

happy to give you a deeper dive/demo on how we process 2M records/sec/node, cut analytic queries from hours to seconds, or process events from billions of sensors and visualize in real-time. only way to do it is avoiding the traditional serialized memory and storage stack (a.k.a ZFS).

not confused about Enterprise IT, we spend much of our energy in delivering the industry most secure data platform, data governance, and ways to seamlessly integrate with both traditional DBs/Filers as well as Hadoop, Spark, Cloud and emerging IoT patterns. but what makes the most impact on our customers and why some call us "magic" is how we solve very hard business challenges. learned the hard way that its hard to sell to IT, IT is process oriented and move slow, you need to solve biz problems and the Biz unit will push IT. its another challenge the shrinking storage camp have, they can hardly make a difference at the App/Biz layer to justify decent margins.

i hope you wont dismiss my feedback too easily, i like to see you succeed, I have many Oracle friends, but it may require you to adopt some fresh approaches

thanks again, yaron

[Steve Kenniston]

Chuck,

I wouldn’t actually call it “confirmation bias” but more like vendor marketeering bias. It’s a little disappointing that you, of all people, would write this and not expect people to see through it. It’s a little disheartening. Yaron (in the comments below) and yourself are two of the most prolific bloggers in the industry and I read almost everything you guys write. Usually your posts take a vendor neutral approach and you lay out the facts as facts, not put down a vendor or industry. But Chuck, come on, since going to Oracle I have seen your stuff lean a little into the wind when it comes to software and cloud as Oracle have more solutions in that space. Yes, people write about the environments they are in, but this piece is over the top. If you had predicted the demise of the storage business while you were at EMC (and written about it), I would give you the props to say, “see, I told you so,” but never once did you talk about what may happen in this space.

I don’t know where to begin. First, I need to take exception with your statement about “supply exceeding demand.” If you look at the chart from your post and understand anything going on outside the database world, it is clear “supply has” NOT “exceeded demand.”

When you think about what is going on in Genomics, IoT, electronic medical health records, AR/VR, fog computing etc… It continues to grow – exponentially. If you read Chris’s post more deeply though, it comes down to money and yes, I would agree, consumption models. Clients want to store as much as they can, as cheaply as they can. While it does talk about the AFA market growing – growing from zero to where they are today would explain the growth, not that utilizing fast media provides any advantage more than speed. As Chris rightly points out “In general, enterprises want their bits to live in trailer parks and not fancy condos. But they'll pay for condos if they can get really fast access to all the floors.” We see this all the time with our conversations with clients at INFINIDAT. Clients don’t need to get to all the floors really fast.

The rosy years you speak of in the business, transformed in part due to the fact there was no innovation, vendors just charged $100,000 for their software on top of a $5,000 commodity server. So, the storage days of old may be gone, but to that I say, good riddance. The reality is the old guard of incumbent storage vendors encapsulate everything that is terrible about the old storage industry. The business model of running storage companies like they deliver a luxury product has failed and is never, ever coming back. While it is true that storage has matured, you say early on there was unique IP and differentiation – mind you that unique IP and differentiation was developed by Moshe Yanai, founding innovator of EMC’s first serious foray into storage, and now the founder of INFINIDAT and with 130 patents, there is plenty of innovation in storage.

I would agree that putting the most expensive media in an array to solve a problem isn’t innovative or the answer, so just throwing SSD drives at the problem doesn’t address the bigger issue, but that isn’t what all vendors are doing. Your argument that Oracle’s differentiation is the fact that it works with its own database is an advantage is really vendor lock-in, the one thing clients hate the most (and I do hear it when I visit with them). This again is nothing more than saying “If you want your database to run right, you need to buy my stuff.” It doesn’t help the customer, because you are right, there has been no new computer science developed in storage for decades – until now.

I also would disagree that the storage vendors have seen their best days. Today we, as a storage industry, have one job; build new kinds of software for storing incredibly large amounts of information at radically lower cost than EMC, and other old world tech companies, as well as the likes of new cloud storage solutions from folks like Amazon, and do so in a much more flexible, cost effective manner.

No matter what Chris’s article says, data is still growing and it needs to be stored somewhere – people are going to want to store it where it makes the most sense, so vendors will need to continue to innovate – not something I have seen from the database market.

[Chuck Hollis]

Hi Steve. Thanks for the lengthy comments. I gather your at INFINIDAT now?

You accuse me of vendor bias. Guilty as charged. I freely admit it, but I think you might have the causality reversed. I came to Oracle because I believed in certain things, and not the other way around.

I saw the demise of the traditional storage array business as a serious threat about five years ago. That was part of the reason I left for VMware when I did. And worked on VSAN by the way ...

To be realistic, no one in their right mind would publicly proclaim the demise of the company that's paying their bills. You wouldn't, and I wouldn't.

I stand by my assertion that the market demand for differentiated IP in traditional storage arrays is dwindling. All those storage boxes are starting to look like, well, commodities to me. You may have a different opinion, naturally.

I also stand by my assertion that supply for traditional storage arrays is greater than demand. The IDC numbers bear this out. Yes, that overall demand for storage is going elsewhere -- got that.

You call it lock in, I call it co-engineering. The technical evidence is compelling, when you co-engineer the database and the storage together, you can do meaningful things that are relevant to customer outcomes.

If someone doesn't see the value of that integration, the Oracle database runs nicely on just about anything. Of course, it won't run as fast or as efficiently. Think of it as another choice.

Best of luck with your new venture. It would be great if you and the crew at INFINDAT could prove me wrong.

-- Chuck

[Raj Srinivasan, CCS Technologies]

A very well researched write-up. But this change was long time coming.