You might have seen EMC's announcement of Invista 2.0. I thought this topic deserved a bit of context.
Storage virtualization has turned out to be one of those long-simmering topics in this industry. And it's still going to simmer for a while, I'd predict. This game isn't anywhere near from being over yet.
Best as I can recollect, although the industry was talking about at the end of the 1990s, the discussion got very hot at EMC in the 2002-2003 timeframe.
We knew that this was going to be an important storage technology, but we had some hard choices to make.
If anything, one of the things I like about working for EMC is that we study the problem, and are willing to make the hard choices -- and stick with them -- to get to where we want to be with our customers.
And with the recent announcement of Invista 2.0, I think it's a good time to reflect on the journey, where EMC and the industry has ended up, and maybe a bit of what happens from here.
A Few Basics
Storage virtualization is nothing more than a convenient abstraction of storage between the device that has it (usually a storage array), and the application that uses it.
Like other virtualization abstractions, it can potentially offer benefits around consolidation, moving things around, certain aspects of management, and so on.
But, I'd offer, there's no single *best* way to implement it. "Best" defines on what you're looking for. And, since there's such incredible diversity in customer requirements, it's a hard argument to win.
From a pure technology perspective, there are generically four ways one could implement such an abstraction.
Server-based storage virtualization (aka volume managers) have been popular for over a decade. VMware's recent announcement around Storage VMotion will probably renew interest in this approach.
Another approach is to add an appliance in the I/O path. IBM and others have had some success with this approach for certain use cases.
There's a pure network approach, where the network device is taught how to support virtualization concepts (EMC's Invista and RainFinity fall into this category, for SAN and NAS respectively).
And, finally, an array can support many virtualization capabilities, either with storage located within the array (EMC's CLARiiON and DMX, among others), or with the array attaching to external storage (Hitachi's USP-V).
And, just to confuse matters, you'd make one sort of product choices around storage (or block) virtualization, and another set of choices around file virtualization. Since many organizations are finding they have bigger challenges with their file-oriented storage, rather than their block-oriented storage, it's not always obvious where the best place to start is.
Lots of choices, aren't there?
The Case For (and Against) Storage Virtualization In The Server
Doing storage virtualization in the server is relatively cheap and easy to implement compared to other alternatives.
The good news is that -- depending on your choice of "volume manager", you can get a certain degree of storage consolidation, somewhat easier management, and -- with Storage VMotion -- the potential of non-disruptive movement of storage objects from one array to another.
Does these approaches scale well? Perhaps not. With most traditional volume management solutions, there's no single point of control -- most everything is done on a server-by-server basis. And every server needs some piece of functionality to be loaded, configured and managed.
And if you're really into moving information around without disrupting service levels, let's face it, there's no free lunch between using server cycles for application processing, and using cycles for moving information from place to place.
Think about the performance impact of a backup, as an example, and you'll get the idea.
The Case For (and Against) Storage Virtualization In An Appliance
Many small players here, and one big one -- IBM. The idea is to create a server-class appliance that sits in the I/O stream, terminates I/Os, processes them, and sends them along to wherever they need to go.
IBM has poured a ton of R+D into this approach (perhaps the only storage technology they're heavily invested R+D) and made a go of it. They have delivered decent levels of performance and availability for moderately difficult use-cases.
Is it better than server-resident virtualization? At a minimum, you don't have to put software on every server you own, there's one place to go manage it (at moderate scale), and you have the benefit of the appliance's CPU cycles to do data movement, so service level impacts should be more moderated.
There's a thorny issue around caching write I/Os in an appliance, and maintaining integrity with the "real" image on the array.
But the only substantive criticism I'd offer would be the challenges of larger scale. Have lots of servers and lots of storage? You'll have lots of server virtualization appliances to go with it.
The Case For (and Against) Using A Storage Array As A Storage Virtualization Device
Most external storage arrays do a fair amount of virtualization-like functionality for the devices they contain. Hitachi took this idea one step further by allowing external storage to be attached behind an enterprise-class storage array, and extending the paradigm.
On one hand, this is an interesting approach. Conceptually, you're extending array functionality to subordinate arrays. Array technology (and its firmware) is relatively mature. And if you're managing the array, you're also managing the storage behind it. Performance and availability is decent as well. After several years, HDS can now point to customers who've implemented it and have it working.
But there's another side to this. First, an enterprise-class array can be an expensive device by any metric -- cost per port, footprint, cache etc. And when you're talking real scale for storage virtualization (lots of arrays), pooling and moving across domains isn't easy -- it's not a flat, pooled space where everyone can see everything.
And, since the most common use case for this approach seems to be repurposing stranded storage assets that are on the books and have to be there for a while, I have difficulty comparing the utility of reusing three-year-old+ storage devices as compared to the newer stuff that's faster, cheaper, more reliable and more energy efficient as well.
It's hard to build IT strategies when finance imposes some funky rules -- but that's the world many of us live in. And HDS has a potential answer for you if that's your situation.
What EMC Did
This isn't widely known, but one of the most free-wheeling (and extended) debates I've ever seen at EMC was to answer the question of "what do we do in this space?"
Lots of people looked at all the alternatives above, some very passionately. There were proposals to build an extended volume management capability that ran on servers. There was a proposal to build an SVC-like appliance. There were even proposals to enhance the Symmetrix to do what Hitachi eventually ended up doing.
Everyone had an intelligent and passionate opinion. It went on for a very long time.
But finally, some clear thinking emerged.
First, we believed that storage virtualization had its strongest value proposition at scale.
Yes, we could see the potential for customers of all sizes to potentially take advantage of what storage virtualization could do, but the more we talked to customers, the more we realized that the bigger you were, the more you needed this stuff.
Now, scale is not just capacity -- that's way too simple. Scale is performance, number of devices and ports, sophistication of requirements, complexity, etc. It's a multi-dimensional concept, at least for us.
And once we internalized the "scale matters" concept, one by one the alternatives became less and less attractive.
We began to see that the only candidate technology that could possibly scale economically was storage networking technology.
At the time, Cisco / Brocade / McData (as well as a few others) were proposing building SAN devices that would support the hardware enablers needed for storage virtualization at scale. Yes, there'd be a need for external control and coordination of these new ports, but -- at least through an engineer's eyes -- there was the "right" solution to the problem, as we saw it.
And, at the same time, we realized that this would be a long journey. The technology foundations were new. We wouldn't be first to market, as other approaches (use an appliance, repurpose an array) were easier to develop and get to market.
It was a hard choice -- either take the time to build something you believe in, or take the shortcut and get to market sooner. And there was another extended debate around that topic as well.
But our decision to build an entirely new storage virtualization platform on intelligent storage networking technology was fundamentally the right one in the long term -- and, one thing you've got to understand about EMC is that we do think in longer timeframes than most vendors.
Where We Are Today
I'd offer that we're starting to see some market validation.
Sure, it's easy for competitors (and a few smarmy journalists) to poke fun at Invista being "invisible".
We've got enough production customers now who are genuinely happy with what we've delivered that we feel we're seeing a bit of vindication in our very difficult (and expensive) decision.
We've built them a next-gen network storage virtualization platform that they can run their business on. Not some repurposed appliance, and not some feature on an existing storage array.
We know it will scale -- today, and in the future.
We know that -- as it scales -- the economics will win out over other alternatives.
We know that -- at scale -- it's easier to manage and support than other alternatives.
We know that we can deliver integrated management solutions that make large storage environments easier, and not more complex.
We know that -- as protocols change to 8Gb, 10Gb, FCoE or iSCSI -- we've got them covered.
We've convinced ourselves -- and more than a few customers -- that we've made the right architectural decision, and we're starting to see it play out in more and more customer environments -- especially environments that are operating at scale.
Maybe we haven't wone the battle (for the time being!) in the popular press. So be it.
It's not always about being the most popular kid on campus. Sometimes the geeks are right.
Does that mean you'll never see additional server-based storage virtualization capabilites from EMC?
I wouldn't take that bet. Nor would I bet against never seeing an appliance-oriented storage virtualization device, or even arrays that connect to other arrays.
It's not hard to see that -- over time -- customers will have different use cases, want them in different packages, and so on.
When you put our entire portfolio on the table for other storage functionality: arrays, tiering, replication, etc. -- you'll see a broad range of offerings with no one "right" solution that solves all problems.
But, at the same time, it's much easier to take lessons you've learned at large scale, and apply them to smaller environments. As opposed to taking small-architected solutions and trying to make them scale.
Congratulations to the Invista team -- it's been a long journey over the last few years. My thanks for keeping with it, and delivering a storage virtualization capability to our customers that gives them a differentiated, sustainable competitive advantage.
So, guys, what are you thinking about for Invista 3.0?