Once Upon A Time

Once upon a time -- and a very long time ago -- there was a very successful storage company. 

It made a good product, sold it effectively to customers, and supported them well.  It made a lot of money, and was seen as successful.

However, the seeds of its success ultimately proved to be its eventual undoing.

I Was There

It's the late 1990s, and I was working at EMC.  We had a very successful product: Symmetrix.  We posted continual revenue and margin growth, and were very much in love with ourselves and our stock.

But there was a darker side to all of this.

Because we had only one product, we only had one answer to any problem.  For some customers, our single product was a good fit.  For others, it wasn't.  And customers tend to resent it when you sell them something that doesn't fit.

It didn't matter what the question was, the answer was always "Symmetrix".  If we didn't have it, you didn't need it.  We rocked, everyone else sucks.

At the time, our success bred a certain type of arrogance that you rarely see these days.  We couldn't accept that there might be other technologies in the market that did a pretty good job as well.  Nobody "got it" as we thought we did.

We furiously argued our religious perspective with customers, often to the point where they didn't want to talk to us any more.  If you worked for the company, you had to drink and share the kool-aid every day. 

As I remember the time, basically all we did was blab about our product, our company and our stock price.  It was all about being "on message".

No discussion or debate permitted.

One example was around backup.  Since we didn't make tape drives -- or backup software -- at the time, we tried to convince people that replicas were just as good as traditional backups.  I think we ended up doing some harm to the people who believed us at the time.

Another example was RAID 1 vs RAID 5.  As the industry moved to RAID 5, we steadfastly hung on to our belief that RAID 1 was just "better".  Reminds me of how some vendors are reacting to things like flash drives and autotiering.

Worse, we didn't keep an eye on our pricing.  At the time, we saw an opportunity to get high margins for our product, and we did the capitalistic thing.  Customers knew we were were doing this, and got resentful.

We weren't investing in the future, either.  Our product was starting to look a bit old-school compared to the newer stuff out there.  We got almost all of our revenue from a single architecture that wasn't getting any younger.  And there was no incentive to diversify our portfolio or offerings.

After all, why mess with success?

Wall Street Vs. Customers

Perhaps the worst part of it at the time was our obsession with our stock price, and what analysts were saying about the stock.  For many people at EMC at the time, you'll recall that the ultimate barometer of success was the daily stock price.

It became less about the really important stuff that supported the stock price long term, like creating innovative products, or fostering long-term relationships with customers.  And it led to some pretty bizarre behaviors and decisions at the time.

It's hard to be open to things like long-term customer satisfaction -- and secular shifts in the industry -- if you've convinced yourself that you're successful. 

After all, if you're doing good, why mess with success?

Payback Time

I got the first inkling that something was seriously wrong in our happy, self-obsessed bubble-land at a customer council in the late 1990s.

It was a 1x1 customer conversation along the lines of "the world has changed, and we're doing everything we can to get off of EMC".

A sobering moment.  I started to probe around.  Not good. By the time you see the first termite, the damage is already done.

Such was the case with EMC at the time.  As the economy contracted rapidly, and the dotcom bubble burst, we were badly exposed. 

We had a single product that didn't fit what the market needed.  We had treated many of our customers poorly, so they generally weren't rooting for us. And there was plenty of "good enough" solutions that did what customers wanted for a fraction of the cost.

Worse yet, we had a bunch of people who were in complete denial, and were now part of the problem instead of the solution.

Serious times indeed.  It was one of the grimmest times of my career.  Many of us worked night-and-day for many years to repair our products, our positioning, our relationships -- and, ultimately -- our corporate culture.

From The Ashes

Before it all came undone, Joe Tucci had just been appointed as the new CEO.  He'd been on the job a while when it became apparent EMC was in deep yogurt.

Fortunately for all of us, Joe had seen this movie before.  He knew what had to be done, and got on with it.  Joe's vision, strategy and passion is directly responsible for the EMC you see today. 

Someday, I hope his story is written and shared.  It's a study in leadership and character that needs to be told as an example for others.  I feel very privileged to be a small part of it.

But here's the lesson: without our near-death experience, we wouldn't have changed our ways.  Ultimately, we became a far more viable and strategic player in the IT industry rather than just being a single-product storage company.

So, Here's The Question ...

Does history repeat itself?

Comments

[Wordofpie]

Chuck, great post. History always repeats itself. The goal is for you not to repeat history that you have lived or studied. While this story had a happy ending, it is a story that doesn't always end that way.

Just ask WordPerfect what happened when they didn't jump on Windows.

-Pie

[Vaughn Stewart]

Chuck,

Nice story. I'm glad that EMC has not only survived but thrived since those dark days. You guys are truly a very different company than you were in the 90s.

Thanks for sharing the story.

[NerdBlurt]

I would say History can repeat itself if you haven't learned from the mistakes of the past. That is what gives all of us our character today. Change 1 thing from past and your future has a different outcome. Very good post!

[John F.]

No Chuck, these days EMC is a multi-product player in the industry. The problem is that, instead of solutions, EMC is currenty offering cardboard cutouts of competitor's products along with vaporware "visions". Given the EMC IT aggregation trend through aquisitions, the market expects a lot more. When will EMC truely integrate the disparate products it offers? Anything less leads to disapointnment. EMC has a huge portfolio of great technologies; it's long past time to start delivering.

YANM

[Sto Rage]

Ha! I think this is a tongue-in-cheek reference to NetApp's success using their single platform (ONTAP).
I don't fully agree with Chuck on this one though, unlike the Symmetrix, NetApp's ONTAP is very flexible to make is a compelling solution in today's market.
It still has some legs, the next step for them would to be to get N+1 clustering right.

[ChrisB]

Ithinka couple of the commented have missed the point all together. Chuck's post is about EMC's journey and the hubris that allowed them to hit a low in the early part of the last decade. His question (and warning) were internal. Those fighting the changes in play at EMC will find themselves in the position that many of the "dinosaurs" were in back then.

The fact that some other manufacturer's fans or employees may feel this applies to their current situation should be a warning to them. This is fact, not an insult. The best thing all of us can do is learn from each other's mistakes.

[Howard Marks]

History always repeats itself. I've found that companies where the rank and file watch the stock price on a daily basis are ready for a fall. I remember working at NetWorld and the Novell guys were all a twitter that the stock hit a new high while I was ripping into them about how the NetWare Naming Service was ruminant excrement.

[Joe Svankanski]

Wow the culture message takes time to flow from HO to the outposts! The EMC I worked for in Australia in 2004-2006 was like what you describe for the 1990's.

[Fan of Joe Tucci]

Great post Chuck!!!

Here are a few links that reminds us of Mr. Joe Tucci's greatness.

http://www.usatoday.com/money/companies/management/2005-09-25-emc_x.htm
http://www.businessweek.com/magazine/content/02_10/b3773092.htm
http://www.danielroth.net/archive/2002/07/can_emc_restore.html

I hope he isn't retiring until my journey is complete. =)

[Victor]

History does NOT repeat itself, but it rhymes....

[Chuck Hollis]

Hi everyone

Just a reminder -- the rules of the road are clear: please disclose your affiliation if you work for a vendor.

Not only do I demand it for this blog, it's generally good practice to be transparent about your affiliations.

Especially if your goal is to throw mud -- John F, I'm looking at you!

(by the way: YANM == Yet Another Netapp Mudslinger)

-- Chuck

[Chuck Hollis]

BTW, I'm *this close* to writing an "emperor has no clothes" post on the whole "secure multitenancy solution" from NetApp. Every time I see that in print, my blood pressure goes up a notch.

It takes about 10 seconds to show that (a) it's not secure, and (b) it's not multitenant either.

It's not secure in that it does absolutely nothing to protect the tenant from the service provider. As in the service provider has unfettered access to tenant data and code.

And it's not multi-tenant in that there's no QoS isolation in the storage subsystem. As in any tenant can consume IOPs freely, with no hardwall.

Other than that, it's a darn fine piece of marketing. Pinocchio would be proud.

-- Chuck

[twitter.com/HPStorageGuy]

Hey Chuck,

Nice to hear such honesty from you since I was on the receiving end of many the attitudes and arrogance you described. HP and EMC were originally partners through the mid-90's - I managed the midrange of our HP portfolio. I was always shocked at the bravado of EMC reps calling on HP customers trying to stuff all of them into Symmetrix.

Did our HP blog talking about LTO5 start you down this path (http://bit.ly/aXEU4x)?

And of course you know I work for HP. And you'll be glad to know that I'm not at all Lonesome these days ;)

[Mike Riley]

Hi, Chuck,

NetApp employee here. I think the EMC turnaround story is a great one. Sometimes people ask about NetApp's success and the market turns that seem to be hitting NetApp in its sweet spot at just the right time (e.g. virtualization, unified storage, flash as cache, etc.) - was it luck or vision? Just like I'm sure folks at EMC see it: we'll take it either way. Believe it or not, I really don't want to see history repeat itself over at EMC. It's a great story.

As far as SMT goes, I'd recommend people refer to the Cisco validated design and implementation guides:

http://media.netapp.com/documents/cisco-validated-design.pdf
http://media.netapp.com/documents/SMT-CVD-deployment.pdf

You'll find the info you need on the layers of security and QoS. We (NetApp/Cisco/VMware) have customers doing this today. You can lock out the SP if you want but at some point decisions do need to be made on items like physical access and the extent of service you would like from your SP. Vaughn Stewart put together some additional detail at his blog:

http://blogs.netapp.com/virtualstorageguy/2010/04/cisco-netapp-vmware-secure-multi-tenancy-updates.html

Thanks,

Mike

[Craig Steel]

As part of the vSpecialists Sales organization here at EMC I see the underpinnings of Chucks "Warning" every single day with our field sales org and with customers. My understanding of this message is things are changing. Joe calls it “A Sea Change in IT” namely “The Private Cloud”, and in the framework of “Cloud” computing Storage no longer sells Storage. You have to take a broader view of what customers are trying to accomplish in delivering IT as a service. Yes, we need to continue to “feed the mouse” in our 90 day sales cycles to keep the street happy however, our customers are counting on us to deliver solutions that help them deliver an SLA to the business at a low cost in the most operationally efficient manner possible. The days of customers caring whose logo is on the box that holds the spinning disk are coming to and end. I trust Joe Tucci and believe his vision for the journey to the private cloud is of course spot on!
-Craig

[Chuck Hollis]

Love the fact that EMC competitors use my blog to pimp their latest. Anyway ...

Mike, the very doc you refer everyone to is the one that raised the issues in the very first place. I can go back and quote the specific passages if you like.

Better to address important issues like "security" and "multitenancy" head on, rather than attempt to cloak them.

You've consistently ducked the core discussion, as has Vaughn. So it's pretty clear to me that none of you really want to talk about it. That's OK. I won't talk about it if you don't.

To restate: there is nothing -- repeat nothing -- in the solution that appears provide any degree of security protection from the actions of the service provider.

It's basically a "trust and pray" model.

And there appears to be nothing -- repeat nothing -- that prevents one tenant from consuming a disproportionate amount of storage CPU, cache and bandwidth resources.

So it ain't multi-tenant, since you can't guarantee minimums of these resources for tenants of a given array.

Your observation that "people are doing it today" logically does not parse. For example, people smoke, eat fatty foods and drive without seatbelts -- this doesn't mean that these are good or smart things to do.

Attempting to use other vendor's names to help validate NetApp's work doesn't really change the discussion either. A wrong answer is a wrong answer ... no matter whose name you plaster on NetApp's work.

I can always re-post the videos from GestaltIT's Field Day, where @texiwill took the solution apart piece by piece using different arguments, if you'd like a different take from a non-vendor.

It wasn't pretty then, and it's not pretty now either.

If you'd like to go a few rounds here on this blog, I'm game. I think I owe it to the community to understand what's really being offered here, and how it can get them in deep yogurt if they don't do some careful digging.

My goal would be to have a truthful and candid of discussion about what actual capabilities are there, and, more importantly, what's not -- and let people decide for themselves.

The best suggestion I've heard from a NetApp employee is that it should have been called "Simple Multitenancy" -- because -- in effect -- that's what you've actually done.

Cheers!

-- Chuck

[W. Curtis Preston]

EMC was a company that made one product that did one thing - block storage. It was SCSI & FC disk in an array that you could connect to via SCSI/FC or FICON. And it could replicate to another array just like it and make BCVs along the way.

You've come a long way since then, starting with (from my memory) acquiring a more moderately-priced disk array line that helped you to appeal to a different type of customer (Clarion). But then you had TWO products that did one thing: FC/SCSI. You needed more. Then you added the ability for them to do NAS and iSCSI.

Then, led by Mr. Tucci, you acquired four different backup software companies (Legato, Dantz, Avamar, & Mozy), some backup hardware (Data Domain, as well as Quantum & Falconstor via OEM agmts), some archive software (*xtender) and hardware companies (Centerra), a document management company (Documentum), a server virtualization company (VMware), a security company (RSA), a home storage company (IOMega), and a bunch of others that are escaping my memory. So now you are a company with a bunch of different products that do a bunch of different things, most of which do not talk to each other very much.

Meanwhile there is another company who has, as you say, "one product." But unlike your one product of years gone by, it can do many things. Their product can do several things: FC/SCSI, iSCSI, & NAS. It also can house three different classes of disk in the same array, and connect to several differently-powered controllers, allowing them to appeal to the high and low end of the market all with the "one product." They also acknowledged that not all data in the world is going to live on one of their arrays, so they also wrote their own backup product that can do near-CDP backups of open systems to their arrays. They also expanded the capabilities of that one product to support archiving. They did all of what I described above without acquiring a single product. (I'm not saying they never acquired anyone. I'm just saying they didn't have to acquire any products to do all of the above.) The result is "one product" that behaves like many products -- all of which were built to talk to each other.

You may argue that your collective suite of products is better than theirs, and I'll leave it up to them to argue the opposite, but I'll say a few things in conclusion.

Some people like companies who build everything themselves, because it tends to create products that work really well together. Other people like companies that acquire best of breed tech, because they like getting best of breed tech all from one place and are less concerned about how those products interact. Therefore, some people will be naturally drawn to you or those other guys. Neither approach is "wrong," and their advantages to both approaches.

But the main thing I'm trying to say with my comment is that the veiled implications of this article do not apply to the company that you're trying to apply them to. Neither them, their culture, nor their product line is anything like the EMC of old.

[Rob]

> Does history repeat itself?

Yes. I read once that HP is continually re-inventing itself to the point of killing off successful product lines so they don't get stuck I guess in a product rut.

Focus on core is another mantra, countless examples of that. New CEOs come in and get rid of all these divisions that aren't core.

In some senses, I think you are drawing a parallel for that "N" company. "If all you do is storage..." Could be. Maybe the bottom totally drops out of storage someday. Not far from the realm of possibility. The price plummets and certain folks are left with a single arrow. That was McNeally (former Sun CEO). "Get behind
the arrow." You see how well that worked. But I think N would argue they do more than storage (software to manage
replication, other value adds)

I wouldn't want to be very narrow in IT these days. There may be a shift that leaves your company in the dust because a better cheaper solution came along.

[Dimitris Krekoukias]

Hey Chuck,

Dimitris @ NetApp here.

Just to clarify Re secure multi-tenancy (unless I'm missing something):

All NetApp systems come (no charge) with QoS software that controls CPU, memory and I/O prioritization.

So, customers at a provider absolutely could enjoy fencing.

And they already do, since the product is deployed at the largest providers.

Just wanted to clear that up.

Great story, BTW!

D

[Chuck Hollis]

Wow, Dmitris -- that's a complete surprise! It wasn't part of the solution brief, and no one can find a link to what you speak of.

If you could provide a link to the technical documentation, I'll be glad to admit my error and stand corrected once I've had a chance to review it.

BTW, I'll be looking for tenant granular control (presume a per-VM model for starters), and policy-based prioritization of CPU, cache, and port bandwidth within a shared array.

I'll also be interested to see how you do I/O prioritization on a shared aggregate when multiple tenants are sharing the same physical storage.

I also would suggest that you publicize this software capability of which you speak, because we run into NetApp customers all the time who are struggling to segment workloads on your products.

BTW "deployed" doesn't mean the same thing as "effective", does it?

Looking forward to your sharing ...

- Chuck

[Chuck Hollis]

W. Curtis Preston

Always a pleasure to hear from you.

I guess I'll have to go back to one of my favorite quotes, e.g. this industry is comprised of three groups of people: those who build technology, those use technology and those who make a living criticizing what the other two are doing.

Best regards ...

-- Chuck

[Mike Riley]

Hi, Chuck,

Yes Chuck, I think both the NetApp story and the EMC story are great things. I didn't know that was pimping but, O.K. (Are you the same guy that squirts kids with the hose when they come near your lawn, too?) I kid but isn't there room for two good stories in the industry?

I'd be happy to help round this issue out. I wasn't attempting to cloak the issue unless of course you consider design and implementation guides validated by VMware as well as Cisco a giant Kabuki dance. I was merely trying to be brief since both of those guides together amount to about 200 pages of detailed configuration information. If folks want to check out your or my claims, the guides are there and engineers from Cisco, VMWare and/or NetApp would be happy to field questions.

I also believe that, yes, having this design validated by strong 3rd party partners is a testimony to the validity of the solution. The alternative ventures down a conspiratorial path with a touch of paranoia thrown in for good measure. I know every security expert needs a touch of paranoia but we're not all out to pull the wool over customer eyes. As testimony to that, I point out that we have many very large customers (service providers) using this SMT solution today. Unless your fairy tale includes a pair of ruby red slippers, I'm not sure how you can get away from the fact that a) it is a validated solution that b) customers are indeed using today in providing c) a secure multi-tenant environment. Res ipsa loquitur.

On a technical level, QoS, as Dimitris points out, has been a part of NetApp systems for a few years now. Your simply wrong on that count. I don't mean to question your credibility but this is a binary point.

As far as locking out users/administrators, you'll have to give a description of just how a service provider would provide their service? The general answer without knowing details is you can absolutely deploy a vFiler and lock everyone out including the service provider. Now, we can go 'round-and-round with the different permutations and combinations but the basic question is can it be done.

Now, one point of confusion may be that we don't provide these services in the same way as EMC. To coin a phrase with from Chad Sakac, I'm not saying EMC can't do it. I'm just saying we can - not wrong but different.

[Chuck Hollis]

Hi Mike

Nice comment - really -- but there's so much bobbing and weaving going on your part that I'm getting dizzy.

So, I'm going to slow you down a little, and narrow the discussion. I'm looking for evidence on each, not one blogger referencing yet another blogger.

PART ONE -- MULTITENANT SECURITY

NetApp claims that storage resources can be shared and secured using a multi-tenant model. I ask "gee, how do you protect tenants from service provders?".

You change the subject quickly. I think I understand why.

Locking down individual hardware components by individual tenants isn't what was promoted by NetApp initially. That ain't multitenancy. But you knew that, didn't you?

I'll share with you what's really needed in a bit ...

PART TWO -- RESOURCE ISOLATION

Dimitris suddenly claims "well, we've been doing QoS for years" when we have strong evidence to the contrary. I believe he either doesn't understand the question, or perhaps is stretching the truth.

So something in the form of tangible evidence would be very helpful. Supply documentation to the contrary, and I will be more than happy to back down publicly.

Let me know when you can get around to that, OK?

PART THREE -- VALIDATION

NetApp used VMware and Cisco technology in creating the SMT solution. That's about it. Ask for documentation on the "validation" process, and you'll get none.

Nice marketing dance, but I don't think EITHER company is going to come help you out on this particular issue.

VMware? Cisco? Bueller?

It's your hole you're digging, it's up to you when to stop digging.

PART FOUR -- CUSTOMER REQUIREMENTS

It's pretty easy.

Tenant gets unfettered access to allocated storage resources, including performance, without worrying about another shared tenant impacting service levels.

And, just because you like to wiggle, that would be multiple tenants on the same storage array.

Regarding security, that might mean that tenant also gets to encrypt all their data on a per VM basis, using their own key management software. Or, at the minimum, inspect and validate security and compliance processes being used by service providers if encryption isn't desired.

Now, if you can detail to me how you'd protect a tenant from information theft by an employee of the service provider (again, multitenant) I'm all ears. Just send me a bit of documentation, and I'll be more than happy to back down on this one as well.

THE BOTTOM LINE

C'mon guys -- admit it.

SMT ain't really all that secure, and it ain't really all that multi-tenant. I'm sure it started as a decent effort, but it now being way oversold by NetApp management.

Confession is good for the soul, guys.

-- Chuck

[Zeh]

You get angry that NetApp folks post comments here as if you were not posting this whole story just to say: "NetApp only has one product, don't trust them". Funny.

(Not a employee of any storage company).

[James Haefele]

Jousting..it's energizing to a point, and then it either needs the burden of proof and facts, or the joust is over :-). EMC'r here, and at one point , 2004-2005 i dabbled at HDS.

The important point to me in the history of EMC, is watching the effects when the change doesn't come at inflection points, or when companions do not learn from their mistakes to the detriment of their shareholders, employees, and yes - their customers too. I am not throwing stones here, but rather the experience of a past employee. In a short period of time i watched this at HDS - failure to capitalize on their AppIQ partnership and key reseller agreement, failure to capitalize and understand at the time what the NetApp partnership could translate too, failure to capitalize on that period in 2004-2005 that partially is reflected in Chucks blog here when Tagmastore and Thunder were fresh and EMC was redefining, and failure to capitalize on Alacritus as it was swooped from beneath them (of course we know where that story goes..)

I think back to a memo that had been framed by my boss at the first IT Department i worked in where (in the 80's) a prior IT Manager had wrote "LANs will not expand, the use for peer computing is short-lived".

Thanks for the post Chuck..it helps to continuously remind ourselves - and continue consulting vs telling..

[Dimitris Krekoukias]

Hi Chuck...

Go to netapp.com/library and look for flexshare. All documented Re QoS.

And yes, we can do resource isolation within volumes in the same aggregate.

For years now.

For free.

Is it a feature everyone uses? Probably not, just like thin provisioning or dedupe are not used by everyone. Just like CX's NQM is not used by everyone.

But it's undoubtedly there.

D

[W. Curtis Preston]

You work in marketing, right? How many products has the marketing department built lately? And I know you don't use any. And you do spend a good deal of your time writing blog posts criticizing your main competitor, including the one to which I replied.

So you're saying that you make a living criticizing what the other two are doing? Oh, wait, you meant ME!

Your reply to my comment was disrespectful, disingenuous, and pretty much amounted to a giant " you." Having said that, I directly addressed the point of your post. You then ignored what I say and then dismissed me because I'm nothing but a critic. Nice move. Way to elevate the discussion.

[Edward L. Haletky (@texiwill)]

Hello,

Since I was mentioned (@texiwill btw), I should make a comment or two. For a system to be 'Secure Multi Tenancy' you need to have several layers of protection. The problem is that until the solutions catch up with requirements, there is just no 'Security'. There are 3 aspects of security, Availability, Confidentiality, and Integrity. At the moment, Cisco pretty much admitted that CVN was a QoS play to ensure availability through the hardware to the storage device. Once the data gets to storage, yes encryption of data at rest is possible using Decru or an encrypting FC Switch. However, that is not necessarily the entire picture, it is just a small part of the picture.

In an agile environment, is Data Ever at Rest? Perhaps, but most likely not.

So where is the Integrity or Confidentiality? You rely on VMware products for both and they do not provide several very specific requirements. For one it does not disallow the cloud/virtualization administrator the ability to see, touch, or modify the data. In actuality, it is possible to even bypass all security logging to do just that. So if your 'Cloud' Administrator has been social engineered, or is just having a bad day, etc. The 'tenants' data could be stolen, modified, destroyed without anyone the wiser. One of the Virtualization Security Podcasts (http://www.virtualizationpractice.com/blog/?page_id=4852) also had Vaughn and others from NetApp on for a discussion of SMT. GestaltIT was briefed by Cisco, neither of them spoke about unified security. NetApp spoke about storage and the vFiler security which was interesting, Cisco about Networks and QoS, but where is the unified security discussion? It just has not happened from anyone.

Please review "Secure Mutli-Tenant Virtualization – How to get there?" (http://www.virtualizationpractice.com/blog/?p=5271) for another discussion on this. In essence, SMT currently requires you to trust your data with a third party whom you do not know, and cannot control. Any Administrator of a virtualization host can pretty much bypass any controls as Mandatory Access Controls do not exist. There are tools that close some of these gaps but nothing is closed 100% and these tools are NOT mentioned within the CVN writeup.

SMT security is ALL about securing the Data in motion and at rest, NOT about the individual bits that make up the solution. CVN's Solution just does not do this. It is a start at defining SMT, but misses on defining the crucial component, the Tenant. Those who need differing levels of QoS is not in my mind the definition of a Tenant. To me that is those who are 'LEGALLY' responsible for the data.

[Chuck Hollis]

Hi Dmitiris

I think you're referring to http://www.netapp.com/us/library/technical-reports/tr-3459.html from 2006? That can't be right, because this description of FlexShare doesn't do what you say it does.

Is there another document I should be looking at? Or should I spend the time to disassemble your assertions now?

-- Chuck

[Chuck Hollis]

W. Curtis Preston

Sorry, I guess you didn't see the humor in my response, and took it as an insult. It wasn't meant that way.

I work for EMC in a variety of capacities, as people who know me will attest to. Some of that is marketing-related, some is not. But I don't think you meant to turn this into a personal attack, did you?

Best regards ...

-- Chuck

[Chuck Hollis]

Folks, I decided to take some time and offer up a thorough response to W. Curtis Preston's thoughts around integration.

Please see http://chucksblog.emc.com/chucks_blog/2010/05/my-friday-rant.html

Thanks!

-- Chuck

[W. Curtis Preston]

Chuck,

This is not your first day on the Internet. If you had meant your comment as a joke, you should have indicated so. I therefore don't believe you meant it as a joke any more than I believe that this post wasn't aimed squarely at NetApp, even though you say in a follow-up post that it wasn't.

My follow-up comment wasn't a joke either, but I don't think it was a personal attack. I simply stated the fact that you build about as much technology as I do. If you can say, I can say it.

[Chuck Hollis]

W. Curtis Preston

So, I guess you're not big on apologies, forgiveness, tolerance, etc. That's unfortunate.

If we ever get together for beers, I'll be glad to bore you with stories of the technologies and offers I've helped to build -- whether directly or indirectly.

Not really relevant for this blog, though.

Cheers!

-- Chuck

[Storagezilla]

"You've come a long way since then, starting with (from my memory) acquiring a more moderately-priced disk array line that helped you to appeal to a different type of customer (Clarion). But then you had TWO products that did one thing: FC/SCSI. You needed more. Then you added the ability for them to do NAS and iSCSI."

This is incorrect. Symmetrix started as a Mainframe product to which Open Systems support was added and it was priced accordingly for the market it served at the time. Interest in NAS started a project which developed the Celerra gateway and to meet the price point of the mid and low end of the market Data General was acquired for Clariion.

Support for Celerra and Clariion as a unified system was then developed.

Today there are three block storage enterprise platforms. Clariion, Celerra and VMAX, One Unified Storage platform Celerra and one gateway Federation system VPLEX.

"Then, led by Mr. Tucci, you acquired four different backup software companies (Legato, Dantz, Avamar, & Mozy)"

Mozy is a service you don't buy any software. Indeed all the software in Mozy is in the backend and that’s running with Atmos.

"some backup hardware (Data Domain, as well as Quantum & Falconstor via OEM agmts)"

EMC bought one backup hardware company Data Domain. It licensed code from the other two and now licenses code from just one.

"some archive software (*xtender)"

The Xtender products were part of Legato.

"and hardware companies (Centerra)",

Centera was software not hardware.

"a document management company (Documentum), a server virtualization company (VMware), a security company (RSA), a home storage company (IOMega), and a bunch of others that are escaping my memory. So now you are a company with a bunch of different products that do a bunch of different things, most of which do not talk to each other very much."

Talk to each other very much?

Some of the intelligence from those products is embedded in those other products. Virtual Provisioning on Clariion came from Celerra. Compression for both came from RecoverPoint. RecoverPoint code runs in the Clariion FLARE Kernel and RecoverPoint algorithms optimise SRDF traffic on VMAX. Replication Manager and PowerSnap use the same core. The extender logic is embedded into Celerra. Celerra and Documentum can archive information into and out of Centera. Iomega was showing off integration for Atmos and Avamar at EMC World. The deduplication client libraries from Avamar are part of the NetWorker client. The BOOST plugin for NetWorker is from Data Domain. All the products above use RSA crypto code for security and most of them are integrated with RSA enVision.

I could go on for pages.

Literally.

Because EMC builds products from components and the components are reused across the portfolio. EMC doesn’t sell one box, it sell systems and software made of many reused software components.

Steve Todd has been blogging about this for a while now.

http://stevetodd.typepad.com/my_weblog/2010/05/of-components-and-services.html

"They did all of what I described above without acquiring a single product. (I'm not saying they never acquired anyone. I'm just saying they didn't have to acquire any products to do all of the above.)"

I'll defer to someone from NetApp to explain what they integrated into 7G from Spinnaker and there was non-cluster related technology from Spinnaker integrated. Alex McDonald said as much on Infosmack.

As for all this talk about comparing both companies I really couldn't care less.

If you have to define who you are by whom you're not, *you're no one*.

[W. Curtis Preston]

>This is incorrect. Symmetrix started as a Mainframe
>product to which Open Systems support was added and it was >priced accordingly for the market it served at the time.
>Interest in NAS started a project which developed the
>Celerra gateway and to meet the price point of the mid and
>low end of the market Data General was acquired for
>Clariion.

Honestly, I think we said the same thing in different words.

>Mozy is a service you don't buy any software. Indeed all
>the software in Mozy is in the backend and that’s running
>with Atmos.

Not really sure the point you're trying to make there. Mozy is a SOFTWARE AS A SERVICE company. If I don't load the Mozy client on my PC, it can't do anything.

"some backup hardware (Data Domain, as well as Quantum & Falconstor via OEM agmts)"

>EMC bought one backup hardware company Data Domain. It
>licensed code from the other two and now licenses code
>from just one.

Which is what I tried to say, just poorly.

>The Xtender products were part of Legato.

I was just trying to segregate backup and archive.

>Centera was software not hardware.

Again, we all see it as hardware. AFAIK, I can't buy Centera software. I buy a Centera. To me, that makes it a hardware product. But I still don't see your point.

>Some of the intelligence from those products is embedded
>in those other products.

I'll concede that I under-stated the degree to which EMC products talk to each other, and definitely concede the fact that they share code. But I was thinking more about where they DON'T talk to each other. I could also go on for pages, but I actually have other work to do today.

"They did all of what I described above without acquiring a single product. (I'm not saying they never acquired anyone. I'm just saying they didn't have to acquire any products to do all of the above.)"

>I'll defer to someone from NetApp to explain what they
>integrated into 7G from Spinnaker and there was
>non-cluster related technology from Spinnaker integrated.
>Alex McDonald said as much on Infosmack.

None of the functionality I mentioned came from Spinnaker.

>If you have to define who you are by whom you're not,
>*you're no one*.

Not sure what that has to do with the discussion at hand, but OK.

I don't like comparing the companies either. And I like even less commenting on one vendor's blog about another vendor. But when a blog from one vendor unfairly bashes another vendor, I feel obliged to respond. Today, it was Chuck.

[Chuck Hollis]

If anyone is interested in a *serious* discussion of secure multitenancy, I've moved the discussion over to here:

http://chucksblog.emc.com/service_provider_insider/2010/05/towards-a-serious-discussion-on-secure-multitenancy.html

-- Chuck

[Storagezilla]

"Honestly, I think we said the same thing in different words."

You got the order in which things came to market wrong. Having an accurate timeline matters as it shows how the products developed. What I listed was the order in which things happened.

"Not really sure the point you're trying to make there. Mozy is a SOFTWARE AS A SERVICE company. If I don't load the Mozy client on my PC, it can't do anything."

You're not subscribing to the software you're paying to keep your data on Mozy's servers. It's not Software as a Service. Salesforce.com is software as a service. Mozy is storage as a service with a client transmitting the data. You don't even have to pay for your first 2GBs.

"Again, we all see it as hardware. AFAIK, I can't buy Centera software. I buy a Centera. To me, that makes it a hardware product. But I still don't see your point."

The point was Centera is software and was acquired as software. EMC created the packaged appliance after the fact but the core is software on x86 processors and always has been.

"But I was thinking more about where they DON'T talk to each other. I could also go on for pages, but I actually have other work to do today."

I don't think you could show me any vendor with more than one product which doesn't have use cases where products don't speak to one another. The implication that isn't any integration is demonstrably wrong with very few of the products not using common component technology right now.

"None of the functionality I mentioned came from Spinnaker."

I'd prefer if someone involved with development said that as I don't think it's accurate. There's been work put in there for six going on seven years but if the people involved say there was no contribution then there was no contribution.

[W. Curtis Preston]

>You're not subscribing to the software you're paying to
>keep your data on Mozy's servers. It's not Software as a
>Service.

Potato/potato, snapshots/near-CDP ;)

What I said was that you bought software, and I maintain that you did.

>The point was Centera is software and was acquired as
>software.

OK.

>I don't think you could show me any vendor with more than
>one product which doesn't have use cases where products
>don't speak to one another.

Agreed. Which is why I was comparing it to a company that did most of what they did by NOT acquiring companies.

>"None of the functionality I mentioned came from
>Spinnaker."

>I'd prefer if someone involved with development said that
>as I don't think it's accurate. There's been work put in
>there for six going on seven years but if the people
>involved say there was no contribution then there was no
>contribution.

With the exception of open-systems snapvault, every piece of functionality I mentioned existed well before the acquisition of Spinnaker. No expert needed.

[Dane]

Howdy everyone... can't believe I'm actually going to throw in with all of this, but where's the fun in not doing that? First off, I'm a storage customer, I don't sell anything; only buy it.

Secondly, Chuck, great story, I honestly enjoyed and learning more about the company we currently have storage arrays with. (your DMX product line, I love and is a tank that won't stop) and actually love that you're having such a candid discussion with competitors and everything.

Now to my actual points. These are also just my opinions as i see trends, ideas, technologies, & Have to support storage, so don't beat me up to much.. :-):

Yes, history repeats itself. Bluntly honest, I feel like i can see your own history repeating itself with you (and honestly NetApp, you're both getting to big and are loosing contact with your customers). Few reasons why:

1)I feel like (with/in storage & some other EMC products) EMCs method is to just force their technology down customers throats instead of going out to seeing what the customer wants. I think the fast technology released with the V-Max is great, but I don't see customers & general IT wanting to go that method with that technology. I see a stronger attraction toward storage virtualization & cloud type arrays.
-Virtualization allows the customer (me) to uncouple from a specific array vendor (EMC/NetApp/3Par/IBM .. ETC) causing an actual competition between vendors, driving costs lower and making you guys actually compete for our money (YAY!!!). Specifically, x86 node-based Virtualization (IBM SVC, DataCore SANsymphony, & LSI SVM) environments instead of array-based front-ends. (NetApp & Hitachi HDS)
-"Cloud-Arrays" is more based on self-healing slower, larger ATA drives, but with it spread across so many spindles, it can still handle the I/Ops of those faster and much more expensive disks (and has a lower cost). I feel like this self healing cloud type storage is the way storage arrays will eventually be going (Examples: IBM XIV, XioTech Emprise/ISE, Compellent, etc...) that direction instead of manually built arrays of 7+1 R5s & 14 R10s n so forth.

-I don't see EMC (or NetApp for that matter, in all honesty so could be mistaken though) going either of those routes (I just listened to our Technical Pre-Sales guy lay-down what he can of the EMC futures road-map. What I do see is two large companies trying to force their own blend of storage to customers instead of listening to what the customers want. (I experience that a LOT with vendors.. can't tell you how many times i just want a nitty gritty tech-talk from a vendor but instead i get a marketing-sales dog n pony show... end 'rant') I believe not listening to your customers is a HUGE detriment... which hopefully you guys have learned in the past (from your story... which i think its great you're doing this blog)

2) I have to agree with John F. (commenting about EMC's habit of buying up product after product be not really integrating said technologies). I've actually had conversations with coworkers & issues with this exact practice.

-Conversations: EMC is getting some great products, but it feels like after EMC buys the product, they do an initial ingestion/release back to the wild & than no longer document or further or integrate the technologies to work together.
-Issues: we've had multiple EMC products that just WON'T work together. Honestly, it's 4:30 on a Friday * i can't think of the exact examples, but trust me, it happens a LOT.. and i don't really want to point out to many negatives... i don't like when people trash-talk... i vote let people get/make an opinion themselves.

Summarize this; You guys have some great products, i have no problem with the buying up stuff, but make sure as new stuff is released, that you integrate that old stuff with it. I could see a LOT of great potential in the technology EMC already owns (and you guys are definitely NOT alone in not integrating products... i can't think of a single company that DOESN"T have internal compatibility options, it happens, we know this.. :-D)

Ok, i've ranted enough and need to head home.
Good stuff here, i like seeing these candid discussions!
Enjoy your memorial day weekend everyone!!

[Tony Pearson]

Hi Chuck,
TonyP here (IBM employee). Great post! Sadly, history does repeat itself, seemingly over and over. Most readers are probably too young, but IBM went through similar series of circumstances in the 1980s. Paul Carroll wrote a book in 1994 about it called "Big Blues: The Unmaking of IBM". Required reading at IBM, since you can only avoid repeating history if you know it happened in the first place to learn from it.

http://www.amazon.com/Big-Blues-Unmaking-Paul-Carroll/dp/0517882213

Enjoy the weekend!
-- Tony Pearson (IBM)

[Chuck Hollis]

Dane

You have great comments to add, so don't be shy please ...

First, I'd love to understand more about your environment. Generically speaking, what you're asking for is a scale-out architecture, which usually implies a very specific set of workloads, which is why I'm interested in what you're doing.

Lots of work going on in the storage world along the lines you suggest -- some by EMC, some by others -- and it's definitely an emergent area that everyone is focusing on.

As far as "not integrating", understand you're tired, etc. but some examples to go look at would be helpful. Tell us where we suck, we can usually make it better. But it helps if you tell us :-)

Thanks for taking the time to rant ...

-- Chuck

[Storagezilla]

"With the exception of open-systems snapvault, every piece of functionality I mentioned existed well before the acquisition of Spinnaker. No expert needed."

And OSSV as you brought it up contains technology from BakBone. BakBone says so in it's 10Q filings.

http://google.brand.edgar-online.com/EFX_dll/EDGARpro.dll?FetchFilingHTML1?ID=5619917&SessionID=SYs-HjneTZeIhX7

As for Spinnaker I don't compare systems developed ten years ago to systems in production today and you shouldn't either. There is Spinnaker code in today's systems touching a lot of what you mentioned.

[John F.]

:Just a reminder -- the rules of the road are clear: please disclose your affiliation if you work for a vendor.

Not only do I demand it for this blog, it's generally good practice to be transparent about your affiliations.

Especially if your goal is to throw mud -- John F, I'm looking at you!

(by the way: YANM == Yet Another Netapp Mudslinger)

-- Chuck"

Oh, Chuck...

Move your mouse two inches. Click the link on my name, I not only disclose where I have worked the past five years, but my real name, many of the other places I've worked, and as an added bonus one of my other affiliations.

What is your problem? Does everyone else who has responded to your post provide this same level of detail? Is it too much information for you?

J

[Chuck Hollis]

Hi John F

I guess you're new around here.

Historically, I've had a problem with individuals with vendor-slanted points of view either (a) not represent their affiliations, or worse (b) mis-represent their affiliations.

So, a while back I started a "disclose your affiliations" policy if you're going to offer any commentary in support of one particular vendor position or another.

Dmitris, for example, started to work for NetApp a while back, and he immediately went on the attack as a result.

It took us a while to get him to finally admit that -- yes -- he was now a paid employee of NetApp.

We also have clear examples of people posing as customers, but strangely having NetApp's IP address. I guess they were just visiting.

For some reason, it seems to be mostly a NetApp problem -- very few problems with people who work for other vendors. I don't know why that is. Feel free to scroll through the 2000+ comments on this blog, and you'll see what I mean.

Also, as long as we're talking about rules, no personal attacks, keep your language and conduct professional, nothing too seriously off-topic, and please don't write commentary under the influence of alcohol or other drugs -- it isn't pretty.

All of these have caused problems in the past.

If you don't like these rules you're always free to post your commentary elsewhere. My blog, my rules.

Cheers!

-- Chuck

[Zeh]

Concentrate on showing what's good about your products. Stop bashing it other like high school girls. Customers are not dumb, they can think for themselves.

What have storage marketing turned into ? This is worse than gossip magazines.

Show me your technological advantages and I will think and vote with my wallet. Make me waste time filtering the noise and I'll vote elsewhere.

(not employed by any storage company. actual user pissed at vendors messing with customers by manipulation)

[Chuck Hollis]

Zeh

Good advice to all -- thanks!

-- Chuck

[John F.]

"Hi John F

I guess you're new around here."

Now that's a downright wierd response that skates around the question. Although my reply that triggered this angst was admittedly laden with sarcasm, the point I made rang true with many (unless you have legitimate reason to believe otherwise) customers that also responded. Is your paranoia justified? I have no clue. I'll leave that up to you to explain. I said what I said coming from me and my personal experiance over an entire career. It had absolutely nothing to do with who I currently work for, or who I have worked for in the past (which I fully disclosed in the link on my name).

I think your response says volumes about the EMC mindset.

J

[Chuck Hollis]

Hi John F.

Your responses are progressively getting stranger and stranger.

I've tried to explain the rationale for requesting that vendor employees identify themselves when commenting here.

I've also made it clear that if you don't agree, you're free to leave your commentary elsewhere.

Strangely, you've decided to come back with yet another attack.

As a result, I'll ask you to please go elsewhere to do your venting, otherwise I'll start simply deleting your offending comments.

-- Chuck

[John F.]

(comment deleted by blog owner)

[Chuck Hollis]

John F

I had to delete the last one, because you were way over the line.

Ranting personal insults aren't in the spirit of this blog (as I've made repeatedly clear), so your comment goes away.

Should you reconsider, and wish to converse in a professional and mature manner, you're always welcome.

-- Chuck