Odds and Ends

Didn't get a chance to blog too much this week, as I was mostly on airplanes. 

Things are busy, to be sure.

Anyway, just wanted to share a few things that didn't really merit their own blog post.

Virtual Geek Shares The Goldman Sachs Report

Chad writes a great blog.  He doesn't post too often, but when he does his posts are chock-full of great reading, often three or four posts' worth of material at a single sitting.

His most recent posts are no exception.  In addition to excellent VDI performance information, he previews most (but not all!) of EMC's various activities at VMworld.  And, oh by the way, lists dozens of new EMC proven solutions around specific VMware use cases.

And, despite trying desperately to remain above the fray, he too finds himself having to smack down the blogging team at NetApp for poor sportsmanship.

I was going to write a post on the GS tech trends report, but he beat me to it.  For those of you who may not be familiar with this particular report, it's the defacto standard on tech spending trends.

Personally, I've followed it for at least 10 years or so.  Rarely, if ever, does it get things wrong.

Chad duly notes that VMware comes out #1 in terms of software tech spending intent in the survey panel (Fortune 1000 type companies), and also duly notes that EMC is far and away the preferred storage behind those VMware implementations.

But what he didn't share was the overt preference of SAN vs. NAS in these environments, which I found interesting.

Personally, I've been in the protocol-neutral camp on this for several years, but it looks like this audience is now strongly going SAN (presumably FC) for their growing VMware farms.

Last year, it was 43% SAN vs. 29% NAS.

This is was 46% SAN vs. 25% NAS.

And this same panel is saying that in twelve months it's going to be 59% SAN vs. 24% NAS.

So, what does this mean?

First, it seems that customers are thinking about the same heavy-duty storage infrastructure for their virtual apps as they are for their physical applications.  This means that more and more companies are moving well beyond the test-and-dev mindset, and thinking serious workloads.

Second, if you look at the vendor scorecard, the shift is very noticeable towards serious SAN vendors, and not the NAS-oriented vendors.

I guess I have to revise my protocol-neutral perspective -- the data is in, it looks like it's mostly going FC SAN in larger enterprises.

HP Does SPC-2

Not a month or two seems to go by without some vendor running an SPC test, bragging about the results, and challenging EMC to a showdown.

And, just as predictably, we go look at what they've done, shake our heads, and further resolve never to get involved in any of this nonsense.

This latest one from HP seems to be no exception.  In case you're not familiar, the SPC-2 is a sequential I/O test, much like you'd see in large file serving, video streaming and the like.

First, if you had such a requirement, I bet you wouldn't be choosing something like the USP-V (er, XP24000) for this task.  Most people would think in terms of clustered, scale-out file systems, such as HP's own NAS thingie just announced, but I digress.

Nope, I popped open the test report, and it took me about 60 seconds to spot the trick: take an high-end enterprise array designed to support over 1000 drives, populate it with 128 mirrored pairs of 146 GB 15K drives evenly spread across the controller hardware, and run your sequential I/O benchmark.

I should hope it turned in a good score.  But I don't think anyone is going to spend the $1.6m for this nifty benchmark special.

Interestingly enough, EMC's DMX seems to be gaining strong share in the high-end marketplace over the last few quarters at the expense of the HDS and its OEM variants.  Of course, we don't usually end up recommending a DMX as a video streamer, but -- hey, who knows? -- maybe there's another market there we haven't thought about ... :-)

The whole thing strikes me as a complete waste of everyone's  time.

And -- The Final Word On The Capacity Brouhaha?

If you're a regular reader of this blog, you remember the ruckus I caused a few weeks back with a "usable capacity" comparison between EMC's CX4, HP's EVA and NetApp's FAS.

It was a simple enough idea: configure 120 usable 146 GB drives using vendor-supplier recommendations for an Exchange environment.

Oh boy.

As I said before, I think we got things wrong on the HP EVA.  We configured 7 disk groups, HP recommended 2.  The rules of the game were clear: go with what the vendor recommended.  So we were wrong in that regard.

Even with the changes, I think EMC still has a usable capacity advantage, but it's not as pronounced.

I also expressed my concern about small numbers of disk groups leading to challenges with performance isolation and availability isolation, but was largely shouted down by the HP bloggers.

Until I got this comment:
------------------------
Chuck

Coming from an EVA environment I can say that your assessment is good overall.

Although HP does recommend fewer storage groups, in practicality this proves problematic. I had storage groups for FC disk and FATA, which each placed a large number of spindles in each. If ALL the array did was backend exchange it would have been fine but in a mixed use scenario, as was our case, we found this recommendation troublesome.

With singular large storage groups and several multi-purpose hosts attached you now have the potential for a single host to affect the performance of the entire storage group.

We had numerous *whole* array outages with no clear cause, but tracing through events in the time line pointed to probable suspects.

To avoid this you would *have* to break out the SGs as Chuck indicated, but if performance and reliability aren't a concern for you..
---------------

However, this sort of characterization was above and beyond what I initially set out to discuss, so we'll save that for a later day.

In the final analysis, HP was reasonably polite about pointing out my error, and I think I was reasonably polite in my response.

The NetApp blogger reaction was another story entirely.

There was no argument about the losses due to "right sizing" the drives.  Nor was there arguments regarding the minimum 10% "reserve" for WAFL performance, nor its overhead for its data structure.

The entire argument focused on whether or not NetApp needed 100% reserve for snaps in an Exchange environment, or not.  I maintained that every piece of documentation I could find pointed to a clear NetApp recommendation for a 100% snap reserve in Exchange environments.

The NetApp bloggers insisted the documentation was old, or I wasn't reading it right, or that it was really the user's choice, or -- well, I can't remember all the different reasons they gave me as to why I was just plain wrong.

Interestingly enough, I started to receive a quiet trickle of email and comments from NetApp users who said, in effect, I was right, and the NetApp bloggers are wrong.

Here's an example of one comment that appeared towards the end of the discussion:
---------

Chuck

I've read all the blog entries and comments and it does seem that a hornets nest has indeed been opened. I think open debate is important so thanks for being game enough to raise an interesting topic - and for taking the flack that it produced.

However, not one to be swayed by Vendor kool-aid, I really wanted to do my own research and I have to say, I'm not surprised there is a level of confusion here.

Firstly, while the EMC Best Practice on Exchange does say that RAID 5 and RAID 10 are acceptable - it also implies a 4+1 implementation. However, it doesn't rule out 8+1 so I guess that's fair.

What is more surprising is how Val can state that 100% Fractional Space Reservation is not a recommendation from NetApp. I agree with you - I can't find a single "Customer Facing" document from NetApp that concurs with his point of view. In fact, if you look at their own ESRP Documentation,

http://media.netapp.com/documents/NetAppFAS3170ESRPStorageSolution.pdf 

the Total Formatted Capacity  on page 10 would validate your findings.

HP also think that there is something fishy with using NetApp for Block I/O.

http://h71028.www7.hp.com/ERC/downloads/4AA1-4386ENW.pdf

Despite the heat you have taken, I think the weight of evidence is in your favour on this point and if Val truly believes that Fractional Space Reservation isn't required then NetApp Best Practice documents should reflect this - otherwise Val's comments lack any level of credibility.

Thanks for the blog.

---------

OK, maybe I wasn't on drugs ...

But there's more -- check out this thoroughly researched missive that showed up in my email -- I guess this person didn't want to get bullied by the NetApp bloggers ..
---------

Chuck,

I’d been following the discussion you started on usable capacity, and wanted to pass on some bits of information I thought you should know.

To start off with, I’m a NetApp and an EMC customer.  We have a good amount of experience with both platforms.  My personal opinion is that both vendors have areas where they excel, but I’m putting my personal views aside to be as impartial as humanly possible.

My understanding of your capacity post <http://chucksblog.typepad.com/chucks_blog/2008/08/your-storage-mi.html>  was that the scenario was Exchange, 146G drives, vendor published docs, and 120 or so drives.  I never saw any reference to performance; this was really just a useable capacity exercise.

Some of the comments posted to your blog almost seemed to try to mislead the reader from the details of the exercise.

LUNs on a NetApp do not run out of space

·        See sections under space reservation, what can happen…, automatic deletion, and attention.

4 years ago 100% reservation was recommended, now customers can simply enable autodelete.

·        The Space reservation sections says that SnapDrive creates LUNs with 100% space reservation

·        The Automatic deletion section says that due to Exchange-aware features, the automatic deletion of snapshots does not necessarily prevent an out of space condition on the volume.

Claiming that that something dramatic happens when NetApp FAS arrays run out of LUN reservations is simply FALSE.

·        Under Attention it states:  If a storage system volume runs out of overwrite reserve space, write operations to a LUN on that volume will fail and the host application may terminate, report I/O errors, or exhibit unexpected behavior.

The reference made to various safe options for setting reservations to zero.

·        The Thin provisioning paper http://media.netapp.com/documents/tr-3483.pdf states the following about thin provisioning of Snapshot space. If Snapshot copies are being used, having available less actual disk space than the total size of the LUNs will rarely, if ever, be practical.

·        The same document also states that “Given the previous explanation of how LUNs will typically fill up close to 100%, strict thin provisioning of LUNs often will not provide significant savings for an extended period of time.”

So with all of this information, I find it mind-boggling that some people claim that you are citing outdated information.  Everything referenced here is current as per NetApp’s site http://now.netapp.com.

While I agree with the statement that space reservation is not required to be 100% for everything; it would certainly appear that where Exchange is concerned that 100% is the standard.  The last sentence below I think sums it up.  It’s direct from the SnapDrive 6.0 Admin Guide

·        The storage system volume hosting the LUN should be at least twice the combined size of all the LUNs on the volume.

In closing, I just want to say that I’m not taking sides.  It’s just really frustrating when a potentially valuable topic for customers gets going, and one side resorts to mudslinging and name calling when they may not come out on top. 

Chuck, you should hold your head up.  Except for over-configuring hot spares by 4 drives, which in the end won’t alter the percentage very much if at all, it looks as though the NetApp numbers were right on.

Here is the documentation I used:

SnapManager for Exchange 5.0 Admin Guide

http://now.netapp.com/NOW/knowledge/docs/SnapManager/relsme50/html/software/admin/cfgapp11.htm

Space reservation

SnapDrive creates and manages LUNs with space reservation enabled. That is, additional space on the storage system volume is automatically reserved for overwriting blocks that belong to a LUN. By default this additional space is equal to 100 percent of the total size of all space-reserved LUNs in the storage system volume. If space reservation is disabled, write operations to a LUN might fail due to insufficient disk space in the storage system volume and the host application may terminate, report I/O errors, or experience unexpected behavior.

Fractional space reservation

With fractional reserve, the amount of space reserved for overwrites is set to less than 100 percent of the total size of all space-reserved LUNs in a traditional volume or a flexible volume that has the guarantee option set to volume rather than file. The space that is preallocated for space reservation is reduced to that percentage. Fractional reserve is generally used for volumes with LUNs that store data with a low rate of change.

What can happen with a fractional-space-reserved volume

When a LUN is fully space reserved, write operations to that LUN are guaranteed against failure caused by an out-of-space condition due to Snapshot copy disk space consumption. When the overwrite reserve for a volume is set to less than 100 percent, however, write operations to the LUNs on that volume are no longer guaranteed when the storage system volume runs low in free disk space due to Snapshot copy space consumption.

Attention

If a storage system volume runs out of overwrite reserve space, write operations to a LUN on that volume will fail and the host application may terminate, report I/O errors, or exhibit unexpected behavior.

Automatic deletion of Exchange backups

Select the backup retention level based on your SnapManager backup creation and verification schedule. If Snapshot copy deletion triggers, enough backup Snapshot copies should be retained so that at least one verified backup remains on the volume. Due to these Exchange-aware features, the automatic deletion of Snapshot copies does not necessarily prevent an out-of-space condition on the volume.

SnapDrive 6.0 Admin Guide

http://now.netapp.com/NOW/knowledge/docs/snapdrive/relsnap60/html/software/admin/provisioning/concept/c_sdw_prov_recs-for-use.html

Follow these recommendations whenever you use SnapDrive for Windows.

·        Use SnapDrive to create and manage all the LUNs on your storage system.

·        Never disable the space reservation setting for any LUN managed by SnapDrive.

·        If you want to dedicate all free space on a volume to LUNs, do set the snap reserve setting on the storage system to 0 percent. In this case, Snapshot copy creation is not guaranteed.

·        Place all LUNs connected to the same host on a dedicated volume accessible by just that host.

·        Unless you can be sure that name resolution publishes only the storage system interface you intend, configure each network interface by IP address, rather than by name.

·        If you use Snapshot copies, you cannot use the entire space on a storage system volume to store your LUN.

The storage system volume hosting the LUN should be at least twice the combined size of all the LUNs on the volume.

--------

Wow. 

All I can say is that there are people out there who really, really do their homework. 

W. Curtis Preston, did you follow all of this?  :)

And finally, a comment from Martin G who's noticed this poor utilization, even without snaps ...
---
Chuck

Just a quick post, I recently had some sizing done which was for pure capacity and how many spindles I would need to give me 400 Terabytes of useable space (Base 2 Terabytes) on a FAS. To give me 400 Terabytes of useable disk using 1 Terabyte drives would take 840 spindles made up of

689 data spindles
116 parity spindles
29 hot spares

coming to 834 spindles

which was 60 shelves of 14 disks.

This sizing was done by the NetApp account team. It was a pure capacity exercise and takes no account of the underlying performance requirements of the application, no account of Snapshots, no account of dedupe but simply how much data can I get on an array.

I wonder if the other vendors following this could carry out a similar exercise. 400TB of data on 1 Terabyte drives and I have an availability requirement of five nines (so if you think RAID-5 on 1 Terabyte drives is going to cut it...I might look askance).
-----------
So, even without snaps in the game, we're looking at less than 50% storage capacity efficiency from this NetApp user.

And A Note On Conduct

After a while, the NetApp bloggers decided to make it very ugly.  One of their bloggers decided to write a whole post trashing me personally (see here), and -- if you read further into the comments, decided to go even further take a swipe at EMC's morality and ethics.

I also had to take the unfortunate step of banning some of this individual's more choice comments from my blog for the time being.  My apologies to those of you who felt that this was heavy-handed on my part.  

I hated doing that.  I like the openness of this forum.

If you read through the comments, you'll see that I generally allow all sorts of viewpoints here, even those that are a bit off-the-wall. 

But when it gets very mean, and very personal, I'm going to draw a line, and send a clear message as to what's acceptable and what's not.

So, from now on, you'll see at the bottom of every one of my posts a line I stole from Robin Harris, over at StorageMojo:

"Courteous comments always welcome, of course ..."

Comments

[Lee Razo]

Hi Chuck,

NetApp guy here - courteous comment to follow:

I think at the end of the day, the whole NetApp discussion is going to amount to one about documentation. While I agree that that is an important topic, if that and a couple of specifically selected emails from anonymous senders is all you got against us, well shoot - I don't see what all the fuss is about either.

--Lee

[iStorage]

I've been using NetApp filers in both iSCSI and FCP (SAN) modes for a few years now. Initially we had to use 100% space reservations, but with the recent releases of DataONTAP, SnapDrive & SnapManager we've easily been able to scale that down to 0%.

However, I see NetApp space efficiency in an entirely different way. It's not how much water I can fill in the bucket, it's how many usable buckets I can get by buying just one.

In that regard, Curtis Preston is bang on. With NetApp, I can use 10's of buckets and still only buy one. With other arrays like CX or EVA, I'm still trying to make the most out of just the one bucket.

Regarding "conduct", I say you reap what you sow. If you call someone out in your blog (and in your comments you did just that Chuck), then expect a response and be man enough to live with it.

I still think filtering comments is weak. I would let them fly and have the public measure for themselves whether the commenter is enhancing or diminishing their own credibility by their own words.

[Dave McDonald]

Chuck,

I know you want to vindicate your position about NetApp's space efficiency against the opinions of an independent expert and their CTO, but this whole exercise is futile.

Reagrdless what you can produce via private emails and chosen passages from their texts, who are you trying to convince?

If I want to understand NetApp, I'll go ask them about their functionality, not EMC. If I need to double-check, I'll go to a 3rd party, not EMC.

There's no way for you to settle this, so why do you persist?

[Chuck Hollis]

You're absolutely right -- there is no way to settle this, and -- to be sure -- that is not my goal.

However, I found many of the responses interesting in context, and wanted to share.

Just for the record, the individual in question is not NetApp's CTO, he seems to use a variety of titles in different situations. And the "independent expert" in this case was the first to admit he had limited expertise in this area.

I think someone else here said something to the effect of: if you had a choice of listening to vendors, experts or users -- choose the users -- which I did.

[Chuck Hollis]

Thanks for commenting, iStorage

Please forgive my cynicism, but your comment sounds like it might have come from a NetApp employee.

Not that this would be the first time; those of us who blog at EMC routinely field comments from NetApp IP addresses pretending to be "users".

If this is not the case with you, my apologies in advance.

I'm curious though -- what types of application might you be running on a mix of both iSCSI and FCP? Usually it's one or the other in a given shop, and very rarely both.

Thanks!

[Alex McDonald]

Chuck

Martin G's requirement of 400TB (base 2) is fulfilled with usable capacity of *greater than* 50% of the drives -- not less than 50% as you state.

1. 400TB in base 2 is 429.5TB in base 10
2. 1TB drives (which are always quoted in base 10) have less than 1TB addressable bytes; around 977GB (base 10)

That's approximately 440 drives worth of data, *not* 400 drives. A 10% difference is significant.

I will be blogging this coming week on NetApp figures collected from user data. Much more accurate and verifiable, and much more preferable over anecdotal and erroneously calculated numbers.

[Chuck Hollis]

Hi Alex

Yes, that will be interesting to see how NetApp positions this with "official" user data, and not run the risk of being accused of gaming the results by picking selected data points, rather than representative averages.

My take?

I think you guys know you've got a problem with this issue. Part of it is attributable to your design, part of it is attributable to the recommendations you give your customers.

If nothing else, I bet this is an active discussion within your engineering community now ...

Cheers!

-- Chuck

[Alex McDonald]

Chuck

The data will be drawn from the same set of data used in our research papers; for example http://www.ssrc.ucsc.edu/Papers/leung-usenix08.pdf higlighted by Robin Harris over at StorageMojo last week (http://storagemojo.com/2008/09/09/our-changing-file-workloads/)

Unlike the authors of that paper, I am not a researcher and I am not a statistician, so my work based on the same data (and it will be *my* work) won't have the professional polish or peer review of an academic paper.

But neither am I a liar.

I resent the suggestion that I would "game the results". You have been quick to point out the unacceptable nature of what your perceive to be attacks on you personally. Given that, I would suggest that you need to consider what you have said here very carefully.

I look forward to your comments on my work in the coming weeks. Courteous ones.

[Chuck Hollis]

Alex

You need to go carefully re-read what I wrote.

I said that you might be accused of gaming the results by others.

This is the same problem that any vendor faces when presenting "independent" research.

It is not limited to NetApp, and it is not a personal attack.

Good luck!

[Martin G]

Alex,
sorry you are right and it was me who misled Chuck; in my defence, I would say that the calculation that I got from *NetApp* was not especially clear. I've gone back and checked it.

The calculation actually left out the 10% Snap reserve, so yes I would get 440 spindles. So apologies all round! I had asked for a quote for 400 Terabytes useable, unfortunately I got a quote for 400 Terabytes+10% snap-reserve.

Now, just to upset the apple-cart and in fair disclosure; our Exchange 2003 environment currently sits on DMX-3 146 Gig drives in a RAID-1 format. Why, because Microsoft and EMC told us so. Throw the protected BCVs into the mix and our utilisation in this environment is terrible!!! It drives me to distraction that Microsoft could have written an application which handles i/o so terribly!!! The sooner our Mail guys go to Exchange 2007, the better!!

[Steve Klinkner]

Chuck said

"If nothing else, I bet this is an active discussion within your engineering community now ..."

There's some truth to that at the surface level.

However the discussions are focused on correcting mis-representations in public forums rather than on product deficiencies.

From a NetApp engineer's perspective: the follow-ups have been enlightening, but with all due courtesy the root cause (marketing on a soap box) is *technically* non-interesting.

[W. Curtis Preston]

Yes, I followed all that, although the arguments are starting to lose my interest.

First, let me say that I think you have a point, although I don't think it's the point you're making. NetApp's docs seem to be pretty solid on 100% snap reserve. NetApp customers that I've worked with seem happy with something significantly less than that. (Which is why I don't agree with the point you are trying to make, that you HAVE to have 100%.) If NetApp is OK with less than 100%, why is it not documented? We'll see what changes come in the docs.

I still disagree that this means that EMC arrays are more space-efficient than NetApp arrays. I believe that how people USE their NetApp arrays takes up more disk than how people USE their EMC arrays. That's not the same thing.

A few issues...

The user you're quoting says they're a NetApp customer, but they don't mention anything from their own experience. All they do is quote NetApp manuals, which is the same thing you're doing. His/her comment would hold a lot more weight for me if they spoke from their experience, not from the docs.

You say you've never launched a personal attack, then you put "industry expert" in quotes and imply that Val is making up his title. How are those not personal attacks?

As it says on snia.org and communities.netapp.com, Val is in the Office of the CTO. He is in charge of competitive positioning. If that translates into CTO-at-large, so be it. But to imply that he's making up his title is a personal attack and is unfair.

As to putting "industry expert" in quotes, the only reason for doing that is to imply that I'm not an expert. Since my entire livelihood comes from BEING an expert, I consider that a personal attack. See? It's all about the vantage point.

If 15 years of doing nothing but backup and recovery using just about every product imaginable, three books, and a website with thousands of registered users doesn't qualify me as an expert, I'm not sure what does.

As to Val's blog entry that you considered a personal attack, EMC bloggers bring it on themselves by censoring comments. Uncensored blog comments allows the reader to determine who is full of it and who is not. Censored blog comments make the blogger look like they're not fighting fair. Google "censoring blog comments" to see what the blogosphere thinks about it. It's always associated with a lack of ethics. If you don't want to be accused of it, then don't do it.

[iStorage]

Hi Chuck,

I'll forgive your unfortunate "un-courteous" accusation that a former EMC customer must somehow be a NetApp employee impersonating someone else ... and I'll leave a you "courteous" comment instead so perhaps you can follow what you preach.

The use-case for a truly unified storage system (such as NetApp's) is an app development and VM certification environment. Our developers or VM infrastructure qualifiers are connected over iSCSI for low cost and flexibility.

When their apps and/or VM's are finished a testing or certification cycle (as the case may be) we promote them over to FCP (NetApp's name for FC-SAN). For small apps we do this in place with no LUN migration to get to production, and for larger apps we (snap)mirror and (flex)clone to a production filer over FCP for fully isolated and predictable performance.

We tried this with both Celerra NS systems as well as CLARiiON CX3's and it was far too cumbersome to work. I haven't seen anything in FLARE28 on CX4's to indicate this kind of dual block protocol integration has improved, but perhaps you can enlighten me?

[Chuck Hollis]

Curtis, I appreciate your comments.

However, I don't believe in crowd-sourcing personal ethics and values.

I was raised to expect certain standards of behavior and conduct from civilized people, and to express my disapproval when their interpersonal conduct doesn't meet my standards.

Call me an elitist (or any other name you choose), unless people draw lines, all of our values tend to degrade to zero.

I'll allow that I've made a snide comment or two on our NetApp friend's excessive posturing on titles and roles, but I think that the multiple blog posts from him and others that essentially attempt to collectively trash us as individuals -- and our company as whole -- well, I don't think I have to endorse that sort of conduct.

And, frankly, you shouldn't either.

I'm not the only one who's experiencing this -- if you're interested, go watch the interaction on virtualgeek.typepad.com

Different players, same pattern.

It's ugly, and none of us should condone it.

I don't know if you've ever had someone cross the line on your blog, but -- at some point -- you may feel the need to take a stand and say "enough".

And, if you ever encounter this sutations as I have, I for one will understand your decision to do so.

[Chuck Hollis]

My apologies, iStorage -- we've got a ton of NetApp employees trying to spoof us six ways to Sunday these days, so please accept my very humble apologies.

Now that you describe your use case, yeah, I get it, it makes much more sense why you use NAS, iSCSI and FCP all at the same time -- you're developing software and need to qualify in a variety of environments, and -- frankly -- the NetApp device would make a great test bed for your use case, perhaps better than what EMC could offer in this case.

But, you'll have to admit, your case is a bit different than most production environments, yes?

And, not to offend, but I would tend to guess that you probably don't do a lot of performance envelope testing in your environment?

Thanks for taking the time to set me straight -- cheers!

[W. Curtis Preston]

(Had to look up crowd-sourcing...) So if I understand you correctly, you're saying you don't care if many of the readers of your blog think you're being a (insert expletive here). You're going to do it anyway. Interesting. ;)

BTW, I haven't experienced someone going to far in my comments because I'm usually not making blog entries that make then want to say things like that. (You know, like saying that their product is crap.) ;)

Hmmm.... Not wanting to take a side here...

I don't believe in competitors bashing each other. I think it's unprofessional. Tell me what's great about your product, not what sucks about theirs. I (and other customers I've worked with) have taken points off in the past when a vendor starts slamming the other guy.

One of the biggest reasons that this is the case is that (the proverbial) YOU DON'T KNOW THEIR PRODUCT; YOU KNOW YOURS. Competitive info like that is almost always WRONG. Until you've actually used their product, don't tell me how it works.

So I don't condone what either of you are doing.

[Chuck Hollis]

Curtis --

Your first observation is correct. I'm not trying to win the November election here.

For me, a key blogging concept is individuality, authenticity and transparency -- warts and all. I'm not pretending to be someone I'm not.

Put differently, if someone doesn't agree with my views on certain issues, they're free to go on to the next item in their RSS reader.

As far as vendor behavior, points noted -- vendors should (ideally) talk about what they do best, and not slam the competition. No problem there.

If we only lived in an ideal world ...

But what happens when one vendor believes another vendor is intentionally misrepresenting their capabilities? And doing so in what appears to be a consistent and intentional pattern?

Who has the responsibility of calling them out for misleading and/or incorrect claims?

Or, if you personally thought that some vendor was being slightly unethical in their representations, would you call their product cr*p for a particular use case?

If you've got an answer to that one, I'm game ...

-- Chuck

[Craig Simpson]

Chuck, about HP’s SPC-2 result (http://www.communities.hp.com/online/blogs/datastorage/archive/2008/09/08/a-new-world-record-for-xp.aspx) : No, most won’t use an XP24000 just for video streaming. But lot’s of folks use them for storage consolidation. Might that include some data mining? Yep. Might that include some video clips in this day and age? Yep. Might it all need to be backed up? Yep. Might it include a whole bunch of transaction processing, for which SPC-1 is the standard benchmark? Yep. Does XP have top notch performance and commensurate benchmark results for all cases? Yep. Does EMC’s DMX have ANY benchmark results? NO!!!!

We challenge you, whether it’s SPC-1, SPC-2, or even capacity utilization, to submit to an independent third party test. Until you do I think everybody can see that if DMX could put up a good SPC-1 or -2 benchmark number you’d do it. No amount of smoke screen about relevance can hide that.

[Chuck Hollis]

Hi Craig

You seem like a nice guy, so let me explain.

All of us here are very familiar with the UPS-V aka 24000.

Go take a hard look at the configuration you benchmarked. Can you honestly say that anyone would buy one that way? 128 146GB drive pairs in your 1152 drive box? A pure video streaming workload?

Help me here -- what's the point of all of this? What are we proving here?

I'm pretty good at positioning things, but I would have a tough time explaining to a customer what this particular test proved or disproved, especially in the context of a specific problem or concern.

I was discussing this fascination that some vendors have with the SPC the other day with a customer, and he had something very insightful to say, "market leaders point to market share, market laggards point to benchmarks".

I don't know if I entirely agree with that sentiment, but it made me think.

-- Chuck

[Alex McDonald]

Time at ShadeOfBlue Towers is in short supply, but I eventually managed to start the riposte to your capacity assertions;

http://blogs.netapp.com/shadeofblue/2008/09/space-is-mind-b.html

More to come, if I can get out from underneath a pile of stuff on my desk...

[Chuck Hollis]

Actually, I've kind of lost interest in this one -- there seems to be so much variability in use cases, vendor recommendations, tradeoffs, etc.

I think that we both can agree that -- in the long term -- customers may be taking a harder look at effective utilization of storage assets, much as they have with server assets.

Perhaps we can pick a topic of mutual interest that's worth debating to our mutual audiences?

I've got a few ideas, if you're game ...

[Craig Simpson]

Chuck – First the point. Pretty much everybody who buys an XP (or a DMX) has to back up the data on it with time constraints on getting those backups done. That’s a sequential workload represented by SPC-2. Lot’s of people use XP’s for data mining operations. I can’t speak for whether or not people use DMX’s that way. But SPC-2 represents data mining as well. A good number of folks use XP’s for some serious number crunching, also represented by SPC-2. Lots of folks use XP’s for transaction processing represented by SPC-1. The mix varies a lot from customer to customer. With both SPC-1 and SPC-2 numbers available people can get a good relative understanding of how an XP will perform on these portions of their workload based on standard, independently audited tests. No shenanigans, just useful, objective data that’s applicable to every use of a high end array.

Second, you seemed to think the SPC-2 config was pricey. The XP is a high end array. Those are pricey whether they’re ours or yours. In a lot of cases the bullet proof availability, blazing fast single array performance, and ability to consolidate many workloads in one place are a must, justifying the price. That’s why there’s a market for high end arrays, ours or yours.

Now to your question about buying an XP like that for video streaming. There are folks out there with huge amounts of video content who need blazing fast, bullet proof access to some of that content. Based on their business results they’re pretty sharp people. Can I honestly say anybody would buy an XP for that? They have!

[Chuck Hollis]

Hi Craig -- thanks for the response.

Now, seriously, how many people would take an array designed for 1152 drives, lightly populate it with 128 pairs of 146 GB 15k drives, and run a single sequential workload against it?

And how many vendors would even suggest that's a "representative configuration" to someone who knows better?

My dad always told me, when you find yourself in a deep hole, stop digging!

-- Chuck

[Craig Simpson]

Certainly the workload and configuration are in use for our products. I can’t speak for yours.

You seem to be hung up on some “representative” config or workload. What I observe out there is a pretty good diversity in customers’ specific needs. We choose to provide objective information that can span a wide range of needs. You don’t.

[Chuck Hollis]

Hi Craig -- thanks for the response, I think.

Maybe we look at this benchmarking thing fundamentally differently. I can't explain it any other way.

When we at EMC say "benchmark", we mean "representative workload, representative configuration" which we usually summarize as "use case".

In this case, I'd agree that "representative workload" for the SPC-2 is pretty clear -- it's sequential streaming (no writes, just multiple data streams). No argument here.

Where I have a major beef is in the "representative configuration" department.

I think most of the readers here would recognize that a full-blown USP-V aka 24000 with a full complement of controllers and cache, capable of supporting 1152, but very lightly populated with 128 pairs of mirrored 146GB 15K drives doesn't really satisfy the "representative configuration" for this workload.

So, step back a minute.

I'm imagining a customer that we're both calling on, and the HP guy comes in with their $1.6m proposal for 18TB usable, and the rest of the competition comes in for anywhere between $250k and $600k -- and offers roughly equivalent performance and availability.

That's where this approach fails, Craig.

Your benchmark testing doesn't reflect what happens in the real world, does it?

Sure, you provide data. It's just not useful data, is it? Do you expect customers to make informed buying decisions based on these results? Seriously consider a USP-V / 24000 for this specific workload?

How does that work?

I mean, HP published the benchmark, so it implies that HP believes it's useful, valuable information to help customers make informed buying decisions, since HP doesn't believe in cheap marketing stunts, right?

And that's where I think the two companies philisophically disagree.

I'm just waiting for you to see the light here ...