A Quick Note On Primary Data Dedupe and I/O Density

My, that's a geeky title for a post, isn't it?

Wanted to dash off a quick observation that might impact your thoughts around this topic.  At the very least, it's something to be aware of.

Fasten your seatbelts ...

Dedupe Basics

Data deduplication is useful stuff.  Much the way that compression shows up everywhere in the infrastructure stack, data deduplication can be thought of in the same way.

Lots of wonderful use cases, the most prevelant being backup and archiving.  Great gobs of redundant data to go after here.  Following that, large file system repositories seem to be an attractive use case.

One particular vendor is starting to push dedupe for primary data stores (e.g. hot application data), and I've recently observed that they're starting to bump into a few laws of physics that fly in the face of aggressive marketing.

Consider I/O Density

If you think about data deduplication at a high level, you're taking something that used to comprise many blocks of storage, and now using fewer blocks of storage to do so.  The greater the deduplication level, the greater this effect.

As fewer blocks (and presumably spindles) are being used to store information objects, I/O density inevitably goes up.  Put differently, the number of I/Os per unit of storage increases.  As most storage devices are physical spinning disks these days, it's worth keeping this in mind.

When we're talking about use cases such as backup and/or archiving, this isn't a major concern.  I/O density on these sorts of information objects is relatively infrequent.  Squeezing 50%-98% of the redundant data out of the disk farm doesn't stress any of the disk spindles, and everyone is happy.

Let's move to large file system repository use case.  The vast majority of information objects are infrequently used, so we're in good shape here.  But if certain files get very popular (and are highly deduped), there's a slight potential for a performance problem.  Generally speaking, though, we're in safe territory.

Now let's consider the primary data store use case.  By definition, it's a "hot" storage workload.  Maybe you've taken a database that used to run on 20 disks, and now found that you can fit it on 10.  The I/O density of those 10 disks has now doubled.

Will it be a problem?  Maybe yes, maybe no.

Working Around The Problem

Typically, when storage admins run into I/O density problems, they have two fundamental approaches: more disks, or faster disks. 

By spreading the load on more spindles, I/O density can be reduced.  However, you now require more storage, power, cooling, etc. -- which kind of tends to defeat the whole premise of data deduplication to begin with.

The other approach is to use faster disks -- either disks that spin faster, or perhaps enterprise flash drives.  Invariably, these cost more as well -- again tending to defeat the whole premise of attempting to save money through data deduplication.

There is an argument to be made that different classes of information can be intermixed with the hot data, i.e. put archival data on the unused portion of the spindles that were needed to overcome the (new) I/O density challenge, but we're talking more analysis and effort to accomplish this -- no free lunch.

One of the attractive premises of technologies such as FAST is the ability to automatically detect I/O density problems, and transparently move them to faster media without the need for manual intervention. 

Indeed, if one of those media types is enterprise flash, primary storage dedupe can create incredible I/O densities, and we're good with that.

A Few Examples

Boot volumes for large-scale VDI is an interesting example of the phenomenon.  When you consider the storage requirements, there are two general classes: program binaries and user files.

User files tend to have low I/O density, so deduplicating them like any other file system makes sense -- EMC's Celerra does a fine job of this as a standard feature. 

Program binaries also make an attractive target for data deduplication: whether at the storage array or via VMware's linked clones facility.  Lots of redundancy there.

However, program binaries are subject to "boot storms" -- scenarios where all the virtual machines are booting simutaneously.  Not a huge effect if you're talking small-scale VDI (maybe a hundred machines) -- but a substantial effect when we're talking many thousands of virtual machines all trying to read the exact same files at the exact same time.

As some customers have been experimenting with deduping other forms of primary data (databases, email, etc.) they're starting to notice the effect as well -- they've ended up with increased I/O density that can only be overcome by -- well -- adding more storage!

There's a resiliency aspect to this discussion as well.  Spinning disk drives fail -- it's a fact of life.  And when they fail, the array has to rebuild them.  This inevitably puts a big hurt on I/O response times during the rebuild -- different schemes have different impacts.  Mirroring schemes tend to have less impact than parity schemes, but require (wait for it!) more storage.

Ironic, isn't it?

Final Thoughts

Data deduplication is a useful tool for taming information growth, but it's not a panacea.  Everything in life has a tradeoff of one sort or another -- there are very few magical silver bullets around.

Do I want to live in a world of deduplicated primary information stores? 

You bet -- but not at the cost of performance and availability. 

Comments

[kostadis roussos]

Chuck,

As always it's unfortunate when you wade into technical matters you don't understand.

Your assumption about boot storms is the following:

- large number of shared blocks on a few spindles
- large number of IOPS to shared blocks
- poor performance.

Now let me open your eyes to what we, at NetApp, call DeDup aware caching.

- first client does a read, loading disk blocks
- second client does a read, disk blocks are read from *memory*
- NO IOPS to disk
- Great Performance.

Something to think about.

kostadis

[Chuck Hollis]

@kostadis

Please save the arrogance for someone else, it's annoying and unproductive.

EMC has large non-volatile storage caches on our products as well, and have had them for quite a while.

As we both know, large read caches can be ineffective when presented with a random I/O pattern, such as found in many databases and email stores.

BTW, storage cache is *not cheap*. And if the premise is "save money on storage through primary dedupe", don't we need to add in the cost of the large storage cache to make performance acceptable?

Thanks for writing.

-- Chuck

[kostadis roussos]

@chuck

Pot, kettle, black, arrogant, calling me. There is a saying that uses those words ;-)

I think you missed the point and redirected.

You started making claims about exchange and databases, but I'm not making claims about that, today.

So let me try again, you claimed that dedup impacts READ IOPS because you're pounding the same set of spindles for VDI. I observed, that's not the case because of intelligent caching. You then said EMC has intelligent caching. I agree. So, if both systems have intelligent caching, why is my claim false, other than NetApp can do this today, and EMC cannot?

cheers,
kostadis

[Chuck Hollis]

@kostadis

Dealing with NetApp folks can be so tiresome. I don't know if there's any value in it from my perspective. Everything becomes insults and fights. Very unproductive.

So, the blog post is about I/O densities increasing as one deduplicates data. Any arguments about that point? No, I didn't think so.

I offered up a few examples where one might potentially see this effect for primary data stores. VDI was one example, there are many others.

I did not make any claims as to whether vendors "can" or "can't" do something specific. I would agree that both NetApp (and EMC, and other storage vendors) have various capabilities in this space.

It is also hard to argue that remediating potential I/O density resulting from aggressive primary deduplication can incur additional cost -- whether disk, memory or human effort.

Personally, I think you might give some thought towards lining up the marketing message with the technical reality.

-- Chuck

[Carter George]

You have a very good point around I/O density, spindle count, and dedupe, Chuck. However, if you take it to the next level, reducing data footprint allows you to take advantage of the next tier up in storage performance. If you can reduce the size of something by 90%, and you leave it on the same drives, then you may well run in to problems of disk I/O latency and performance. But if that data reduction allows you to move the physical location of storage to enterprise Flash or to get much higher I/O buffer cache hit rates, then your performance could get a lot better.
The two key questions are:

1. does your dedupe get you enough data reduction to make the economics of moving the data to a faster tier feasible; and
2. does your dedupe engine allow you to control the physical placement of duplicate blocks on different volumes?

If you have to store deduped blocks on the same volume they came from, as NetApp does, then you can not take advantage of this “move up a tier” benefit of dedupe.
On the other hand, if you have a solution like Ocarina’s ECOsystem for EMC Celerra, you have total control over where the deduped blocks live.
Files might live on volumes A, B, and C, but when they are deduped, only their metadata headers will remain there on those volumes; the actual physical blocks may now be placed on any physical storage accessible to Celerra – including solid state. There are no rotational delays on solid state, and if a data reduction of 90% makes it possible to afford to put a data set on solid state, it could be a win all the way around – on cost and performance.

[Vaughn Stewart]

Chuck - This is a great discussion - but you are misleading customers by misinforming them.

I wouldn't suggest that your tactics are intentional, you're obviously very bright and been in the industry a long time, so maybe you're just passing along poor information.

Maybe this will help: http://tinyurl.com/y8qvhjq

Cheers,
Vaughn

[John F.]

"If you have to store deduped blocks on the same volume they came from, as NetApp does, then you can not take advantage of this “move up a tier” benefit of dedupe."

You absolutely can and do take advantage of the "move up tier" on NetApp. The move up tier is PAM, an extention of deduplicated cache.

John

[Preston de Guise]

Chuck - great to see a comment at the end of your post that dedupe isn't a silver bullet. While I agree it's got a place in total storage optimisation, at times it comes across as overhyped. Having someone in storage assert that it's not a total panacea is a real breath of fresh air!

[Aaron Chaisson]

funny that the NetApp guys think that they have an advantage here. the earlier comment stated
- first client does a read, loading disk blocks
- second client does a read, disk blocks are read from *memory*
- NO IOPS to disk
- Great Performance.
Congratulations, you just explained how read cache works ... we've all got it. EMC, NetApp, EQ, HP, IBM, you name it. If you're in the storage business, you've got read cache and they all help with boot storms the same way, You just sell a TON of it at exorbitant costs. if you think Flash drives are expensive, you should see the cost per GB of read cache (or PAM as NetApp calls their add-on boards).

Now, where EFDs really help things like VDI and other use cases where data reduction technologies like Thin Provisioning and DeDup increase IO densities is in steady state IO scenarios and large widespread operations. Let’s start with steady state IO. While Read cache is great for boot storms (for the reasons previously outlined) read cache is not so good for steady state IO where thousands of VMs (let’s stick with the desktop use case) are simply running normal every day tasks. At 5-10 iops a piece that would be 10k-20k iops for 2000 desktops. And because those iops are purely random once they hit the array, read cache will do nothing for you, unless of course you have a TON of it allowing a higher probability that the FIFO nature of read cache would have any given data still in cache. Of course the cost of that much read cache offsets much if not all of the savings dedup offered in the first place.

The other scenario would be things like application patching for persistent desktops or mass antivirus runs (which should be staggered whenever possible) These scenarios generate very large io loads that can flood the system and things like app loads are write intensive, so read cache again buys you nothing. I’ve spoken with many customers who are experiencing insane response time modulations under these scenarios because there simply aren’t enough spindles behind that deduplicated storage. Here is where the second great benefit of Flash Drives comes into play, they can absorb large spikes in load while maintaining relatively flat response time curves, because there is no increasing seek times due to rotational and actuator latencies. Drive a fibre channel drive from 30%-80% utilized and you’ll experience at least an order of magnitude impact of performance. Do that with flash and you may not even notice it. Sure, performance suffers, but we’re talking 2ms up to 6ms, not 8-10ms going up to 300ms or more.

Chuck is 100% correct. In many cases, data reduction technologies without faster drives doesn’t save you anything. As io/cap ratios increase, having a variety of performance options at your disposal is the only way to guarantee the most efficient use of capitol assets.

[Chuck Hollis]

@John F

PAM costs $money$, just like other memory-based accelerators. As you bandy about the COST SAVINGS OF DEDUPLICATION, perhaps you'd like to include the cost of PAM to get performance back to something near normal?

Just a thought ..

-- Chuck

[John F.]

@Chuck

EFDs cost a lot more $money$

John

[Josh S.]

@Chuck

"When we're talking about use cases such as backup and/or archiving, this isn't a major concern. I/O density on these sorts of information objects is relatively infrequent. Squeezing 50%-98% of the redundant data out of the disk farm doesn't stress any of the disk spindles, and everyone is happy."

I would have to disagree with this statement. We saw very clearly that processes such as FSClean on the DDFS data domain boxes, block pool reads and reclamation on the DL3Ds and even Avamar's HFSCheck become big issues when arrays and even RAIN grid get bigger.

It's also interesting to see how one sided the viewpoints are. Your blog post specifically talks about dedupe on primary storage being not viable as a technology (not even talking about cost). I also disagree with this. That's where PAM II becomes a significant advantage on high read i/o environments and will make dedupe work better... I also assume that the amount of cache being put into Vmax will eventually if not already serve the same purpose albeit without advantage that WAFL enables.

Josh S. (recently ex-EMC who has done plenty of POCs now netapp guy)

[Chuck Hollis]

@John F

Yes, flash drives cost more, PAM costs more, additional spindles cost more, etc.

The point remains the same: before a certain vendor claims "DEDUPE SAVES YOU MONEY", perhaps they should factor in the various remediation costs to restore performance to an acceptable level?

-- Chuck

[Peter Smails]

Chuck et al...interesting debate re: dedupe but if the objective is data reduction without compromising performance or availability, we should be discussing compression, specifically real-time compression. Better reduction for primary data and typically enhances performance by reducing I/O. Data reduction, performance, and availability. Chuck, you should join the primary dedupe panel at SNW.

[Chuck Hollis]

@Josh S

Yes, there are probably exceptions to everything. Forgive me, I was talking about the general case. Exceptions might exist, haven't heard of any specifics as you mention.

Forgive me for being suspicious, but you wouldn't be the first NetApp employee to spew erroneous technobabble FUD.

One sided? Please go back and reread the post.

I never claimed that primary dedupe was "not viable". Why would you put words in my mouth that I never said? Please go back and prove me wrong.

If you read carefully, I said that increasing I/O density would might have to be remediated, and that might increase costs.

Can we move on now?

Best of luck with your new gig at NetApp.

-- Chuck

[Chuck Hollis]

@Peter

Good point. Real-time compression takes advantage of the CPU bonus we've got these days. EMC believes that the two technologies complement each other nicely.

For example, the EMC Celerra uses a combination of approaches to reduce data footprint. Our tests (which will be eternally refuted to the death by NetApp fanbois) shows overall better data reduction and improved application performance compared to the brute-force approach used by our competitor.

Rather than argue it out here with the exremists, I would invite any customer to put both boxes side by side, and see which product does the better job with *THEIR* data. It's an easy enough test. You'll also find less overhead between raw and usable -- not surprising.

I think they'd be a bit surprised on just how effective the data reduction is on a Celerra these days.

Thanks for invitation to SNW, though ... :-)

-- Chuck

[Chuck Hollis]

All

I'm having a tough time having a reasonable discussion here on industry topics. It seems that the NetApp crowd treats everything as a personal vendetta.

I thought I made some reasonable arguments here that had more to do with real-world tradeoffs with physics and economics than competitive bashing.

The points I make are always open to debate, of course.

While I want to keep this an open blog with respect to commentary, I'm getting a bit tired of the continual off-topic competitive assaults here.

They're time-consuming, and generally unproductive.

Any suggestions on how we might improve this situation?

-- Chuck

[Josh S.]

@Chuck.

Definitely no technobabble FUD from me and if you don't believe me, talk to Shishov and Morihiro and whoever the DD guys are now. It's the inherent technology limitation of deploying dedupe.

In regards to putting words in your mouth, I was specifically referring to this:

"One particular vendor is starting to push dedupe for primary data stores (e.g. hot application data), and I've recently observed that they're starting to bump into a few laws of physics that fly in the face of aggressive marketing."

but no mention of PAM II as complimentary. I apologize if I took it out of context or misunderstood your intentions in regards to cost being your primary focus.

Another question for you - Celerra deduplication is purely SIS with compression and it's the 1st iteration. Please explain how that is more efficient than a block based deduplication algorithm on WAFL. I for one have seen the demo of both products and would love to see another bake off.

I think this is healthy debate. Competition drives innovation and transparency to our customers are important.

regards,
-josh s.

[Chuck Hollis]

@Josh S

I *will* go talk to them about it. However, you're veering off the topic, which is *PRIMARY STORAGE* running *PRODUCTION WORKLOADS*

Looks to me like the big PAM cards are read-only. Primary storage and production workloads are read/write. Sustained write performance has never been exactly a strong point for WAFL.

I don't see PAM II as complementary because large read caches have limited use in many production workloads (been there, done that), and absolutely *NO* value when it comes to writing data.

And those half-terabyte memory cards do not look cheap. If I remember correctly, they use flash, which brings up the whole wear-leveling question, which you should be familiar with.

Am I missing something here?

As far as your last question, it's often the case that data may be compressed effectively without being deduplicated.

As a simple example, imagine an unload of a 50GB Oracle database to a single file -- hard to dedupe, easy to compress.

Many slightly-different copies of the same thing favor dedupe, such as backups, which is why dedupe is so darn popular for this use case. Hence our recent strong interest in DataDomain to complement Avamar and other dedupe assets.

The filesystem bakeoffs we've already done show varying results based on file mix: sometimes we're in the same league, sometimes we're ahead, a few times we're not.

Again, this stuff isn't magic and won't solve world hunger.

I remember getting into this long and pointless argument about who had the better filesystem dedupe capabilities, and -- at the capacities we were talking about, a single 1TB disk was more than enough to make up any difference between the two vendor's approaches. I felt bad pointing this out, but sometimes we need to maintain perspective.

Thanks for writing.

-- Chuck

[kostadis roussos]

@chuck and @chiasson

Point of clarification on caching.

In a traditional storage system if client A reads file 1, and client B reads file 2, then file 2 will turn into a sequence of read operations. This is because, none of file 2 data blocks are in memory.

But with "intelligent" caching, if file 1 and file 2 have deduped blocks, then something different happens.

When client A reads file 1, the shared disk blocks with file 2 are also read such that client 2 has to read substantially fewer disk blocks from disk.

This is a new behavior and is tied to how dedup works in WAFL.

I don't suggest this is a fundamentally new technology, but rather an example of how the "physics" and "science" issue can be addressed.

As for the cost of PAM vs the cost of disk, that's a good question, and I urge customers to do their own analysis. We at NetApp, of course believe this is a good deal.

cheers,
kostadis

[twitter.com/randydarthur]

A couple of thoughts on the whole NetApp vs EMC as a Virtual Infrastructure storage back end Twitterstorm of Thursday night. Let me start off with this disclaimer. I am speaking only for myself and not as a representative of CSC management. Furthermore, let me also say that both EMC and NetApp are valued alliance partners for CSC and we do good business worldwide with both parties.

I must confess a bias towards NAS storage for Virtual Infrastructures as opposed to Block access protocols. I believe that the abstraction of NAS greatly reduces the amount of administrative effort required to configure and support the VMWare storage environment. There are fewer places where defects (misconfigurations) can be injected in the process and therefore reliability is higher. Working for a service provider, I want simple, reliable and cheap. Because you are dealing with objects in a file system, there are fewer administrative touches required. The overall level of staff training can be lower. Properly engineered, NAS should generate a cheaper overall virtual infrastructure than one based on shared storage connected via a SAN.

Although I hear that FC performance has been improved in vSphere, in ESX 3, NFS demonstrated superior performance characteristics under higher concurrent I/O workloads with more graceful degradation as more VM-s were added. NFS was only edged out in raw throughput by FC when a single virtual machine was running. There are workloads where FC is the right choice, but the vast majority of VMWare workloads work just fine on NAS. With NAS, there is no need to go for FCoE in order to get to a converged datacenter network fabric: you already have one in your existing data center LAN.

Everyone’s mileage varies, but most “average” VM-s in our environments seem to generate on the order of 25 I/O-s per second at 8KB block sizes. If you multiply these rates by 320 (or more) virtual machines in a single blade chassis, our experiences bear out Chuck’s statement that (paraphrased) spindle count will be the biggest determinant of how the back end storage performs at these I/O densities. Flash Drives and FAST-like in box tiering will be welcome improvements (shoutout to AmberRoad ;-) ), but Cache or Flash drives are still comparatively scarce resources that can eventually be saturated. I think the number of spindles will still matter going forward but perhaps not as much as today.

As to the dread “boot storm”, CSC designed its first iteration of Dynamic Desktop (VDI) infrastructure around NetApp with Cache (PAM) accelerator boards. Chuck implied that the cache memory would be prohibitively expensive, but we found that the additional cost of the cache was far offset by the 5X scaling factor we were able to get. So in our case, paying for the additional cache was a significant net cost saving to us.

For me, the critical attributes of VI back-end NAS storage are as follows:

1) Writeable snapshots – Create a generic, gold image of an operating system and stamp it out so there is minimal storage consumed by each new VM instance. Just the blocks with different data between images are consumed.

2) Data deduplication – As you patch each image, each “branch” of the writable snapshot will have a lot of duplicate data added to it. To optimize storage consumption, you need a process to come along behind and collapse these blocks to reclaim the duplicate space.

3) Tight integration between the storage device and Virtual Center to ensure consistent snapshots of virtual machine images in can be efficiently generated without human intervention as workload moves around the server farm and replicated (if required).

In my opinion, NetApp has historically had the edge in these categories but EMC’s Celerra team is catching up rapidly. I believe that today, NetApp will likely retain a modest edge if all it comes down to is a NAS functionality shootout with Celerra for the VI back-end. But that is not the ground that EMC is choosing to fight the battle for long-term VI back-end supremacy on. Far more worrisome for NetApp it seems to me is that EMC is positioning itself to leapfrog NetApp in terms of fundamental enabling technologies on a couple of fronts.

First, EMC has acquired FastScale. And that technology takes DeDupe to a whole new level for VMWare. Will there be teething pains? Of course, but for Virtual Infrastructures, this technology works at a much higher level and yields memory savings in additions to disk space reductions. This will be pretty cool if EMC can get the price and reliability right for this technology.

Chuck has circled back a bunch of times on the barriers to vApp mobility. Data movement is the big problem he keeps coming back to. I have seen a spate of recent posts of his on the subject. Solving this is key to getting VMWare’s vCloud world off the ground. ATMOS is certainly looking to handle the lower I/O tier apps and aspires to be the file/object store of choice in the vCloud that can facilitate workload mobility and be a persistent object store. But what about distributed storage for high performance block applications like data bases that need to move between datacenters in a hybrid cloud? What about VMDK mobility? There is still a gap in the technology to address these high I/O needs and really permit the vCloud ecosystem to take off.

I have to believe that given the increasing levels of cooperation and integration between VMWare and EMC (compared to the past) that EMC will be well positioned to leverage or influence development of new hypervisor capabilities to address this high performance cloud I/O functionality gap. Since the hypervisor knows about “dirty” blocks and I/O hot spots it is probably the layer best positioned to exploit shortcuts to keeping VMDK or vApp images in synch over a distance. To me, this seems to be the logical place to start looking at optimizations. That doesn’t preclude NetApp or other storage vendors from playing, but it probably means EMC will be first to market with a comprehensive set of solutions across all classes of I/O workloads in the vCloud. It also probably means that NetApp’s traditional strength in terms of efficient, low-cost distance replication will be to some extent be negated by a combination of hypervisor functionality, advances by network equipment vendors or refinements to array based replication targeted at virtual server environments from other storage players like EMC, IBM and HDS.

For me, the questions come down to simple ones. How do new EMC and NetApp storage innovations enable me to lower the cost of delivering each virtual server instance? How do these storage technologies facilitate the widest range of vCloud application mobility? It does me no good to have cheap vCloud compute resources that are prohibitively expensive to access either because of onerous network requirements or massively expensive storage infrastructure. If the infrastructure is cheap but complex and hard to support, that does me no good either. I think bun fights over the gory technical details of today’s storage technologies (let’s face it - all of which are fit for purpose, reliable enough, perform well enough) are really proxy wars over which technical approach yields the best “real world” TCO today and whether or not the various vendor approaches will best facilitate workload mobility in the vCloud ecosystem going forward. Both companies have a good technical solutions and sound value propositions. It will be interesting to see how their different approaches play out over the next couple of years.

I would welcome any comments on the above to improve my understanding of these issues.

Cross posted to (http://blogs.netapp.com/virtualstorageguy/2009/09/run-everything-virtualized-and-deduplicated-aka-chuck-anti-fud.html )

[Chuck Hollis]

@Randy -- thanks for a long and detailed comment.

I missed the twitterstorm you refer to, besides, I find it's hard to have a meaningful conversations 140 chars at a time.

There's a lot to agree with you on.

Most of us agree that a filesystem is more convenient for the storage backend -- until you start pushing the hairy edge of I/O, and then it's useful to have (real) SAN in your bag of tricks.

Your thoughts on performance should be revisited with vSphere (vastly improved I/O stack) and optionally PowerPath/VE. Newer SAN abstraction models also relieve a lot of the provisioning/mgmt tedium you mention. For those reasons, I wouldn't count SAN out of the game just yet.

It's a different world these days -- so much so that we were able to revise the all-important VMs-per-server number signficantly upwards for moderate workloads, and dramatically increase the size of the applications we could now consider virtualizing.

It's eye-opening.

This applies to CSC's business model in two ways: (1) potentially lowering the cost-per-virtual machine by increasing the number that can run (assuming sufficient CPU and memory a-la Cisco UCS), and (2) being able to virtualize the most demanding workloads which now require a physical server infrastructure at CSC.

Your successful approach to VDI footprint reduction was developed long before VMware had tools to do this (i.e. linked clones), which brings up an important question -- are you sticking with your storage-specific architecture, or moving to a storage-agnostic approach?

Not surprisingly, we are recommending the storage-agnostic approach these days. Better to use the tools provided by VMware, we believe.

You're right about the amazing potential of the FastScale technology to not only dramatically reduce the memory footprint of virtual machines, but to better control the code objects that comprise them from a consistency, compliance and licensing perspective.

We're very excited as well.

You and I would agree on many of the the ultimate goals of virtualization at scale -- (1) increasing the number of VMs per unit of resource [CPU, memory, disk], (2) dynamically increasing application performance of a given VM cost-effectively, (3) mobilizing VMs and their data at ever-increasing distances, and (4) managing and securing VMs in enterprise or multi-tenancy service provider environments.

And, your quite right about all the other various investments going into this area. Many you've seen and listed above, several have yet to be discussed publicly.

At some point, I'd invite you to have a f2f conversation on these topics with myself and others who are working in this area.

It's pretty obvious that you think long and hard about these topics (as do we) which would make for a great discussion!

-- Chuck

[Paul Carpentier]

Chuck,

I guess Preston de Guise put it rather well. It's refreshing to see the whole de-dupe hype right-sized for once.

I particularly like the fact that you did this - even if you didn't realize - using a very innovative new hybrid word: "prevelant".

At first one might think that you were merely talking about backup and archiving being the **prevalent** use case for dedupe, with a typo easily overlooked in the heat of the moment.

But I think there's more to it. To me, you were clearly - and rightfully - pointing out that this constitutes the only **relevant** use case as well. I couldn't agree more.

Unless there is very intricate metadata-based insider trading involved (Ocarina comes to mind), the general-purpose primary data use case for block-level dedupe doesn't make any sense. Whatever the one trick pony guys keep WAFLing about.

[C.J.]

This is a great debate and I for one believe that dedupe will always have it's place on both primary and secondary storage and for that NetApp appears to be better positioned.

However, why does EMC keep bringing up an archaic product like Celerra? This product has never and will never live up to the hype EMC hoped it would and the fact that if you buy virtually any other product line EMC will throw one of these in for free??? come on... "if you give to people, they will use it."

Not in this case.

[nate]

Can't we all just get along!! :)

I took the opposite approach last year with our storage refresh. We tried data domain hoping it could reduce our storage requirements, about 50% of which is compressed text. We threw a few hundred gigs at a DD box and didn't get any benefit.

So rather than trying the more complicated world of dedupe on primary storage and hoping it works, we went with 100% SATA disks, and it worked out that our "hot" vs "cold" data requirements lined up well that we got a good balance of lots of space with SATA and enough performance to keep things running very fast. Even now with the spindles sustaining 60ms+ response times for disk writes, the front end is about 3ms. Read response times are ~25-30ms. For a while I thought the increase in load would ever end.

So rather than going the de-dupe route perhaps more people should be considering SATA for their tier 1 storage stuff, of course it only can work if you have enough disks depending on the architecture. I wouldn't deploy a 32-disk system with all SATA, but 100+ disks, sure, at least with my trusty 3PAR.

Adding another 100 disks early next year to help with I/O, and that will give us another 100TB, that much more data we can store online, and still fit everything in 1 cabinet.

[nate]

whoops two cabinets not one, was thinking of space not drives, 240 drives in first cabinet, 320 in additional ones so we'd be in two cabinets with 300 drives(have 200 today).

[Chad Sakac]

** this was posted to the NetApp blog, and wanted to post it here for thoroughness **

-----

Disclosure Chad (EMC employee here) I'm going to try to help, rather than fan flames here.

Deduplication - everywhere and as much as possible is fundamentally a GOOD thing.

Use of more RAM (and use as efficiently as possible) in storage arrays (particularly as RAM and Flash prices continue to drop) is fundamentally a GOOD thing.

Frankly - every technology we can deploy that reduces footprint/cooling/complexity - while meeting the SLA requirements of the customer = GOOD thing.

It's up to each vendor to prove to each and every customer that "all in" we can provide the most efficient (where efficiency is measured on capex, opex, flexiblity and ease-of - including space/power/cooling) - while meeting the SLAs for the various use cases.

This is a non-triviality, though our respective marketing and sales teams (as all do) need to try to make it as simple as possible. (this is also important, as engineers - including me - have a habit of making everything sounds crazy complicated)

As someone in the comment thread pointed out, each company implies that the efficiency technologies of the other are useless in general, and at the same time imply that the efficiency technologies they have solve world hunger - each while furiously working to fill any competitive gaps.

EMC offers production data dedupe on Celerra, but this technique, while highly efficient in the general purpose NAS use case - it currently skips large (>200MB) files, so has limited use in cases where the files are large (VMDKs as an example).

I've shared a lot of public customer feedback showing it within or above the NetApp's production dedupe approach by percentage points with that class of dataset. @ Mike R - just to highlight that no one can cast stones here (we all live in glass houses) - our approach continues to be "discounted" by NetApp folks.

We aren't stopping there - and you can BET we're working furiously to apply the idea of deduplication everywhere we can (including the production block use cases). Of course, NetApp isn't staying still either - as a respected competitor - I would hope for no less.

We would of course position the "total capacity efficiency" - of all production use cases (including NAS), as well as backup use cases as the measure of how much we can save our customers. We would also include ideas where we're still pushing such as auto-tiering (at file and block levels), dense storage, spin-down and archiving as efficiency technologies.

Now, on EFDs, as an interesting note - the performance data we have published (which is public) shows that where we see a 30x uplift in IOPs and 6-10x lower latency of use of solid-state on our block devices (CLARiiON/Symmetrix), on our NAS device (Celerra) we see about a 7x-10x more IOPs and about 4x lower latency. Now some would claim it's because of our NAS code (meaning ours specifically not NAS generally), but I suspect that EMC has smart engineers, as NetApp does. It looks like the long-lived code paths in the Celerra stack have a harder time maximizing 10x lower device latency and 30-50x more IOPs from a drive. The flip side is that the extra abstraction layer makes things like deduplication easier (which is why it's there first). As NetApp introduces SSDs (which I'm sure they will at some point - since after all, it's as "simple as plugging them in", right? :-) we'll see. Perhaps PAM was an easier architectural model for NetApp - and that doesn't discount it, or make EMC's approach less innovative.

Each of us has architectural models which make certain things easier, and others harder. that doesn't invalidate each other's innovations, and each other's efficiency technologies.

If I read Chuck correctly, he wasn't saying production dedupe is bad - but rather that is a capacity efficiency technique. It needs to be coupled with performance (IOPs/MBps) efficiency techniques for maximum effect on reducing power/cooling/aquisition cost (or be applied to workloads that don't have a need for performance, by only GBs). In some cases this effect is extreme, and in others, less extreme.

NetApp responds with PAM, and it's up to customers to evaluate the cost/benefit advantage (no implication that it isn't there). I think what he's saying (and I don't speak for Chuck, but just looking at the thread): "does the cost of PAM, and using new hardware (including using vFilers if they want to use an existing array) translate into the full ROI picture you expect after a sales call?"

I will say this - the cost of EFDs are dropping VERY fast - and there is a benefit of being a standard form factor, in that everyone is competing to make those denser and less costly.

As an "reductio ad absurdum" example - in a recent competitive campaign between EMC and NetApp for a VDI use case, the customer did their own math and concluded for their 7000 initial user deployment, that leveraging their existing Symm and adding SATA with a small number of EFDs was the cheapest (acquistion and operating) costs. The idea of production deduplication was very appealing (particularly because of "simple" positioning along the lines of "add this to your environment and reduce your storage cost by 50%"), but in the final measure, with quotes and costs from both - they chose one way. Other customers choose otherwise.

When it comes to efficiency (and again, I'm speaking for myself here), I respond with claims of good utilization, writeable snapshots, thin provisioning, parity RAID (R5/R6 depending on the customer), production deduplication with general purpose NAS, and backup deduplication (at the source and target). I talk about EFD for focused use cases today, and I also show them where we are going with fully-automated tiering in the near future to broaden it's applicability.

We (and others) both have broad use of Thin Provisioning and writeable snapshots used widely (across broad use case sets)

I try to spend the bulk of my time on showing customers HOW to leverage technology, and how to integrate it with their virtualization/datacenter transformation efforts. In my experience, more often than not - that's where the efficiencies are really made/lost.

Then, I go back to my day job, working on VMware/EMC technology integration with the goal of making storage invisible - and apply as Randy points out in the datacenter, and in the cloud (including cost points that are 1/10th traditional enterprise storage).

I guess what I'm trying to say is:

1) This stuff is never as simple as it gets painted out to be.

2) that perhaps we aren't as different as we might think we are (wildly different in technology - but not different in the sense that we strive to be as efficient as we can).

3) As soon as we assume the competition is dumb, start to discount what customers tell us, or refuse to accept that innovation occurs everywhere - that's a dangerous place for anyone - an individual or an enterprise.

@ Randy - you were at my 1:1 CSC briefing, so have visibility into not only what we have but where we are going. Thank you for your comments.

-------

There was some dialog (suggesting I was mis-positioning Celerra dedupe as applying in the VMware use case, and also questioning whether solid state is real or not)

This was my response...

-------------

@Vaughn - I was VERY clear in my post Vaughn.. I know you're busy, but so am I. please re-read. Perhaps the caffeine suggestion is a good one :-)

My point was not to equate your dedupe to Celerra dedupe, and call them the same (they are not the same).

My point was that it's an example of the fact that we share a view: efficiency = good. Each of us has areas where we can be more, or less efficient. Dedupe for production storage is easier when you have an abstraction layer between the block storage and the data presentation layer. This is where TP happens (on all storage arrays), and where that layer is mature, it is the natural target for how you do dedupe. In NetApp's case (from my understanding - I don't claim to be an expert), the dedupe function is a natural side effect of some of WAFLs characteristics. In otherwords, you are leveraging your core architecture to provide customer benefit. GREAT.

My point on Celerra dedupe (to reiterate - LITERALLY requoting myself from the earlier comment: "EMC offers production data dedupe on Celerra, but this technique, while highly efficient in the general purpose NAS use case - it currently skips large (>200MB) files, so has limited use in cases where the files are large (VMDKs as an example).". I don't know how I can be MORE clear. Currently - our dedupe approach on NAS does diddly for VMDK on NFS use cases. As I stated, we're not stopping there, and we will offer production dedupe over time for more use cases, and across different platforms. It's not easy with our architectural model, but count on it.

BUT - customers arrrays are used for LOTS of use case - not just one. My point is that we're all striving for efficiency, using the tools in our quivers. In some cases we're each ahead, in others behind.

Now - to answer the other question you asked:
"Does EMC recommend Celerra NFS as their preferred platform and storage protocol for VMware environments?"

NO WE DON'T. I've trained everyone at EMC I've ever talked to - pick the right protocol for the customer. Sometimes its' block, sometimes its NAS - in general, the most flexible configurations are a bit of both. Sometimes it can all be met with one platform (here I'm not speaking in general terms, I'm speaking about EMC - customer can determine if it applies as a principle to others) sometimes it can't.

Come on man - look at the joint NFS whitepaper we posted together.

It's as wrong to be reflexively "NAS always" as it is to be "block always".

We do both, we do both well. Now, in our case, there is a difference between the functions, and behavior of Celerra NAS and the block targets of CLARiiON and Symmetrix. There are of course differences between functions on NetApp too based on protocol. For examples, look at the dialog we had about ALUA and FC/FCoE relative to iSCSI on NetApp FAS platforms, or file/vm level snapshot/clone operations. But, I'll acknowledge that those differences are minor relative to the differences between a CLARiiON and a Symm.

Sometimes I see that as a weakness and sometimes I see it as a strength.

The question is - do you always see it as a weakness? If so, I beg to differ.

What I was trying to point out is that every vendor (EMC included) suggests that their efficiency technology is all that matters, and what others have is bunk. The reality is there's goodness all around. Customers choose what's the best for them.

Vaughn - the core principle you state has indeed served Netapp and it's customers well, and been an important part of your growth to date is that there is a sweet spot where the tradeoffs of single design (and add features/functions) is a good position to take.

That said - the Data Domain acquisition battle was an interesting one. if NetApp's dedupe is so on the mark, in all use cases, ALL THE TIME - why battle it out so furiously for SISL? The answer is that for B2D where only real performance metric that matters (ingest rate) - the inline dedupe model works better than a post-process. VERY hard to engineer into a production storage target. If you're aiming for the highest SLAs, the N+1 storage architectures of HDS and IBM DS8000 and Symmetrix rule the day - VERY hard to engineer into a classic "two head" storage architecture (not impossible - and you know that NetApp and EMC are working hard in this direction).

This has been my point - respect for differences, acknowledge innovation where it occurs, and sometimes a strength in one context is a weakness in others.

That's all.

@Nick - boot storms are not the practical challenge with VDI use cases, and are addressed with: a) relatively small/primitive caches; b) staged boot; c) configuring the broker for log-off rather than shutdown on client shutdown; d) configuring the client guest swap to be fixed.

The real challenge (in my experience) are the other elements of the desktop lifecycle - if process change is out of bounds. These are AV, patching, and other similar tasks. It is aleiviated through user data redirection to NAS, but this introduces some complexity (check-in/check-out gets busted for example if you don't use folder synchronization). This has been feedback from NetApp and EMC customers.

More than all of this - most VDI projects get sidelined (to date) based on end-user experience unless the use case is narrowly defined to not include general knowledgeworkers.

Re: SSD adoption - don't know what you've heard but here are the fact that I have access to. Every customer is actively looking. We've sold out for the last 6 quarters. I just tried to order 40 (for various VMware/EMC performance testing use cases), and we're backlogged. To understand why, understand - we're not talking about Intel X-25Ms here. While they are 8x more expensive when expressed as $/GB, they are 1/4 cheaper when expressed as $/IOP, and 1/95 cheaper when expressed as watt/IOP. They are also on an incredibly steep price decline.

That DOES NOT make them a replacement for production dedupe, Rather, they are orthogonal, but related efficiency technologies for storage use cases. Other examples are: "dense" - like more drives per enclosure or 2.5" disks; or fully automated storage tiering, or spin-down. depending on the use case - each is more or less important - but most customers have a LOT of use cases.

[Chuck Hollis]

@chad

Well said, as always. Thanks for taking the time.

-- Chuck

[shiningarts]

I believe the primary data store dedupe is definitely worth pursuing because it is where you are going to get more bang for your buck eventually, although there might not be an adequate use case for now because the technology has not yet caught up. The I/O density issue MUST be resolved for all of the up-and-coming new technology such as Dedupe, Virtualization, and Private Cloud.

One of EMC’s strategic initiatives is Solid State. This does not appear to be realistic at this point because of the feasibility and cost. In due course, however, all primary storage devices will utilize Solid State storage and the currently-used disks and tapes will ultimately become the secondary or tertiary backup storage devices.

The use case is applicable to a given technology available at the moment. But forward-thinking technology vendors must look beyond the current use case and steer toward what is plausibly possible despite technological hurdles that may be in the way. If this were not the case, we all would be still talking about the mainframe use cases with Cobol (or maybe even the use cases for the abacus).

[bathmateus]

very good posting. thank you. :)

bathmate

[Todd A Johnston]

Raising the question for Dedup in 2012. The cost being CPU, Rehydrating the data to update and silo'd storage,processors to effectively compress data.

Is the answer for Dedupe to Compress and partition Oracle?
DataDomain?
Delete button?

-SANarchITectUS