Your Useable Capacity May Vary ...

In the US, every car sold has a standardized EPA rating on fuel economy.  Using a quaint measurement system of "miles per gallon", it's not precise, but it does give buyers a rough measure of comparative fuel efficiency. 

And, of course, has given rise to the frequent disclaimer that "your mileage may vary".

Storage arrays may have a roughly analagous measure as well: usable capacity vs. raw capacity.

And is it time to start comparing the capacity efficiency of storage arrays the way we do cars?

------------------------

Update Feb 17 2009:

While the specific conclusions reached in this blog post are now obsolete due to enhancements by the respective vendors, the general topic of storage efficiency is not obsolete. 

I encourage all storage customers to take the time and effort to figure out what their useable capacity might be -- once all overheads are subtracted. 

The differences are still significant, although getting to an accurate figure will take some effort, as can be seen by this post and its comments.]

-- CPH

-----------------------

Why Is This Important?

My impression is that in the US, when gasoline was $1.25 a gallon, not too many people paid attention to that efficiency rating.  Spike gas to $4 a gallon, all of the sudden that EPA rating was very important indeed.

In the storage world, we have the luxury of constantly declining media prices, but an industry average growth in capacity that far exceeds the decline in raw costs.  As a result, most organizations spend more on storage every year.

It's a fair question ...

How much raw capacity are you buying? 

And how much of that do you get to actually use to store your data, once all the overheads are accounted for?

Creating A Standardized Measure

When it comes to fuel efficiency in the US, we have the benefit of a government-mandated standard for comparison.  When it comes to storage, we have no such luxury, so we have to create one.

The proposal I'd offer is a like-for-like comparison as follows:

• a relatively standardized use case that most vendors document with specific recommendations (e.g. Microsoft Exchange)
• a decent number of usable disks, say, 120
• an idealized 146GB disk capacity
• configurations done in accordance with vendor-supplied recommendations for Microsoft Exchange environments

Now, the prerequisite disclaimers, before people start to flame me:

• Yes, every use case is different, but we have to pick one, and Exchange seems like a reasonable proxy of an application that most people have in their environments. 
  
  Although many vendors don't publish recommendations for other high transactional-rate applications such as Oracle, SQLsever, SAP etc. (EMC does, though) I think it's reasonable to extend Exchange findings to these use cases. 
  
  Conversely, I don't think it's reasonable to extend this sort of comparison to file serving, backup-to-disk, decision support and other applications with decidedly different profiles.  Performance and application availability matter in the use cases we're targeting for this exercise.
   
• Yes, efficiency ratios play out differently in smaller configurations (say 10 or 20 disks) or larger configurations (say 500 or 1000 disks).  Every mid-tier array has its optimum configuration points where the numbers play out better than others. 
  
  We didn't try to game this.  We didn’t need to.
   
• Yes, disks are available in many different sizes, but the real issue here is spindles -- the same efficiency ratios play out whether we're talking 146GB disks or 1TB disks.
   
• Yes, vendor recommendations change all the time, but are usually a compromise between decent performance, decent availability, decent protection and decent management.  Just like EPA gas mileage and individual driving styles, you're free to vary from these recommendations, but not without compromising something else.
  
  And we don't want to have to resort to the storage equivalent of "hypermiling".

Where possible, we've provided links to vendor-supplied documentation.  In some cases, these documents have since been removed from public view, so we'd recommend contacting your vendor if you'd like the most -- ahem -- updated version. 

We did the best we could.  If we got something wrong, let us know where we went wrong, and we'll fix it to the best of our abilities.  Our goal is to eventually publish a series of white papers and tools that will help customers figure out the comparative storage capacity efficiency for themselves. 

So, consider this a sort of preview of future materials.

As a starting point, we're going to look at a best-practices efficiency comparison for EMC's recent CX4, NetApp's FAS series, and HP's EVA.  All are offered by their vendors as mid-tier arrays that support these sorts of environments.

Once the efficiency ratios are calculated, they can be expressed in two ways: one way is in terms of "percent efficiency" of raw vs. usable capacity  (ranging from 34% to 70% in this exercise), but it's also useful to express this in terms of price deltas, e.g. because a given array is less capacity efficient, you end up paying much, much more for each unit of usable capacity.

For those of you who are keeping an eye on rack space, we've included that as well.  We haven't yet included energy factors (power and cooling), but we'd like to do that in the future.

Ready to dive in?  I think you'll find it interesting.

EMC CX4 -- 70% Storage Capacity Efficiency

As far as arrays in this category go, the CX4 is near the top of practical storage capacity efficiency without compromising performance, availability and management.  Sure, there's some overhead (as we'll see), but -- compared to many alternatives -- it looks very attractive.

Hot spares -- EMC recommends setting aside 1 disk in 30 as a hot spare.  Hot spares speed recovery of failed disks and provide an extra measure of availability.  The CX4 uses a global hot spare scheme, which means that a small number of hot spares can protect a much larger number of production drives.

Snapshots -- EMC best practices call for reserving 10 to 20% of capacity as a snapshot reserve.  If you run out of snapshot reserve, and more can't be dynamically added, the snapshot fails, and not the application itself.

RAID Parity -- EMC recommends RAID 5 for Exchange environments, configured as 8+1 (8 data disks, one parity disk) which means 15 total RAID groups and a total of 120 usable drives.

Overhead -- All arrays use differing amounts of raw capacity for internal management features.  A portion of the first five drives is used to create a vault for storing both the FLARE code as well as a safety area for the contents of cache -- the remainder of these drives can be used for available capacity. 

In addition, all CLARiiONs store data in 520-byte sectors rather than 512.  The extra 8 bytes is used to provide an additional layer of data integrity, further reducing available capacity by 1.5%.  Not all vendors offer this additional protection against data corruption.  More on this here.

Running the numbers, we see that a CX4 offers 70% usable capacity when configured with RAID 5 and 10% snap reserve, per EMC recommendations.  Choosing RAID 6 instead of RAID 5 decreases this to 65%.  Electing a 20% snap reserve decreases efficiency to 65% for RAID 5, and 61% for RAID 6.

For those of you counting physical space as well, this results in 12 drive shelves.

The chart appears below.

 

 

 

 

 

 

 

Source: EMC CLARiiON Best Practices for Fibre Channel Storage: FLARE Release 26 Firmware Update -- Best Practices Planning

HP EVA – 47% Storage Capacity Efficiency

The EVA provides a very wide number of options in balancing performance, usable capacity and availability.  Unlike other arrays such as the CX4, once these choices are made, changing them can be very disruptive.

The EVA configuration choices include:

- Number of disk groups
- Number of proactive disk management events
- Type and number of disks in a group
- VRAID level (O,1 and 5)
- Disk failure protection level (none, single, double)
- Cache settings

The EVA is slightly unusual in terms of how you think about disk overhead.

First, the EVA is built around the concept of "disk groups".  HP recommends that separate disk groups be used to isolate performance characteristics.  The more distinct high I/O applications you put on an EVA, the more disk groups.  For certain cases like Exchange and Oracle, HP recommends that data and logs be separated to different disk groups.

Given that most arrays do other things in addition to just Exchange, we've assumed (like the CX4 above) that the EVA with over 120 disks will be perhaps be supporting things like SQLserver, Oracle and other high I/O applications.

We've decided to use 7 disk groups for this example: two for Exchange (logs and data), three disk groups for Oracle (two applications each requiring a separate disk group but a shared log), one for a SQL database (perhaps sharing log file with the Oracle log disk group), and - finally -- a disk group reserved for snapshot images with decent performance (e.g. MirrorClones)

Even though HP recommends Vraid1 for Exchange, we have elected to configure Vraid5 to maintain a decent comparison with the CX4 configuration above.  Also in the spirit of fair play, HP recommends a 20% snap reserve for Exchange.  We've elected to make this 10% to maintain a rough comparison with the CX4.

Hot Spares -- The hot sparing scheme is somewhat unique on the EVA -- there's no concept of a global hot spare.  Hot spares are associated with individual disk groups.  And, since everything is "virtual", HP recommends required-case hot spare provisioning in the event that a user later elects to use, say, Vraid1 instead of Vraid5. 

This means that the concept of a "virtual" hot spare has to be twice whatever the largest disk in the group might be, and you need one of these per disk group. For example, if 146 GB drives are used, the virtual hot spare area is 2x146 GB.  If a single member of the disk group is, say, a 450GB drive, the virtual hot spare area must be 900GB.

There's an additional level of hot sparing per disk group as well, "Proactive Disk Management", which plays roughly the same role a second hot spare would in a global design.  Unfortunately, this too is associated per disk group, and must be twice the size of the largest disk in the group.  Each "Proactive Disk Management" virtual spare protects against a single "event" (e.g. disk failure) in a disk group.  If you want protection against more than one event, you'll need more of these.  We configured to protect against a single event here.

HP folks, if we got this wrong, our apologies in advance.  You'll have to admit, it's a pretty intricate scheme.

If we go back to our configuration example, this would be a total of 14 virtual hot spares and 14 virtual proactive disk management spares would be required for 7 disk groups.  Fewer disk groups would require fewer virtual hot spares.  No mention is made of potentially using larger capacity drives solely for the purpose of virtual disk hot spares, but I would assume that this would make for more difficult administration.

Snapshots -- This analysis uses HP EVA's "Virtually Capacity Free" snapshots, which require close to the same amount of capacity as a CX4 would.  It should be noted that CX4 snapshots reside on separate disks to minimize production performance impact. 

For the EVA to match this, HP must use "fully allocated" snapshots which would further impact usable percentages more than what is shown here.  Again, in the spirit of fair play, we’ve given the EVA the benefit of the doubt on this one.

RAID Parity -- The EVA offers two parity RAID choices: 4+1 Vraid5 and 4+4 Vraid1.  For consistency, this study uses Vraid5, even though HP recommends the less space-efficient Vraid1 for many performance-intensive environments.

As a result, for there to be 120 data disks, we'll need another 30 parity disks -- plus protection for the other overhead drives.

Overhead -- EVA makes five copies of its OS and configuration data which it distributes among the first five disk groups.  At least one of the disk groups must be reserved exclusively for log files, although the smallest disk group supported is eight drives.  Interestingly enough, no single log file exceeds more than one disk, meaning that seven of the disks are unusable, probably for performance reasons. 

Finally, to allow EVA to complete its housekeeping in a reasonable time, EVA best practice recommends a further 10% of each disk group be given to the EVA operating system, known as "Occupancy Alarm".

Even with giving the EVA the benefit of the doubt in several areas, we still arrive at a 47% usability factor. 

I think we're being generous here, though.  Anecdotally, we get routinely exposed to EVA customers who have much lower usable capacity basd on specific HP recommendations -- sometimes as low as 33%.

For those of you counting physical space, this is 17 drive shelves.

Here's the chart.

 

 

 

 

 

 

 

 

 

But there's more to this than just efficiency. 

If you assume that, for example, that CX4 and EVA raw capacity is priced roughly the same, that means that every unit of usable EVA capacity is roughly 63% more expensive than the same capacity on a CX4.

And that's without factoring things like power, cooling and floor space. 

Sobering, isn't it?

NetApp FAS Series -- 34% Storage Capacity Efficiency

Although the FAS series can be used in block-oriented environments such as Exchange, it does so by emulating a block storage device on top of its underlying WAFL file system.  WAFL only performs well when it is guaranteed that free blocks are always available for new writes and snapshot data.

When it is used in this manner in high change rate environments (such as Exchange) and snapshots, NetApp often recommends significant amounts of storage overhead to ensure reliable operation and acceptable performance.  This is often called "space reservation" or "fractional reserve".

Hot Spares -- For the FAS series, every disk that is not being used can be considered a potential hot spare.  NetApp best practices state that each file head should have a minimum of two hot spares assigned up to 100 disks, then two additional hot spares for every 84 additional disks.

For a 364 disk configuration, that means 2 for the first 100 drives, and 8 disks for the remainder.

Source: NetApp's Storage Best Practices and Resiliency Guide

RAID Parity -- FAS supports two parity RAID modes, RAID 4 or RAID-DP (RAID 6). NetApp's default is RAID 6 for these environments, organized as a 14+2 scheme.  In our 364 disk configuration, this rounds up to 23 RAID groups, or a total of 46 parity drives.

Snapshots -- As mentioned above, WAFL doesn't want to run out of space when using snapshots.  The question of precise figures required for an Exchange environment seem to be a subject of controversy both inside and outside of NetApp -- some documents point to a 100% reserve space recommendation, others suggest that it's reasonable to get by with a lesser amount. 

One thing is extremely clear -- running out of snap reserve looks to be a very bad thing in a NetApp environment -- there's no place to put an incoming write, usually resulting in catastrophic application failure.  By comparison, other approaches (e.g. CX4 and EVA) simply stop trying to capture before-write data if you run out of reserve -- the application itself continues to run.

From the Netapp "Microsoft Exchange Server 2007 Best Practices Guide", Fractional Space Reservation

"It is recommended to have a 100% space reservation value set for volumes hosting LUNs containing Exchange data.  This guarantees sufficient space for all write operations on Exchange data volumes and ensures zero donwtime for Exchange users". 

And "Block Management with Data ONTAP 7G: FlexVol, FlexClone, and Space Guarantee"

"It is extremely important to keep in mind that a change in fractional reserve percentage might result in the failure of a write operation should the changine in that LUN exceed that percentage (frequent monitoring of space is recommended).  Therefore, one must be sure of the change rate of the data in the LUN before changing this option.  As a best practice, it would be best to leave it at 100% for a time".

I guess you've been warned ...

I would assume that this same sort of recommendation would apply to any write-intensive application: SAP production instances, Oracle transactional applications, and so on.

Overhead -- WAFL has a high amount of space overhead, which comes in handy for a variety of reasons, but comes at a price.

First, WAFL must "right-size" (format) all drives to the same lowest-common-capacity denominator if drives come from different vendors. 

Second, since WAFL is a file system there's overhead for the metadata that all file systems require. 

Thirdly, the WAFL design wants 1 in 10 blocks to be free so that application writes aren't delayed, this means an additional 10% of reserve at all times.

And, finally, there's another allocation for "Core Dump Reserve" should NetApp customer support want to take a look at an ONTAP core dump.

Here's the chart:

   

 

 

 

 

 

 

 

This means that the NetApp FAS series achieves 34% storage efficiency as compared to CX4 for these configurations.  If you're counting rack space, this results in 26 disk shelves.

And, for this use case, this means that a given unit of usable capacity on NetApp's FAS is approximately 2x expensive than a more capacity efficient device, such as a CX4 for these kinds of use cases.

And again, that's not counting power, cooling and space.

Maybe you should ask them for a better discount :-)

So, What Does All Of This Mean?

First, I think there's enough variability in capacity efficiency that -- just perhaps -- we ought to start educating storage users on the difference between raw and usable capacities in real-world use cases.

If roughly the same usable capacity costs X from one vendor, and 1.6x from another vendor, and 2x from yet another vendor -- isn't that significant? 

Second, as every IT organization takes a sharper look at power and cooling, the numbers get even bigger, don't they?

Even assuming all the associated controller and enclosure electronics were roughly equivalent in terms of power and cooling (definitely not the case, but assume it is just to simplify the discussion), doesn't it matter that a given usable capacity has X power/cooling requirement from one vendor, 1.6x from another vendor, and 2x from yet another vendor?

Third, I think this storage capacity efficiency discussion is something that other vendors don't really want to talk about.  But I believe that customers will want to start getting into the practice of requesting quotes for usable capacity, configured in accordance with vendors' published recommendations for their environments -- in addition to asking for power/cooling requirements.

My guess is that many vendors will argue that (a) they have features and benefits that offset their inefficient use of space, (b) we're wrong in our analysis, or (c) this really doesn't really matter.

Just to head off the obvious, if there's a claim that your thin/virtual provisioning feature saves storage, or your dedupe feature saves storage, or whatever it is, please point us to where you recommend that customers use these features with performance-intensive and availability-intensive applications.

Regarding the second point, if you think we're wrong on some substantive point, please accept our apologies in advance -- and point us to the correct answer. As long as we're staying within the use case guidelines outlined above -- we'll glad to make the change.

But please, don't try and argue that this isn't an important discussion for customers ...

--------------------------------------

References, if you're interested, for HP stuff:

HP StorageWorks 8000 Enterprise Virtual Array and Microsoft Exchange Server 2003 – storage performance and configuration, EVA BP for Exchange 2003: http://h71028.www7.hp.com/enterprise/downloads/PerformConfig_Exchange2003_EVA8000_1105.pdf page 18 & 19. 

This does not discuss in detail the additional disk group(s) for backup - presumably on FATA drives but does mention minimally 2 disk groups (w/o backup) and more for isolation or performance.

Replicating Microsoft Exchange 2003 over distance with the HP StorageWorks Enterprise Virtual Array and HP StorageWorks Continuous Access white paper: http://h20219.www2.hp.com/ERC/downloads/5983-1550EN.pdf page 92 (shows 4 disk groups)

From HP StorageWorks 4x00/6x00/8x00 Enterprise Virtual Array Configuration Best Practices White Paper http://h71028.www7.hp.com/ERC/downloads/4AA0-2787ENW.pdf:

"Proper settings for the protection level, occupancy alarm, and available free space will provide the resources for the array to respond to capacity-related faults.

"Free space, the capacity that is not allocated to a virtual disk, is used by the EVA controller for multiple purposes. Although the array is designed to operate fully allocated, functions like snapshot,reconstruction, leveling, remote replication, and disk management either require or work more efficiently with additional free space."

"Additional reserved free space — as managed by the occupancy alarm and the total virtual disk capacity — affect leveling, remote replication, local replication, and proactive disk management.“  Figure 3 gives a great picture of things.  Note that in our calculation we separated PDM and Capacity Alarm.  This was to keep things understandable rather than lumping all the factors into Capacity Alarm.  We chose ONE PDM event although this paper suggests two is also a good choice.  For Capacity Alarm we chose a conservative 10% to include all items beyond one PDM event including replication log file, releveling overhead, etc.

This is the key one that HP took down: ftp://ftp.compaq.com/pub/products/storageworks/whitepapers/MSEx2003designEVA5000.pdf. 

It showed 7 disk groups as a best practice for an Exchange 2003 configuration to support 8000 mailboxes.

------------------------------

And some NetApp (and IBM N Series) references to go look at:

Trading Capacity for Data Protection – A Guide to Capacity Overhead

http://www.issidata.com/specs/netapp/SV-CapacityWhitePaper101106.pdf

.5% additional FlexVol reservation

·         OnTap 7.3 Release Notes (p46)

·         http://www-01.ibm.com/support/docview.wss?uid=ssg1S7002459&aid=1

Drive right sizing

·         Data ONTAP 7.3 Storage Management Guide (p31)

·         http://www-01.ibm.com/support/docview.wss?uid=ssg1S7002468&aid=1

100% Space Reservation in block environments

·         Data ONTAP 7.3 Block Access Management Guide for iSCSI and FCP (p38)

http://www-01.ibm.com/support/docview.wss?uid=ssg1S7002473&aid=1

Warning -- if some of these docs disappear after we post this, our apologies.

Comments

[Stephen Foskett]

Hooo boy, Chuck, you do like walking into minefields, don't you? I want this kind of apples-to-apples comparison to work, but it'll never be fair... See my thoughts on the topic:
http://blog.fosketts.net/2008/08/28/grapples-and-tangelos-why-its-impossible-to-compare-fairly/

[Chuck Hollis]

Hi Steve, I wanted to leave a comment on your blog, but didn't feel like signing up to do so.

You brought up some interesting points, as follows ...

Steve writes: Does EMC really support using the five vault reserve disks on a CLARiiON to hold production data?

Answer: yes, we do.

Steve writes: Would EMC really suggest 8+1 RAID 5 for a production Exchange and SQL Server environment?

Answer: yes -- and we've got the test data to back it up. The performance and availability characterizations are publicly available, I think.

Steve writes: Is one hot spare per two DAEs (30 drives) really sufficient for a whole pile of 9-disk RAID 5 sets that are maxed-out with production data? I’d feel much more comfortable with a few more spares with such large RAID 5 sets.

Answer: the configuration has 4 global hot spares, so it's not useful to think of it that way. Also, the proactive sparing algorithm means we usually get to a drive before it totally fails. CLARiiON engineering considers this recommendation "conservative".

However, please add as many drives as makes you feel comfortable. We can always make more :-)

Steve writes: There is no way 14+2 RAID DP is equivalent to 4+1 RAID 5, let alone 8+1. It’s in a different league of reliability.

Answer: although we'd disagree with that statement in the presence of global hot spares, we just went with what each vendor recommended.

Steve writes: Yeah, NetApp’s space reserve recommendation stinks. But you probably won’t need 100% in production - the real amount is something one would work out when testing and piloting and is probably substantially less than this.

Answer: gee, NetApp's pitch is all about simplicity. You mean I have to run a bunch of trials to get to the optimum setting? That wasn't in the marketing deck I saw!

Also, let's not forget that the penalty for getting things wrong is a catastrophic application crash, which is pretty severe ..

Steve, we simply went with what each vendor recommended, and was willing to support. There are no value judgments here on "right" or "wrong".

I know you think I'm putting my neck out there, but I think this is a good discussion to have.

The differences in approaches are striking, wouldn't you agree?

-- Chuck

[ripvan]

Chuck,
if this is really how you would suggest that I as a customer should set up my CX array for production, please configure an array in this manner and suject it to benchmark testing and post the results to

http://www.storageperformance.org

Yes - benchmarks are not perfect, but they are certainly the best source to get some comparitive information about available arrays on the market instead of listening to your local disk peddler.

[Chuck Hollis]

I think you might be misunderstanding what the SPC is all about.

The goal is for vendors to configure their arrays for optimum results on the SPC test, disregarding things like availability, recoverability, real-world application profiles, manageability, cost-effectiveness, etc. It's all about the number, baby ...

Oh so many problems with the SPC. I really don't want to cover all that again. Go look a bit closer, and then we'll chat in detail.

BTW, check out the latest SPC flimflammery from IBM here: http://news.cnet.com/8301-13924_3-10028216-64.html?tag=mncol

Thanks for writing.

[Lee Razo]

Ahhh thanks for that, Chuck. You made my day.

As one of NetApp's competitive analysts, I always look forward to checking in with you to find out what the latest variation of the EMC shell game is. You should seriously consider going into politics.

Why is it that whenever you guys talk about capacity, you manage to artfully avoid any detailed discussion of performance or resiliency? For that matter, the same seems to happen when you talk about performance. Or resiliency. Why is it that any talk of performance, resiliency or efficiency are never seen in the same room together at EMC?

In this case you even casually mentioned in passing that NetApp FAS comes standard-configured with high-resiliency, zero-penalty RAID 6 - as if CLARiiON RAID 5 was even in the same league (I won't even go into the snapshot performance impacts - the real reason why you guys don't like SPC-1)

Don't worry, though, Chuck. The good news is that the EVA is far worse than the CLARiiON by all measures. You were quite generous in your comparison even though the EVA configuration you described would be barely usable after the first snapshot due to EVA's own unique inefficiencies.

Seriously though, I'd be more than happy to give you a few pointers on how an EVA (or a NetApp FAS for that matter) works in the real world.

It'll be fun, we could compare it to the CLARiiON and make a drinking game out of it - you have to take a shot every time there is a caveat in your best practice guide.

--Lee

[Chuck Hollis]

Glad I made you happy. Go back and read the first third of the post again, and you'll get the answer to your first observation.

As I pointed out in the post, the idea behind this is that each vendor makes recommendations for Exchange environments that provide the best combination between performance, capacity, availabilty, recoverability and manageability.

EMC makes ours, HP makes theirs, NetApp makes yours. All three vendors publish recommendations for this particular use case, which I found interesting.

The purpose of this exercise was not to say that any one vendors' recommendations was "bad" or "good", just to look at each of them from the perspective of capacity efficiency.

And in block mode environments, y'all don't seem to stack up so well. Maybe you should stick to smaller filers storing tier 2 data.

Best regards!

[Lee Razo]

That all sounds plausible, but you actually need to choose current, and "like for like" best practices, not just cherry-pick the ones you like.

To name two:

1) If you're going to use RAID-DP on a NetApp then you either need to use RAID-1 on the CLARiiON (and sacrifice half the capacity) or RAID-6 (and sacrifice 60% of the write performance). No such sacrifice needed on the Netapp

2) Your NetApp snapshot information is outdated. You are not required to reserve 100% for block snapshots nor is it necessary to do any kind of "trial and error". We've supported the functionality to automatically manage snap reserve the same way the CLARiiON or EVA would for quite some time now.

Seriously, man. Get with the times! Technology marches on.

--Lee

[Chuck Hollis]

Sorry, nice try, but no cigar.

Like I said before, we went with each vendor's published recommendation. You don't like EMC's and HP's use of RAID 5, you need to change NetApp's standard recommendation of RAID 6. Feel free to argue your case that it's "better", but that's not the point here.

If NetApp has updated its recommendations for Exchange environments in terms of snap reserve, please point me to the appropriate documentation, and I will be more than glad to update this piece.

Maybe technology marches on, but perhaps you need to update your Exchange recommendations?

Most of your customers default to what the vendor recommends, which costs them big bucks, which is exactly the point of this comparison.

Cheers!

[TomT]

The flaws in your analysis of the EVA are so blatant that reviewing just one of the documents you cited will make this abundantly clear. You have disregarded most of the key recommendations presented in "HP StorageWorks 4x00/6x00/8x00 Enterprise Virtual Array Configuration Best Practices White Paper". These recommendations are not easily missed - they are repeated multiple times in the text, listed in tables, set off from the main paragraphs and even highlighted in color.

While there are many, the oversight that invalidates your EVA analysis more than any other is your use of disk groups. Configuring 7 disk groups is contrary to one of the most fundamental best practices for the EVA. As clearly stated in the aforementioned document:

Best practice to minimize the cost of storage: Create as few disks groups as possible
Best practice to optimize performance: Configure as few disk groups as possible

Besides contradicting HP's recommendations, configuring an excessive number of disk groups is not representative of real world EVA implementations. The vast majority of EVA arrays ever implemented have been configured with one or two disk groups.

Your use (or misuse) of disk groups dramatically skews the capacity efficiency calculations to the point of being ridiculous.

My advice to you....next time RTFM.

[Stephen Foskett]

I've had EMC SE's tell me that the vault reserve disks should only be used for low-I/O static data or not at all to avoid system-wide performance impacts. Were they wrong?

I've also had them tell me that one should configure twice that many spares at least... Overly cautious?

I love the idea of trying to set up an apple-to-apples comparison based on usage, but think EMC's not the best person to make the judgment. How about we have a third party set up a scenario and let each vendor respond with their own proposed configuration, using their own knowledge of best practices and experience to decide how many drives to reserve for this and that?

[Chuck Hollis]

Hi TomT -- would you mind posting a public link to that HP document?

It'd be very helpful -- thanks!

As far as the number of disk groups we used, we saw multiple references to the need to isolate different workloads on the EVA for performance reasons.

At 120 (usable) disks, it's highly unlikely that it'd be all supporting a single application, wouldn't you agree?

If you go back to the post, examine the specific use case we outlined, and have a different number of HP-recommended disk groups, would you mind sharing that here?

Thanks for your comments.

And, trust us, we did RTFM -- in detail.

-- Cheers!

[Chuck Hollis]

Hi Stephen

Yes, that used to be the recommendation, but no more. My understanding is that these drives are used infrequently. I'll get someone more authoritative to back me up on this in a moment.

As far as over-configuring by field personnel in the interest of excessive caution, it's a phenomenon that's probably not limited to EMC. We publish engineering-supported recommendations for customers and field personnel alike. Sometimes people think they know better :-)

As fas as having a third party do all of this -- yes, you have a point -- but no obvious candidate springs to mind.

And we don't want a replay of the SPC fiasco.

Thanks!

[TomT]

Chuck,

The Best Practices document to which I referred is the same one you listed under HP references....you know....the one you read...in detail.
http://h71028.www7.hp.com/ERC/downloads/4AA0-2787ENW.pdf

When I have some time I'll see about addressing your other questions....gotta take off.

TomT

[Chuck Hollis]

Hi Tom T:

Good, so we're talking about the exact same document!

So, do you want me to cut and paste the sections we read in it regarding the need for performance isolation between different disk groups (hence leading to our configuration of 7 disk groups for 120 usable disks, which HP users tell us is typical), or would you choose to pursue another angle on this?

Let me know ... thanks!

[Jen LeClair]

The first four drives of the vault contain the SP’s operating system. After the storage system is booted, the operating system files are only occasionally accessed. Vault drives may be used for moderate host I/O loads, such as general-purpose file serving. For these drives, restrict usage to no more than shown:

MAXIMUM HOST I/O LOADS FOR VAULT DRIVES
Vault HD Type Max. IOP Max. Bandwidth
Fibre Channel 100 10
SAS 100 10
SATA 50 5
SSD N/A N/A

In general, when planning LUN provisioning, distribute busy LUNs equally over the available RAID groups. However, if LUNs are hosted on RAID groups composed of vault drives, when planning assume a busy LUN has already been provisioned on the drives. This will account for the CLARiiON’s vault drive utilization.

For example, assume five equally busy LUNs need be provisioned. The LUNs need be allocated to three RAID groups. One of these RAID groups is necessarily composed of the vault drives. Place two of the new LUNs on each of the two non-vault drive RAID groups (for a total of four LUNs). Place a single new LUN on the vault drives RAID group. Why? Because, before the new LUNs are provisioned, the RAID group built from the vault drives already has a busy LUN provisioned from the CLARiiON.

Recommendations for AX4-5 and earlier CLARiiON storage systems:

For AX4-5 and the earlier CLARiiONs, the CX4 considerations above apply. The reserve LUN pool, clone, and mirror LUNs should not be placed on the vault drives.. Also, avoid placing any response-time sensitive data on the vault drives.

A vault drive failure will normally result in rebuild activity on these drives and a disabled write cache. The HA vault option in Navisphere should be clear, if response-time sensitive access to data on these drives is needed under this condition. With the HA vault disabled, the cache stays enabled during a vault drive rebuild, at the expense of a small amount of protection for cached data. This is especially important with systems using SATA drives in the vault, due to the long time needed for these drives to rebuild and for the cache to re-enable.

[BrianS]

I would be glad to post the link to HP EVA best practice guide.

http://h71028.www7.hp.com/ERC/downloads/4AA0-2787ENW.pdf

As for the definitive answer on the EVA disk groups. Best practice is to always have as few disk groups as possible. Typically heavy sequential versus heavy random is split which typically means transaction logs get moved to a different disk group. Which is an availability best practice. I have deployed dozens of customer EVAs and typically no more than 1 or 2 disk groups are used or needed. The magic of the EVA is the ability to stripe across all of the spindles in the disk group. This obviously reduces array group specific hot spots that traditional arrays have to deal with especially when workloads change unexpectedly, however in very specific cases requirements may dictate guaranteed service levels of a certain workload and then it may warrant a deviation from this practice. In the case with exchange 2003 this is evident on the lack of heavy database caching due to the 32bit nature of the code base. This requirement then translates into direct requirements for transaction response times needed from the array to provide a reasonable user response. Exchange 2007 greatly changes the I\O requirements with the move to 64bit. Rule of thumb is that IOPS\user requirements on Exchange 2007 went from 1.0iops\user to as low as .3iops\user. This represents a significant change in exchange planning and reduces the hard requirements for response times. Here is the EVA Exchange 2007 best practices. http://h71028.www7.hp.com/ERC/downloads/4AA1-4704ENW.pdf

In the end in an EVA it comes down to IOPS. And an EVA has a big bucket of IOPS for mixed workloads to fit into then several smaller buckets of a traditional array.

[Chuck Hollis]

Thanks Jen!

[Chuck Hollis]

Thanks, Brian, very useful.

All storage arrays are "big buckets of IOPs", right? And differences in I/O profiles between Exchange 2003 and 2007 pretty much play out for everyone the same way.

My impression is that most EVAs are somewhat smaller than the configuration discussed here, where your "1 or 2" comment might be appropriate.

As mentioned in the post, please consider a singificantly larger configuration (e.g. 120 usable disks) where you have multiple performance-intensive applications and desire a degree of workload separation.

Would you offer up a different suggestion?

Thanks

[John Ashton]

Chuck,

Funny how you gloss over anything about a mixed workload requirement on the CX array. You simply state that there are best practices for other applications and foolishly claim no such practices exist for competitive products. Of course this claim can be easily dismissed by anybody with access to Google and half a brain.

Try plugging in "SAP EVA best practices" or even "SAP NetApp best practices" for example.

Way to increase your credibility Chuck! So now customers are going to think you are an authority on the competition?

I certainly appreciate the fact you decided to include multiple workloads on the arrays in question. After all this is the sort of thing most benchmarks overlook. They quaintly assume if I buy an array I only run one application at a time on it. Perhaps if my SAN had the same price point as those pesky servers but I digress...

So now we have three arrays running Oracle & SQL workloads.

Just one problem.

You didn't do the research to see what happens when you actually add these workloads.

It is so much more fun to extrapolate isn't it? I mean after all the requirements for Exchange are idneitcal to Oracle and SQL right?

You don't apply EMC best practices for these additional workloads. Heck your own engineers would chastise you for doing this...

So at the end of the day your "efficiency math" is based on all disks allocated to Exchange for the CX.

But wait a second. What about the competition?

Can we penalize them? You betcha!

Watch as the "comparison" unfolds...

One glaringly obvious one I just have to ask a question about:
That lovely EVA multicolored chart has a "file share". Wait a second is this a typo? After all I could have sworn you wrote:

"Conversely, I don't think it's reasonable to extend this sort of comparison to file serving"

So some clarification would be nice...

So many questions Chuck...
How did you come up with the workload?
Are you applying EMC best practices to the additional workloads on the CX?
Why is there no mention of Oracle or SQL workloads (and that pesky file share) for either the NetApp or EMC arrays?
What sort of configuration are you optimizing for here? Availability? Performance? Cost?
Were these configuration goals communicated?
Were the same goals applied for each array?

As you can see plenty of questions an unhealthy amount of spin and very little in the way of answers.

Your title is VERY accurate....your user capacity may vary...

Especially with:
* ambiguous goals
* a lack of transparency
* a fantasy marketing comparison v. real world configurations

But hey, I'm not a VP just one of the schmucks that has to make sure all of this stuff actually lives up to the hype spread by marketing types to clueless executives...

What's your excuse?

[Chuck Hollis]

Hi John

Although you might be making some valid points, it's very hard for me to get past the snide, insulting and sarcastic tone.

Is this the way you routinely speak to people you interact with?

[Scott Fuhrman]

It's important to note the noticeable difference in usable disk space on a Clariion vs. a Symmetrix.

Take two identical, 300GB FC drives.

On CX arrays, we get 288,195,870,721 bytes, or 268.40332 gigabytes usable per 300GB drive.

On DMX arrays, we get 585,937,499 blocks * 512 bytes/block, equaling (299,999,999,488 bytes) 279.396772 gigabytes per usable drive.

So on a CX, we lose 11GB more per drive that we do on a Symm. That's 3.7% right off the bat. It's important to include the details.

[Chuck Hollis]

Ok John -- you deserve a response.

1. Your first point stems from a less-than-careful reading of what I wrote. I simply said many vendors don't supply this material. I did not claim that HP nor NetApp were in this category.

Yes, I've seen the docs you reference. The HP one is good, the NetApp for FC strikes me as fluffy.

2. EMC has characterized multiple workloads against all of our arrays, it's an essential and ongoing part of our business model. Many mountains of detail to share here if anyone is interested ...

We have not done the same for competitive arrays in a consistent fashion, though. For this capacity exercise, we focused on the need for workload isolation. In the cases of CX and FAS, there was no capacity impact. For EVA, alas, workload isolation tends to cost you in terms of capacity efficiency.

3. The error you point on the pictures is my fault. I grabbed an earlier version of the GIF file (same numbers, different headers), I will go fix that -- thanks.

4. As far as consistency in workloads between the three, I could have been more clear here.

In terms of configuring for availability and protection, our Exchange recommendations are almost identical to our Oracle, SAP, et. al. Use it all for Exchange, use it for a mix of 8 different applications which all require decent performance and availability (as well as workload isolation), all the numbers regarding capacity efficiency are virtually the same.

5. In terms of goals, I think the post is self-explanatory: configure 120 disks of usable capacity for a performance-sensitive and availability-sensitive environment (e.g. Exchange), and see how capacity efficiency stacks up. Since all three vendors have relatively current recommendations for Exchange, we have a rough standard between the three.

If it was all about performance, there'd be different recommendations from all three vendors. If it was all about cost, same thing. And if it was all about uber-availability, ditto.

The point I'm making is that each vendor chose their optimum saddle point between the three. You're free to argue that various vendors are making the wrong tradeoffs with their recommendations (e.g. Stephen Foskett's point), but that's a separate discussion, isn't it?

Please go re-read what I wrote, and, if you still have concerns, please let me know.

-- Chuck

[Martin G]

Okay, user here!!! We have just been through an exercise comparing various vendors and one of the metrics was how efficient each vendor used RAW disk for capacity so that we could work out how much it would cost to write a true terabyte of useable DATA to a disk. Comparing Raw disk costs is a mug's game and although I would struggle to hit Chuck's figures for NetApp; we got it down to about 50% for NetApp (including Waffle overhead, Snapshot Reserve of 10% (I think, don't have my calculations with me, Hot Spares, RAID-DP etc). For EMC, we were pretty close to Chuck's figures. To be honest, IBM's figures for DS4xk weren't far off 60 odd-percent either.

We have DMX, CX, FAS and DS all on the floor and they all do a job. I'd always try to avoid using the FAS as a Fibre-Channel array, it's not its sweetspot. And for NAS, we are leaning to a V3170 in front of a CX4. So you both win ;-)

[Chuck Hollis]

Thank you Martin!!!

Now, if you could go back and check your FAS numbers and include all that RAID-DP stuff and other overhead? It'd be *very* useful -- thanks!

[Chuck Hollis]

Hi Scott -- interesting findings between DMX and CX.

Part of that (1.5%?) is due to the difference between the 520-byte sectors used on the CX, I believe. Don't know where the rest of it is coming from, though. I'll go ask one of our superduper experts about this ... because I'm interested as well.

Thanks for writing!

[Chuck Hollis]

TomT:

Ah yes, I found the passage in the HP EVA documentation I was looking for.

According to HP documentation, there is no advantage to workload isolation on the EVA (something I would seriously doubt), but there is an availability advantage.

Page 13:
"Best practice to optimize availability: For critical database applications, consider placing data files and recovery files in separate disk groups."

"Best practice to optimize availability: Assign snapclones to a separate disk group."

Still, if I've got application A lighting up all the spindles, wouldn't I want to keep it separate from applications B, C and D?

Or is the design point of the EVA more egalitarian, e.g. no way to manage QoS on the device?

Thanks ...

[John Ashton]

1) No I don't think I mis-read your content. You stated:

"Although many vendors don't publish recommendations for other high transactional-rate applications such as Oracle, SQLsever, SAP etc. (EMC does, though) I think it's reasonable to extend Exchange findings to these use cases."

As I pointed out you can find recommendations on the applications mentioned for the vendors compared.

2 & 5)

You stated:
"For this capacity exercise, we focused on the need for workload isolation. In the cases of CX and FAS, there was no capacity impact. For EVA, alas, workload isolation tends to cost you in terms of capacity efficiency."

Then you go on to state:
"If it was all about performance, there'd be different recommendations from all three vendors. If it was all about cost, same thing. And if it was all about uber-availability, ditto."

Workloads can be isolated in different ways. Using an EVA you could create separate disk groups (as you did) or vDisks which would be the equivalent to LUNs on an EMC or NetApp environment.

vDisks still provide isolation for workloads and since all performance is striped across all disks it provides a good balance of performance and isolation. If you were trying to achieve a good balance between isolation and performance fewer disk groups would make sense. The point here is that the workload is assumed to fit in the performance boundaries of each array which means building multiple disk groups is a choice - not a requirement.

So why not use vDisks instead of disk groups?

This would allow for a better "apples to apples" comparison. Of course I am assuming that was the point of this exercise and not simply maximizing your positioning...

You can argue that best practcies should be updated and everyone could probably stand updates. After all customers are moving to Exchange 2007. Still is the point to compare disk arrays or best practices?

While it may not be optimized for the absolute in performance you clearly state:

"In terms of goals, I think the post is self-explanatory: configure 120 disks of usable capacity for a performance-sensitive and availability-sensitive environment (e.g. Exchange)"

Which means ultimately you have multiple competing goals which require balance. If the true goal was just about performance you would have used RAID 0. If it was truly about availability you would have used RAID 1.

[Chuck Hollis]

Hi John

1. Point noted.

2. There seems to be some debate as to whether vDisks provide true workload isolation, e.g. IOs from application A will never interfere with IOs from application B. Your thoughts?

3. Maybe one could argue that we used too many disk groups on the HP, but there's an availability aspect to this as well, as noted in previous comments.

If it were you, and you looked at the brief description I provided for the environment above, how many disk groups would YOU configure in a 120-usable-disk EVA?

Thanks.

[Stephen Foskett]

A-ha! Jen LeClair supplies the missing info: "avoid placing any response-time sensitive data on the vault drives." That's what I was remembering, and probably what the EMC SE's were referencing.

Although this discussion is seriously flawed because it takes place on a vendor's blog, I think it's an incredibly valuable one to have. The fundamental point is correct: No two arrays are alike enough to compare on a numerical disks-and-IOPS basis. As is Chuck's comment that each vendor makes their own decisions regarding performance, cost, and reliability (pick two, right?)

The only valid comparison comes from adequately specifying your technical requirements and having each vendor suggest their own solution. EMC can never make NetApp or HP happy suggesting how to use their systems, as evidence from the predictable responses here.

[Chuck Hollis]

Stephen, now you're understanding my motive here -- I want knowledgeable users to ask informed questions from their vendors -- and demand intelligent responses -- nothing more.

Unfortunately, the only tool I have at my hands to do this is my (flawed) vendor blog.

-- Chuck

[John Ashton]

2. The architecture of the EVA allows you to spread all IO traffic across every spindle. The traffic is isolated at a physical level.

Further isolation can be had by creating disk groups. These groups essentially put a fence around physical drives. Allowing traffic to run across all drives in a group but not the array.

Creating a vDisk simply abstracts the isolation from the physical boundary. Think of it like striping together multiple physical disks into a LUN.

Since most people don't know what to do in this model, carving an EVA up into multiple disk groups is the easiest choice.

The EVA 8000 actually maintains enough performance to meet the latency requirements of Exchange 2003 in a single 120-disk array group. If you wanted to create additional isolation you could create a different vDisk for logs rather than another disk group. It all depends on the log transaction rates, mailbox sizes, items per mailbox, etc.

Since you didn't provide any performance attributes for additional workloads I might build a disk group to isolate Oracle & SQL workloads with vDisks for data and log space in a disk group.

But then again all of this is theoretical...

[Chuck Hollis]

Thanks, John, I was starting to doubt my understanding of the EVA a bit there ...

If you want "hard" IO isolation, you create disk groups. Or additional availabilty, per HP's documentation.

If a given disk group is presumed to have enough IO and deliver short enough response times for all applications, carve it with vDisks for different apps. No IO isolation here, though ...

If not, additional disk groups are called for.

And that's where it gets subjective, no?

Thanks!

[Bibble Chubber]

Chuck, you make me laugh!

[Joe]

Hey Lee,
Based off of the system I have, using defaults, when I create a 16-drive RAID DP using 500GB SATA drives I only get 4.07TB usable. Do the math. That's about 50% of "raw" that is usable before even considering spares. That sucks. NetApp is pathetic.

As for "If you're going to use RAID-DP on a NetApp then you either need to use RAID-1 on the CLARiiON (and sacrifice half the capacity) or RAID-6 (and sacrifice 60% of the write performance)" are you joking? Seriously, NetApp NEEDS RAID DP because the drive rebuilds are SO long. How long does it take to rebuild a 1TB drive? on a NetApp box? Your best practices say to use RAID DP for any drive greater than 146GB. Are you kidding me? That's ridiculous. You are the ONLY company that demands "RAID 6" protection for FC drives. What's wrong with your architecture? Could it be that WAFL is REALLY old and just doesn't work well in a FC/iSCSI SAN?

And please don't reply with "We have Veritest reports showing how fast we are". Everybody knows they were baked.

[John Ashton]

You nailed it.

"If a given disk group is presumed to have enough IO and deliver short enough response times for all applications, carve it with vDisks for different apps"

Then again if you are meeting your target do you really need isolation? Think of it this way:

You have 120 146G drives
Select your sparing and RAID level to subtract the overhead.

Now you decide you need some IO isolation. How would you do this on a traditional array?

String together disks into a LUN. Lets say 16 146G drives. Dedicate your spare disk in the LUN (leaving 15) and off you go.

Run into a performance problem and what happens?

You could always re-build your stripe to more closely emulate your traffic after all most people grossly over/under estimate performance. But hey that's messy and takes a lot of time. In most cases we take the easy way out and add spindles.

Now can you add more disks to that single LUN?

Maybe, maybe not.

If you can, great. If not, now its time for another LUN and some sort of concatenation software, presentation, putting down volumes and or file systems etc. Basically all the stuff we hate to do and give to server admins if we can help it...

Now consider this...

That single LUN has a performance boundary right? 16 drives. It is only going to be as fast as those 16 drives spin - ever. By the time you add all the pesky stuff mentioned above you managed to slow it down...

Well what if you didn't have a LUN limitation? What if you could make a LUN as big as your array? How fast would it be?

Using the same laws its just as fast as the physical disks. So if you have 120 disks that is as fast as it will ever get.

Here's the bottom line:

Not using disk groups does not mean you don't have isolation. It simply means that isolation is linked to the array rather than groups of disks.

[Mike Shea]

Chuck,

As a former EMC employee, and a current NetApp employee, I have to say that this post just has me wondering what has happened.

There was a day where if a person 'in the know' said something (like Lee Razo above on teh topic of NetApp SnapShots and disk sizing) spoke, you accepted the professional expertese and moved on.

That is what made you such a steller EBC paritcipant in the past. You used to stick with what was important and out of these crazy ratholes... or is it a minkhole now?

What happened to you sir?

[Joe]

"I want knowledgeable users to ask informed questions from their vendors"

OK.
-- What happens when a vault drive fails?
-- What happens in the case of a double drive failure (0,0 & 0,2 or 0,1 & 0,3)?
-- What happens if 0,0, 0,1 & 0,2 all die. Rare, but what happens?

[Calvin Zito]

Thanks Chuck for stimulating an examination of EVA Capacity Efficiency. We've posted a complete response on our blog at http://www.communities.hp.com/online/blogs/datastorage/archive/2008/08/29/emc-distortion-about-capacity-efficiency.aspx

Calvin

[Ahmad]

John,
I was a CE working with all Digital and Compaq products, I had done a Demo for customer on MA8000 (which I love) and he was extremely happy with the performance, when the customer bought EVA, he ran into rage...
Simply, give me a decent configuration for a small EVA with 18-24 disks to run a small oracle, exchange and small SQL..

EVA is the right equipment for a shop that has no problem with down time and enjoy easy configuration..

I remember that I had to bring an EVA8000 down to upgrade the Disk drives firmware...The other alternative, was to remove each disk from the disk group to upgrade it seperately (you know that this can take a month)

Please search on your knowledgebase, and let me know how many times you loose your qourum data (which was something like 16 copies when I was working with EVA)...

BTW, I was ASE Storage

[Ahmad]

one more input in the subject,

Chuck's team are not right to choose 7 DGs as it is not recommended by HP training courses and BP documents.

96-120 Disks in a DG is the optimum number (according to one of the HP fellows if I recall correctly)

BUT, I am wondering how much time such a disk group needs to be re-leveled after a failure or after adding new disks? How will this impact the availability as Chuck stated..

http://forums12.itrc.hp.com/service/forums/questionanswer.do?admit=109447627+1220062834862+28353475&threadId=990314

[MarinaG]

Hmmm... so it's OK for you to be snide, insulting and sarcastic, but you're insulted when posters respond in kind? If you make provocative statements, you should expect provocative responses.

As a VP I'd think you'd have a thicker skin. Or be less defensive.

Take the high road Chuckles. :-)

Cheers!

[Chuck Hollis]

MarinaG

There are different standards of conduct in the real world, for example, some people are completely socially inept.

The same holds true for online interaction.

I do not insult people, belittle them, call them names, or personally attack them.

I wish other people would do the same.

[Chuck Hollis]

So, everyone, in regards to our HP numbers, the real issue is EMC's use of too many disk groups in the configuration.

We were lead to believe from HP documentation that if you wanted performance isolation and availability isolation between multiple applications, you need multiple disk groups to achieve this.

A few commenters agreed with this interpretation. HP does not, though.

If we've configured the EVA inappropriately, our numbers are wrong, and we need to go fix that.

I'll get back to you soon -- thanks!

[Geert]

You mind publishing links to your best practice guides you seem to have taken your data from? You are happily asking others to provide these, but for some reason you've failed to do the same for your own (I'm not a native English speaker, so I don't remember that saying about the pot and the kettle... sorry).

And as for accurate reading (and presenting), here's a correction on your hot spares calculation for FAS (taken from the NetApp's Storage Best Practices and Resiliency Guide you mention, to be found here: http://media.netapp.com/documents/tr-3437.pdf), it states the following:

- Maintain at least *one* hot spare for each type of disk drive in the storage system
- Maintain *two* hot spares for each type of disk drive in the storage system to take advantage of Maintenance Center.
- NetApp recommends using two spares per disk type for up to 100 disk drives. For each additional 84 disks above that, another [*not* two] hot standby disk should be allocated to the spare pool. In a 364 disk configuration this means ~5 hot spares, *not* the 10 you mention [check the table on page 13 of the guide].
- Aggregate snapshot creation can be disabled and the reserve lowered, unless a MetroCluster or SyncMirror is used, which is not the case in your example.

That's it for now, I won't even go down the path of dissecting all other inaccuracies in your "independent" comparison, not will I go into all other aspects of space savings, utilization & performance increasing and data protection functionality FAS systems brings to market; I'll save that to my chats with customers and prospects.

[egenda99]

Chuck,

I think the true answer with NetApp is more like 399 drives for 17.5 TB usable. Quick point - 144 GB NTAP drives delivers rightsized 119 GB usable (some bloggers say up to 125 GB usable). I am pretty sure that 100% Snap Reserve is AFTER parity calcs and aggregate and snapshot (10%) overhead. So you need to get to 35 TB "raw/usable" to get to 17.5 TB "actual/usable". It takes about 399 drives @ 119 GB with 14+2 parity and a spare every 84 drives to hit your 17.5 TBs which is 120 @ 146 GB. Oh yea, you now have to manage 3 volumes/aggregates since you maxed out the 16 TB (RAW) limit.

[pete mitchell]

Chuck,

You're such a tool.

[Chuck Hollis]

Egenda 99 -- thanks for the insight, we'll correct things for the future.

[Patrick]

Hello Chuck,

mmhh... Do not trust any statistics you did not fake yourself. ;) Did you read the best practise guide for the EVA or did you only posted the link to it? A EVA with 7 diskgroups is far away from reality. If you create a diskgroup for any application, you lay a traditional RAID scheme over a system, that isn't designed for it. If you create more diskgroups it's clear that you waste a lot of space for the disk protection. But this kind of disk protection is much better, then traditional spare disks. Spare disks are a waste of space! If the EVA has enough space left in the disk group, it's using first the free space in the diskgroup, and then the reserved space (=space that is reserves with the disk protection setting). The occupancy level of 90% is to much, I often use 95% and it's safe. You should create as few diskgroups as possible. It's not big deal to run Oracle and Exchange within one diskgroup, also in large enviroments. It is obvious that fewer diskgroups with more disks are the best way. Fewer diskgroups > more disks in each diskgroup > more performance for the application > fewer space wastet due protection level.

So next time you better read the docs, instead of posting a link to it. ;)

Best regards.

[Chuck Hollis]

Hi Patrick

We now realize that 7 disk groups are perhaps excessive, but -- based on our interpretation regarding HP's recommendations and realistic concerns about performance and availability isolation, the right answer might be more like 3 or 4.

We'll be updating things shortly. Thanks.

[Martin G]

Just a quick post, I recently had some sizing done which was for pure capacity and how many spindles I would need to give me 400 Terabytes of useable space (Base 2 Terabytes) on a FAS. To give me 400 Terabytes of useable disk using 1 Terabyte drives would take 840 spindles made up of

689 data spindles
116 parity spindles
29 hot spares

coming to 834 spindles

which was 60 shelves of 14 disks.

This sizing was done by the NetApp account team. It was a pure capacity exercise and takes no account of the underlying performance requirements of the application, no account of Snapshots, no account of dedupe but simply how much data can I get on an array.

I wonder if the other vendors following this could carry out a similar exercise. 400TB of data on 1 Terabyte drives and I have an availability requirement of five nines (so if you think RAID-5 on 1 Terabyte drives is going to cut it...I might look askance).