« The CFO Of Information Is Gonna Need A System | Main | Off The Grid For A While »

July 15, 2008

TrackBack

TrackBack URL for this entry:
http://www.typepad.com/services/trackback/6a00d83451be8f69e200e5539ffd828833

Listed below are links to weblogs that reference The Great Data Placement Debate:

Comments

Martin G

Been having a few conversations recently about this very subject, the impact of larger disks (with less IOPs available per terabyte) and the impact of fast flash disks and what this really means going forward. I am coming to the conclusion that we need two tiers of disk, I call them 'Too Good Perfomance' and 'Just Good Enough Performance'. You might want to consider them the performance tier and the capacity tier.

A high degree of intelligence and autononmic behaviour needs to be built into the 'array' (an array being a single array or perhaps a number of virtualised arrays) with chunks of data being moved around based on I/O characteristics. This might be on a Lun level or a sub-lun level; perhaps it is almost time for us to stop talking about LUNs? You might even decide to move data on observed behaviour; you know that a certain piece of data becomes hot between 2 and 3 in the morning; so you move it then and move it back once its hot period is over.

Not asking for too much am I?

Chuck Hollis

No, you're not asking for too much ... ;-)

kostadis roussos

Hi!

Chuck, I think your argument is flawed because it's based on four flawed assumptions:

1. Humans can always do better than software in a dynamic environment.
2. Storage virtualization can not perform.
3. Orthodox architectures are the only right answer.
4. Automatic and Optimized Data Placement only adds value as a performance optimization.

You can check out a longer version of my rebuttal here:

http://blogs.netapp.com/extensible_netapp/2008/07/chuck-almost-ge.html


Chuck Hollis

Hi K --

I saw your rebuttal, and I think you're still missing the mark. I see the hand waving and protestations, I don't see the facts.

Tell me, exactly, how a NetApp shared device would automagically detect a high I/O stream, such as an Oracle transaction log, and direct it to the appropriate device?

With the current state of storage technology, don't we need a human being to understand that situation ahead of time?

Sure, I'll give you and your architectural brethren the benefit of the doubt on data that you create, e.g. snaps et. al. But that's just part of the picture, isn't it?

Now, play along with me for just a moment.

Let's assume that EFD is very, very expensive. And that its benefit is very very great.

Wouldn't you want a mechanism to diret the storage traffic that could benefit from this performance towards EFD, and traffic that doesn't need it away from it?

Let's face it -- you guys can't do this simple, yet fundamental trick at a LUN or other object level, can you?

Live by your architectural decisions, die by your architectural decisions, I'd offer.

Yes, you've got a hunk of flash sitting on your PCI bus. And you've waved your hands and mumbled some gobbledygook about caching metadata, or something like that. And one of your team has claimed Exchange might be 40% faster, but wasn't really sure.

Now, compare that with a PHYSICAL implementation where we know EXACTLY where the data is (on a flash drive) and we can be REALLY PRECISE as to the exact performance differential.

And, if one wants to do this extreme level of performance with VMware's VMFS (or other layer of virtualization, such as Invisita), it's pretty straightforward to do this sort of alignment at either a device or pool level.

I just can't wait to see how NetApp and the rest of the spindle-randomizers are going to get out of this pickle.

And a strident blog post won't do it, folks.

kostadis roussos

Hi!

Chuck, I think, you once again don't get it.

But at least I understand more of how little you understand of our system design.

Our systems do provide a way for a storage administrator to assign data to physical disks. What we do not require the administrator to do is map the physical disk block to a logical container.

So in your example, using a single FAS system you can choose to put your Oracle log files on an aggregate containing FC disks and your data files on an aggregate containing SATA disks.

How you can take what is a memory expansion unit (http://media.netapp.com/documents/netapp-performance-acceleration-module-datasheet.pdf)
and use that as proof that we do not provide a mechanism for storage architects to assign applications to classes of storage is a mystery to me.

kostadis


Chuck Hollis

So, why aren't you guys supporting real enterprise flash drives?

Keep in mind, the "memory expansion" you reference was positioned as "NetApp Supports Flash Memory" in the press.

And how might you explain the NetApp reference in the article I referenced in the initial post?

Aggregates are a compromise, I'd offer.

You still don't get fine-grained control of what gets written where, do you?

And we've all seen the differences in performance between a partially empty aggregate and a full one.

Unfortunately, with a NetApp device, you just can't turn this off, can you? There's no way to expose native LUNs to applications -- period. There's always some software in the way, even if you don't want it.

And as more people will want greater control of their storage performance characteristics over time -- accelerated by interest in enterprise flash -- I'd consider this an architectural disadvantage.

Good luck trying to convince knowledgeable storage administrators that it's no big deal.

Chuck Hollis

Well, Jay Kidd finally came clean for NetApp --

http://blogs.netapp.com/jay/2008/07/flash-forward.html

Ever the skeptical one, I'll believe it when I see it!

Mike

Chuck, it seems to me like you are missing what customers are looking for on a few points.

I don't think Storage Admins are looking for "more control" over their storage arrays. They are already overwhelmed with their pages of spreadsheets detailing which applications own which hyper. They also seem to be tired of fighting the tug-of-war between spreading applications across hypers from multiple raidgroups to get better performance and filling in those same raidgroups with other applications in order to keep utilization high. In large SAN environments you end up with poor performance, low utilization and a headache. Instead they seem to like the concept of pooling array resources, setting LUN/Volume priorities based on their application needs and letting the array manage the service levels so they can focus on their business not managing technology.

I also don't think your customers are looking at ways to buy the newest, fastest technology. Not in this economy. Those days are over. Instead of EFD, they are trying to buy more of the larger, cheaper, slower disks, then pool them and set priorities to get the performance they need without having to buy small isolated expensive EFDs.

I also think you may be a bit misguided on your comment on VMWare. The single disk I/O queue on VMFS would bottleneck long before any disk contention on todays disk drives. NFS eliminates this single disk I/O queue and allows for greater throughput to the disk drives, but is sadly not natively available on those non-intelligent layout arrays. Once you eliminate the single disk I/O queue and start using EFDs, the only way you can concentrate enough I/Os to require the performance of EFDs is through some data concentration mechanism (like de-duplication), which also is not natively available on those non-intelligent layout arrays.

It seems like the NetApp strategy to have EFDs when the market is demanding them and already have the virtualization in place to make use of them when the time comes is better than the strategy of trying to generate market demand with massive $$$ investments and vague solutions that don't really require EFD technology when given any scrutiny.

Chuck Hollis

Masterful misdirection! If you're working for NetApp's marketing department, keep up the good work -- very well written and put together.

I'd invite others to read Mike's "arguments" carefully, and see where the logic doesn't hold. I saw about a half dozen places -- what can you find?

Arne Rohlfs

Chuck,

you shouldn't post your rants about a competitior if you clearly don't know even the most fundamental things about their systems. Using a single NetApp controller for both, cheap ATA storage and high-end FC storage is normal for most NetApp customers. And of course they can decide where they want to put their data. Yes, they are working with 'virtualized' storage pools (called aggregates), but they have more than one. And in the future there will just be one more pool for the flash drives. What's the big deal? I don't need to decide on which SSD I want to put my data, all I tell the system is that I want it on my SSD storage pool. No idea why you think it would be hard to do that on a NetApp system, it is designed for these kind of things.

Regards

Arne

Chuck Hollis

I don't think you get my point on what it usually takes for fine-grained performance optimization.

Maybe you've never had to do this with a storage array. Lucky you.

Most storage architects would agree that any form of abstraction (including file systems) defeats the ability to precisely place data when you need to. Vendors tirelessly argue that their method is "better", but there's usually no way to turn it off if you need to.

The fact that storage objects are pooled, rather than subdivided, can be problematic when trying to parcel out small bits of high performance storage (e.g. flash) to the precise places where it's needed.

Must be nice to not have this problem in your environment.

Martin G

Just back from leave and as an end-user i.e. customer; we are having to deal with more and more storage with the same amount of staff or less, we have 250Tb per FTE currently managed (2.5 petabytes of storage currently), grew last year at 112%. I don't have time for a lot of fine-grained data-placement; I don't have time for moving data between tiers manually but I also don't have the budget to put everything on Tier 2 (let alone Tier 1). I need vendors to come up with better automagic optimizers which migrate hot spots in the background, I would like to move away from LUNs and simply allocate capacity (but NFS/CIFS is not the answer for this) and let the array/virtualisation controller handle this. No vendor has this at the moment but the one has gets it, gets my attention.

Chuck Hollis

Amen!!!

So, quick question, are we talking mostly file data here? If so, there's a half-decent approach with file virtualization to some of this.

But -- on a broader note -- you're right. There ought to be a pool of different tiers, and the storage figures out where things need to go at different times.

Craig Simpson

Chuck, I just saw this blog and enjoyed reading this blog. But I’ve got to join others in saying you’re wrong here. Storage virtualization has handled multiple drive speeds well for years. When SSD’s really take off it’ll handle those fine too. (http://www.communities.hp.com/online/blogs/datastorage/archive/2008/08/07/data-placement-who-s-architecture-is-really-broken.aspx)

But the real issue is that virtualization in the arrays is going to be a must to keep complexity under control. With it storage professionals have the foundation to avoid mundane tasks and focus on adding real value. Without it you hold contests to try get everyday tasks done in a reasonable time. You even award cars to somebody who can do it fast (EMC World Contest Link). But with the EVA everybody can do it fast! (EVA4400 Video Link). And that’s going to be a growing need for years.

Always happy to continue the conversation.

Craig

Chuck Hollis

OK, it's now official: 100% of EMC's storage competitors appear not to understand the underlying issue here, or want to pretend it doesn't exist.

That's fine with me.

As HP and other vendors start to attempt to put these devices into their existing products, they'll find out the hard way.

BTW, your comment does not strike me as a "conversation".

yfeefy

Hi Chuck,

I think you should provide specific details of some configurations where your fine grained control provides significant benefits. My feeling is such instances are a vanishingly small percentage of the total, and in general the overhead management required not only wastes a lot of time, but actually gets in the way of performance: many admins will not have the technical ability to truly improve performance with these 'features' (but they will think they do), and even if they did, as requirements change, many initial configs will not get modified in a timely fashion or at all to reflect changing requirements.

I get the impression you are grasping at straws trying to defend a storage architecture which is antiquated. Others have already pointed out that different service levels are possible for virtualized arrays (EVA and perhaps netapp), which is the area where you appear to be trying to claim an advantage. The SSD argument you bring up seems a red herring, just because EMC has implemented something one way doesn't define in stone how it must be done.

I have managed various arrays, and EVA is by far the easiest and best to manage. Knowledge of what's going on internally is still important, and I think HP needs to do much more to make in depth information readily available, but it is extremely easy to get good balanced performance from the array because it tends naturally to use everything available to it maximally. Contrast this to an array where drives just sit idle for months waiting for a manager to assign them to a task. This is what happens in real life.

Chuck Hollis

Really, now ... just because you've never had a problem with your specific use case doesn't mean that's true for everyone else, does it?

One thing I will admit -- there are a lot of people out there who aren't taht challenged by data placement issues, and are comfortable with the array virtualizing all the data all the time.

These are the same people who probably won't be interested in things like enterprise flash drives, because -- well -- they don't have any particular storage performance issues.

Lucky them.

Still, I for one would want the ability to turn the darn stuff off if I felt I needed to for one reason or another.

But, if you're still not convinced, here's an invitation ...

Go into any larger enterprise shop. Go find the big hairy applications that run the business. You know, the ones where someone gets fired if it runs slow.

And I bet you won't find a single spindle-randomizing array -- from any vendor -- behind any of them.

Cheers!

yfeefy

Hi Chuck,

Rather than an invitation, I'd rather see the details about how a sizable benefit is achieved with manual "fine-grained-placement". Apart from outer edge placement (which could be done automatically or manually with virtual arrays, although to do it manually, e.g. on a per LUN basis, is not implemented on the EVA I'm most familiar with (why not, hp?)) - I suspect substantially the same can be done with a virtual array, albeit with different techniques. That goes for the utilization of SSDs as well. It's a different philosophy, but the sysadmin still has a lot of control.

I understand the argument about large firms with large apps running large high-end arrays (which are not virtual arrays) and I don't believe it's relevant to this discussion. The main reasons for this is size and speed of hardware (and commensurate price tag) having nothing to do with fine-grained placement, and, well, politics. Just because they've thrown a lot of money at the problem doesn't mean you won't find plenty of suboptimal configurations here.

Chris

We're a fairly big organisation and we have a "spindle-randomizing" FAS and more traditional HP EVA. You're right our critical apps are on the traditional SAN.

However we wish it wasn't. The fact is the FAS is newer technology and as I'm sure you know, the most critical apps are the last to be moved to new technology, especially in larger organisations.

I guess to use traditional SANs you either invest heavily in training, or buy in consultancy. On the other hand I've had no trouble getting what I need from the FAS, with no formal training whatsoever.

Peloton

Imagine a virtualized array able to manage its address space in fairly fine grained chunks (chunk = several MB?), and transparently move frequently accessed chunks to the EFD, and infrequently accessed chunks to the slower disks. In real world applications would these chunks stay hot enough long enough to warrant the movement to the EFD? Or would the array constantly be chasing its tail? Said another way, are we talking about EFD residencies of these chunks measured in minutes, hours, days? I realize its all application and data demographic dependent, but what are some of the more typical scenarios? And if its all automatically done by the array, then I suppose its possible that application 2 all of a sudden gets hot and its data forces application 1 out of the EFD, but what if the service level for application 1 was the most critical? How would the array know? So, it would seem that doing placement manually has some advantages when you are trying to control service levels for specific workloads. Doing it automatically is good if the overall goal is just to maximize overall performance, but not any specific workload. Best of both worlds would be automation with knowledge of the workload service level agreements.

Chuck Hollis

Peloton -- you're thinking in the right direction, I believe.

The trick is in the effectiveness of the algorithms against real-world usage patterns.

If you think about today's DMX (large, non-volatile cache) and its algorithms, it's a reasonable predecessor of what might be needed for what you describe.

The good news? We've got 15+ years experience understanding different workloads, and the algorithmic effectiveness of putting things in cache vs. spinning disk.

Not that there's an enormous amount of work to be done for this use case, but at least there's some work to build on ...

The comments to this entry are closed.

Chuck Hollis


  • Chuck Hollis
    Chief Strategist, VMware SAS BU
    @chuckhollis

    Chuck has recently joined VMware in a new role, and is quite enthused!

    Previously, he was with EMC for 18 years, most of them great.

    He enjoys speaking to customer and industry audiences about a variety of technology topics, and -- of course -- enjoys blogging.

    Chuck lives in Holliston, MA with his wife, three kids and four dogs when he's not travelling. In his spare time, Chuck is working on his second career as an aging rock musician.

    Warning: do not buy him a drink when there is a piano nearby.
Enter your Email:
Preview | Powered by FeedBlitz

General Housekeeping

  • Frequency of Updates
    I try and write something new 1-2 times per week; less if I'm travelling, more if I'm in the office. Hopefully you'll find the frequency about right!
  • Comments and Feedback
    All courteous comments welcome. TypePad occasionally puts comments into the spam folder, but I'll fish them out. Thanks!