The Storage Performance Council has been around since 2002. Its stated goal is to provide independent cost/performance metrics for storage arrays.
But has it achieved its stated goals? And does anyone (other than a few vendors) take them seriously?
We occasionally get asked why we don't participate, and there are some good reasons -- none of them particularly sinister (apologies in advance to consipiracy theorists everywhere!)
But why is there no good independent performance metric for storage?
And why do some vendors (like IBM) continue to insist that the test (and their participation) is relevant?
It's an interesting perspective on the storage industry, and how simple things can get really complex once you look at them.
Way back when when storage devices were generally slow and really expensive, there was a lot of interest in storage performance, especially when balanced against cost.
Not all arrays were designed the same, and there was a very wide range in cost/performance from vendor to vendor.
The goal of the Storage Performance Council (actually, I think the "council" is actually one guy who runs it part-time) was very laudable: establish a uniform testing procedure that could give customers a simple comparison to stack products up against each other.
The first test suite was the SPC-1. If you go to the site, you'll see that there haven't been many submissions there lately. Oversimplified, its model seems to be trying to simulate a transactional environment.
The newer test (SPC-2) tries to target a different performance profile than the SPC-1, in that it seems to focus on sequential data access as you'd find in a video streaming or backup-to-disk environment. More submissions from more vendors here.
So What Went Wrong?
I don't know how many of you have ever spent much time getting deep and gritty into various aspects of storage performance -- especially storage at scale -- but it is mind-numbing in complexity.
No simple analogy with car performance will do here -- there are more wrinkles to this subject than can be covered in a short post, and -- unfortunately -- many of them often matter in the real world.
A simplified test -- of any sort -- that doesn't accurately predict real-world performance will be of little use. And -- to this day -- no one has come up with a simplified storage performance test that can capture enough richness of real-world use cases to be of much value to either vendors or customers.
Note: if you're feeling particularly cruel, ask one of the vendors who tout the SPC to explain how its results can be used to predict performance in your specific environment, and watch the fun ..
Sad, but true. Occasionally there will be a wave of resentment from some of the more vocal industry analysts over this poor state of affairs. Some will even blame big companies like EMC for somehow causing this situation to exist.
Guys, we didn't invent storage, and we don't control the wide range of ways customers actually use it. And if there was some simple test that was an accurate predictor of real world performance, man, we'd be all over that.
Because we spend way too much money and time doing performance testing the hard way.
I think most storage users have figured this out. We've never done an SPC test, and probably will never do one. Anyone is free, however, to download the SPC code, lash it up to their CLARiiON, and have at it.
What We've Learned About Performance Testing
I thought I could capture the essence of EMC's view here, but it's going to take too long. Let's say we believe that there's no substitute for a real world test. And, if you're going to use an array for multiple purposes, the interaction is important. And, of course, performance during degraded conditions (such as a drive failure) will be very interesting. And what happens when the I/O profile changes dramatically.
Exchange drives a different I/O profile than Notes, and it matters in the real world. Ditto for Oracle's I/O profile vs. SQLserver vs. UDB. The list keeps going on, and on, and on ...
And that's just the beginning -- I have not begun to do the topic justice. Maybe the Storage Anarchist will help me out here.
I'd hazard a guess that we've got multiple billions of dollars invested over the years doing performance characterization. That's not a typo. It's such an ingrained part of our culture (and our R+D spend) that it's hard to measure accurately.
So, our belief is that if performance is important to you, you're not well served by simplistic test results.
And if performance isn't all that important to you (which is increasingly common), other factors will dominate your decision-making process.
Which Brings Up The Topic Of Benchmarketing
I haven't looked at the submitted configurations for SPC results for a while, but a few years back, they were excellent examples of -- well -- some of the most creative storage configurations I have ever seen.
I saw stuff there that no sane customer would ever put into production. So, if no one would actually buy a configuration like that, why would you test it and publish the results?
The answer to this, sorry to say, is when vendors confuse value-added benchmarking with benchmarketing.
In the former, you pick a customer use case and say "here's what you're likely to see in your environment".
In the latter, enormous effort is spent to come up with a better number, and then bash all the competitors with it ruthlessly.
It doesn't matter if the number is relevant or not. Just that Vendor A beat Vendor B.
Not to throw rocks, but that seems to be IBM's game plan for storage marketing, at least for SVC.
I found that IBM claimed to be the fastest storage virtualization product on the SPC. If you have a taste for over-exuberant press releases, go have a read.
Last time I looked, they were the ONLY storage virtualization product on the SPC list.
So, while a technically correct claim, it is very misleading. In a contest where you're the only entrant, I'd hope you'd win.
Now, to be fair, there's a smattering of HP array product there, and some of Sun's portfolio as well, and Fujitu's product as well.
No Hitachi. No EMC. No NetApp. Lots and lots of vendors have skipped the exercise entirely. Maybe for different reasons than EMC, but the net result is the same.
And, interestingly enough, only a few select products from the IBM/HP/Sun portfolio. I wonder why this is? You'd think that if the SPC was a valid, customer-oriented test, they'd test the majority of their products, and not cherry pick.
But I digress.
Will We Ever See A Simple Storage Performance Test?
Well, there are already lots of them out there, including the SPC.
Problem is that none of them do a good job of predicting real world performance for anything other than, ummm, running that specific test.
The real question is whether the problem is solvable, or not. I will offer the opinion that the problem is getting worse, and not better.
We're seeing a wider and wider range of storage use cases, each of which translates into unique I/O profiles. We're seeing more and more consolidation, which means running more use cases on the same device at the same time. And data volumes continue to grow, which means that more often, the only valid tests are those done at significant scale.
The prognosis is not good for SPC, or any similar effort. Realistic performance testing is getting harder, not easier.
So What's A Customer To Do?
In my mind, you have a choice.
You can either build a very expensive test bed, and find out for yourselves, or you can partner with a vendor you trust that can understand your environment, make some good recommendations, and hopefully be there for you if things don't go as planned.
Now, I've met very large customers with very critical applications who do their own testing or have us do it. It's worth it to them to know what's fastest, because it's critical to their business.
But, for the vast majority of storage customers, the second option (find someone you trust) is probably more realistic.
Just don't count on a simple test to give you the magic answer.