« Learning To Compete -- The Real Challenge Of ITaaS | Main | Considering Clouds And Public Policy »

September 29, 2011



So, basically Chuck this can be summed up as an all-too-typical customer environment where they're not really very aware of what they have or how they're using it. And more importantly, they do little if anything to improve their situation over time...not through additional technology investments but through good old fashioned organizational and process improvements.

I've seen this dozens of times. A few years ago I wrote, "Chances are excellent that if your requirements can’t be met by available commercial products, your business is either way, way out front or way stuck in the past. Of the more than 100 companies I helped during my engineering career, most fell into the latter group, mired in puzzling, inefficient, and often arcane business processes that should have been replaced—not recreated—in the digital realm."

I'm glad to read you didn't take the "customer is always right" approach which really would not have helped them. They needed and received a little tough love. The customer, contrary to popular belief, is not always right.


Excellent story Chuck! I see so many companies stuck in situations just like this. It really is sad to see rampant incompetence impacting a company. It's even worse when YOU get the blame for it.

I think you handled it perfectly.

Roger Luethy

Hi Chuck

This i a very good in-depth look into what most of us in the industry face a lot. Sometimes customer expects that we can do Magic do solve there problems (which most of the time are not related to technology but rather to business processes).

Thanks !

greg schulz

Great post Chuck

With all of the talk about convergence, that has a focus around technology, until people, processes and their internal organizational issues (politics) can be converged (or at least abstracted), the full benefits of products will not be realized.

Your point about lack of tools is spot on, how can you effectively manage what you do not know about, effectively flying blind, hence need for management tools, situational awareness, metrics and measurements.

Cheers gs



Thanks for the trip back down memory lane. I've spent time on both sides of the table, and am now sitting in the CIO chair.

I particularly appreciate your line 'The problem isn't the technology, it's how it's being used".'

Without a strong knowledge of what technology can do, you wouldn't be qualified to make the statement, but in your case it bore weight. I wish I was in the room.

Now that I'm on the customer side of the equation, I'm working hard to spread the word that IT leaders need to change... it isn't really about the technology, but how it is implemented to add value and provide differentiation for their organizations.

I blog on this topic at TurningTechInvisible.com about the leadership required to turn technology invisible - to make it like oxygen... you don't even have to think about it until it's not there.

You rightly point out that planning, management systems, and leadership need to be layered on top of great technology to make it 'invisible'.

Richard Hintz

Entertaining, unless you were the target, but nothing new here. Usually the scenario is that the functional user beats up on the data center people because the app is slow.

The app is slow because it's legacy and the app developers/maintenance people haven't refactored any of it for years, much less tuned to the new hardware/OS environment. And the DB people are struggling, to be kind.

What's odd about this customer/vendor meeting is that the customer didn't come up with a bunch of seemingly credible, but ultimately BS "reasons" to pin on EMC that would have all been new to the sales team and which couldn't be refuted on the spot. On later, further investigation they would have all turned out to be misstatements or plain ingenuous.

I had to laugh at storage/staff ratio. And I'm sure it was predictable that the SAN failures were lack of diversity as the likely cause. I wonder if you went to the customer site how much has changed to improve things.

Martin G

Oh I wish more vendors would front-up to senior management and have the guts to tell them where things are going wrong. Many times, you would find the storage team in quiet agreement with the vendor; they will have made many of the observations themselves internally and will have been hushed by senior management because it is perceived that doing things the right way is expensive.

The biggest battle that we fight every day is the tools one; now, it is fair to say that the storage vendors may not have done a great job here and the tools have been poor. But now, the tools are getting better and the estates are so large, that to manage without tools is getting incredibly hard.

But come to think of it, is that really the biggest battle; actually, sometimes I think it’s the technical refresh battle. ‘It aint broke, so don’t fix it’ or ‘Refresh doesn’t add any value, so we won’t do it this year’; key environments which are pretty much completely out of support and now people are scared to change them. So at some point, you end up looking at doing a complete replatform of an environment. Yet again, vendors don’t help matters; certification matrices can be opaque and complex. It is entirely possible to find that a firmware upgrade on an array could trigger a firmware upgrade on a switch which triggers a firmware upgrade on an HBA which triggers a driver upgrade on an operating system which triggers an operating system upgrade which triggers an application platform upgrade which triggers an application retest/verification. This is complex entangled infrastructure; [insert vBlock sales-pitch here] [insert vBlock rebuttal here].

Or is it the fight to instigate process? The customer who demands that they are unique and they should not have to follow change control? The customer who demands that their application goes in untested, unverified?

None of this has to be so but it takes some serious mind-set change.

1) IT is an investment and potentially a foundational investment; would you let your offices fall into rack and ruin? Would let your staff work in an environment with a leaky roof?

2) IT requires leadership but IT leadership requires listening skills. Listen to your teams, they will tell you what is wrong. If a team believe they need to spend money to make things better, work with them to build a business case and plan. Most people don’t go around spending their employer’s money for fun; believe in the people and be open to critique.

3) Be open to challenge from with-in and with-out. Don’t shut the conversation down; I think the biggest barrier to change is not the people on the shop floor, it is the people between the shop floor and the executive suite often.

4) What you can’t measure, you can’t properly and effectively manage. Believe this and invest in measurement.

5) Change your financial models, stop project-based, siloed infrastructure decisions. If you look at your estate and find utilisation is below 20% and your teams are still asking to buy more, ask them why? If the answer is that projects/business units are ring-fencing resources because ‘they paid for them’; don’t beat up the infrastructure team for being ineffective, beat up the CFO for allowing such a stupid model to continue.

6) Build a service-based model and allow an adult conversation. This adult conversation may involve saying ‘No!’ once in whilst.

7) Be serious about DR, Business Continuity.

8) Be honest about risk; if we do it this way and cut this corner, we will risk this system. Or we could do it better and not risk the business.

I was going to reply anonymously but it’s nothing I haven’t said before and it’s nothing I’m not prepared to stand-by.

Chuck Hollis


Excellent thoughts. Thank you.

-- Chuck


Good read that. I'm going to circulate it.
I've been in much the same situation - it's often the case that storage is blamed 'by default' as ... well, what amounts to the easiest target. Performance is slow - must be a storage problem.

I've got fairly adept at bouncing back reports that basically say 'not a storage problem' with graphs. But often, I find that the 'storage function' is just not well equipped to do that, can't defend 'their' turf, so get squeezed to pass it on to vendor support. Or otherwise end up having to 'fob it off' because nothing's going to happen without someone making a business case for it.

I also agree with Martin about changing 'internal' storage delivery. Capital expenditure, per project, to buy tin is inefficient at very many levels - it's only very rarely that such an upfront purchase is used immediately, and then with no growth for the entire lifespan.
The vast majority of storage allocations grow, and are overprovisioned at day one.
But you _have_ to switch away from 'project supplies capital, to buy hardware' to a model where the 'storage service' is a product sold by the gig-month and might come in a tiered model, based on how much performance is needed from those gigs.
That represents a substantial change to financial models though, and that can broker resistance. Capital expenditure for project deployment, to operational expenditure for ongoing service. Even if the 'end sum' ends up cheaper...

David Patton

Well done. The customer is NOT always right, but they are the customer. You gave them frank, honest feedback in a professional manner with an offer to help them solve their problem. This is much more valuable to them than just rolling over and accepting full blame.


Sometimes vendors get what they wished for, or marketed for: All this messaging about "convergence," SaaS and taking the pain-out-of-IT, plus years of outsourcing in the widest sense, well, guess what? It worked in many cases. The oft-berated-themselves IT department, tired of being caught-in-the-middle between not-quite-baked vendor offerings, CEOs asking for lower costs, users asking for more functionality and better support, bought the "depend more on us" pitch from vendors, integrators and service providers. The result? Some IT departments have gone so slender, focusing on procurement and simplified operations, their expertise has eroded significantly.

This isn't just a storage issue, it permeates IT. Rather than sell IT on ease-of-use and haggle on the price premium for that often empty promise, vendors should win on that difficult old concept of "partnership." We are in this together, vendor and IT, and it includes:

* Training, yes, so painful, but so useful long-term; closely related to this is transitional training - when there is turnover to key IT personnel, ensure the replacement is up-to-date

* On-going, formalized, post-sale communication; the drop it and run to meet the next quota technique might buy Larry Ellison the America's Cup, but it puts customers at a huge disadvantage. IT departments that don't keep the vendor in the mix as their conditions change are equally as guilty.

* Transparency in marketing (up to a point of course) but most importantly at point-of-sale and configuration: "This solution works for this context, but if that context changes, we better talk." And keep repeating this mantra.

* And finally, the notion that an SLA is a two-way street, a process. Texting every quarter "did you hit the SLA?" doesn't work and neither of the partners should stand for it. Yes, IT needs to understand, and prepare for the fact that vendors are human too and mistakes are made. IT needs to understand that about themselves as well. The blame game leads to, well, just look at our current U.S. political system. Doesn't work in IT either.

Regarding tools, well, some tools inflict further confusion, some help, but regardless they are not a panacea.

Given the never-ending cycle of quotas and quarterly-focused business models of IT vendors, the cold truth is this all mainly falls at the feet of IT. But the other side of the truth is vendors winning on throw-it-over-the-wall solutions have helped create the phenomena of emasculated IT, therefore shouldn't complain much when it comes back to bite them.

Richard Hintz

In the specific case that the post mentions, the storage staff level is ludicrously high, given the total storage managed, so I don't think it qualifies in your slender IT category, though I understand what you mean about level of expertise. But, just to take the specific points you raise, are you saying that ensuring that customer technical staff are trained is the responsibility of the vendor?

And as far as ongoing, formalized post-sale communication, in my experience, this is hard to avoid, especially when the vendor has a services arm. If a customer wants more communication, it's usually as simple as making a phone call. Usually at some point the free consulting stops and the meter starts running, but this is typically negotiable if there's a severe performance problem. I can't see that the vendor is responsible for an overall system outage when the architecture didn't include diversity, for example. They are, of course, responsible for outages of their own components according to whatever they represented contractually.

Transparency in marketing means, too, that the customer understands their own environment and, especially, doesn't have any back-level, legacy hardware/software booby traps.

I didn't understand your point about SLA. If something is wrong, one typically doesn't wait for an end of quarter SLA audit, especially for something like storage.

Tools: trying to run a modern environment without appropriate support tools is a recipe for disaster, especially in a multi-vendor environment.

Sorry, but I really don't understand your push back. It seems as if you want to put an inappropriate level of what's supposed to be technical (and general) management responsibility on the vendor. Presumably all the decision makers on the customer team have had things sold to them before and they can apply a sanity check to vendor claims.

(I'm a customer, not a vendor, at least in this context.)


One of the great things about EBCs is that folks with no direct "ore in the boat" can come in and deliver a clear message that may hurt a bit, then leave. While the account team can blame you for being a "bad guy", the bottom line is you did get through to the customer and probably stopped a dysfunctional cycle that wasn't helping anyone

Chuck Hollis

Not to be too controversial here, but I wonder how much of this just All Goes Away when using something like a Vblock.

-- Chuck

Richard Hintz

Except look at the number of "time to move on," "that angle wasn't working well," "Enough on this topic." The message was being transmitted, but only received obliquely. (Yes, I know this isn't a verbatim record, but I've been in these sorts of meetings.)

I'd be amazed if anything changed at the customer site except a bunch of conversations about how EMC was unresponsive and unwilling to accept their responsibility.

Chuck Hollis


Interestingly enough, it turns out that EMC ended up proposing our SMS -- storage managed service -- to this customer. I've heard anecdotally that there's incredibly strong pushback from the storage team.

At least there's a decent alternative on the table to consider -- from EMC.

-- Chuck

Richard Hintz


The storage team is pushing back because they realize they have to work their way out of their legacy tail and can't digest the SMS solution simultaneously. Sounds weird, but I'd bet on it.

Martin G

How much goes away when using an Infrastructure stack such as Vblock? It's an interesting one but I think it you put any kind of Infrastructure stack into an organisation such as the in original article; you end up with the same mess and probably a more expensive mess for the vendor and the customer to sort out.

You know as well as I do, infrastructure stacks are not the solution or certainly not the whole solution. Without real cultural change; you'll fail.

Vblock may be a short cut to the infrastructure changes you need to make but you can happily do it with any infrastructure; even the customer you mention could probably do it but they need to want to do it.

And in my experience, if you take a mess and turn in it into a managed service; it'll just be a mess run by other people with a bunch of pretty reports which polish a turd. Unless there is real commit by the customer to invest in fixing said turd, you'll end up with frustrated customer and supplier.

Chuck Hollis


A great point. We have seen a few Vblocks go in without that commitment to change we're both talking about, and it hasn't been pretty in the least :)

As far as managed services, I can only speak to EMC's version, and not others. Part of our proposal is always investment in transformation: process, technology when possible, and always have the right people.

There's also a "hand back" option available, which is popular. EMC invests (and gets paid for) setting things right, and -- once done -- the customer has the option of taking the operation back again.

Thanks again

-- Chuck

Richard Hintz


Keep in mind this earlier comment:
"Another serious outage resulted when the customers' team were moving stuff around, and accidently trashed production data. EMC got the phone call after the damage was done. There were problems recovering from backup..."

Oops, how to get the running apps from Point A to Point Z, Vblock nirvana.

Given everything else, I also bet they're constrained on power, space, cooling, so bringing in Vblock is problematic, even if it would be a net facilities win once the transition was over.

I also assume that they are substantially pre-virtualized, so that's another barrier.

Last, given everything else, I assume it would take a year for their procurement to work out the software licensing issues. Those are just the highlights without actually knowing anymore than what you've said, much less doing any real analysis.


Good post Chuck! I can certainly relate.
-- Tony Pearson (IBM)

The comments to this entry are closed.

Chuck Hollis

  • Chuck Hollis
    SVP, Oracle Converged Infrastructure Systems

    Chuck now works for Oracle, and is now deeply embroiled in IT infrastructure.

    Previously, he was with VMware for 2 years, and EMC for 18 years before that, most of them great.

    He enjoys speaking to customer and industry audiences about a variety of technology topics, and -- of course -- enjoys blogging.

    Chuck lives in Vero Beach, FL with his wife and four dogs when he's not traveling. In his spare time, Chuck is working on his second career as an aging rock musician.

    Warning: do not ever buy him a drink when there is a piano nearby.

    Note: these are my personal views, and aren't reviewed or approved by my employer.
Enter your Email:
Preview | Powered by FeedBlitz

General Housekeeping

  • Frequency of Updates
    I try and write something new 1-2 times per week; less if I'm travelling, more if I'm in the office. Hopefully you'll find the frequency about right!
  • Comments and Feedback
    All courteous comments welcome. TypePad occasionally puts comments into the spam folder, but I'll fish them out. Thanks!