The Vendor Beating

We've all seen parents berate their children, and managers berate their employees.  

It's not pleasant to watch, but it happens.  

As an employee of a large IT vendor, I've been at the receiving end of a reasonable number of vendor beatings. 

Occasionally it's richly deserved.  But, sometimes, it's masking a deeper set of issues that have very little to do any vendor whatsoever.

Such was the case a while back when I offered to do a meet-and-greet for a customer coming in to EMC's Executive Briefing Center.

The Role Of The EBC At EMC

EMC has invested heavily in customer briefings ever since I've been here, which is to say a very long time.  There's the large EBC facility in Hopkinton, supplemented by regional centers in Cork (Ireland), Singapore, Santa Clara and a new capability in Durham, NC.

The Hopkinton facility is the busiest by far.  On a given day, there's anywhere between five and twenty customer or partner groups coming through for a day or two.  Sometimes they're looking for product updates, sometimes it's about a specific project or challenge, very often it's to get a better sense of vision and direction.

Part of our culture here is that we all spend considerable time with customers and partners; hopefully doing more listening than talking.  Indeed, during hiring interviews, I make it pretty clear that EMC stands for Everyone Meets Customers.  It starts with Joe Tucci and works its way throughout the organization from there.

Thanks to the EBC, any EMC employee nearby has no shortage of potential customers and partners to interact with -- if they choose, that is.  The downside is sampling bias -- you need to balance EBC interactions with meeting people on their own turf.

I usually make a great effort to learn as much about specific situations before walking into the room with a customer or partner.  Occasionally, circumstances force me to walk in largely blind, and figure out things as we go along.  That can be, well, interesting ...

The Setup

One of the things I'm often asked to do is greet people in the morning.  That means we sit around the table, everyone introduces themselves, and I try to gain a better understanding of what's going on in their world.

The introductions went well enough: a few people on the storage team, their team leader, and the person he reported to, responsible for a portion of infrastructure and operations.  So far, so good.
My opening gambit was pretty generic -- what's going on in your world, and what do you hope to accomplish today?

And then it started.

Here Comes The Flood

The storage team lead opened up with "We're spending way too much on storage".  Hmmm, I'm thinking, that's not good.

"And we're having all sorts of performance problems".  Huh, that's unusual. Wonder what's up with that?

"And there are way too many outages".  If true, that's very serious stuff.

"And your people aren't able to help us".  OK, this is getting very unusual indeed.

It just kept coming out.  Complaint after complaint.  Injustice after injustice.  Multiple threats of switching vendors if the situation didn't improve.

It was Vendor Beating Time.

The Politics Of Getting Beat Up

When someone is pretty upset, it's usually best to hear them out -- no interruptions, no questions -- just let them go.  If you're a vendor dealing with a very upset customer (justified or otherwise), that's especially true.

As the storage team lead started to get on a roll, you could feel the tension level rise in the room.  That's natural.

The storage team lead's direct manager tried to interject and add some perspective to the discussion.  I sort of sent him a look that said "hey, let this guy get it off his chest, it's OK".  A few minutes in, the EMC sales rep started to get a bit defensive as well (after all, this is all looking pretty bad), but I sort of sent him the same message, back down, let it all out.

I started nodding and scribbling down notes to come back to later.  After about ten minutes, you could feel the ferocity lessening a bit, the energy was starting to shift.

When it felt like the time, I asked if it was OK to come back and ask a few more questions about each of these issues.  Of course it was -- what else could be said?

Help Me Understand

I started with his first complaint -- that he felt that he was spending far too much on storage.  Can you tell me more about that? -- I asked.

The picture started to emerge.  The entire IT function was under pressure to significantly cut costs (a separate topic probably worthy of discussion in itself, but not at this time).  Storage expenditures were one of the most visible line items in the budget.  Hence there was a lot of pressure on his team to spend less.

OK, I said.  How much storage are we talking about?  He tossed off a number of around 70 terabytes of primary storage, and about 95 terabytes of secondary (archival-ish) storage as raw capacity.  
A good size, but not enormous I thought.

How fast was it growing?  He wasn't sure, but he guessed raw capacity had doubled in the last twelve months.  OK, I thought, that's something to ask more questions about later.  And how many storage people do you have?

Seven, he replied, and they had an open req.  I was a bit stunned, so I asked for clarification.  

Seven primary storage administrators, exclusive of the backup team, server team, database team, etc.  I didn't hide my astonishment at the large number very well. The storage team lead started to get defensive about my reaction, so I changed the subject.

OK, of the 70+95 terabytes of raw capacity, how much of this is visible to users?  Stuff they can directly see -- like allocated and unallocated usable storage, visible snaps and the like.

He wasn't really sure.  

I explained my interest that "conversion efficiency" was a useful quick-and-dirty metric.  Here's X amount of raw capacity.  Here's Y usable capacity directly visible and accessible to the people who use it.  Do a good job, and you can often be north of 80% of your raw capacity being directly usable for work.  Do a poor job, and it can drop below 50% or worse.

A long and pregnant pause.  That angle wasn't working well.

So I came at it from another perspective -- what tools and processes do you have in place to manage storage usage, and report back to people how it's being used?  Well, they had a bunch of spreadsheets that someone occasionally ran around and filled out.  Hadn't done one in a while, too busy.  Not sure if anyone really read them.

I decided to press my point.  If I'm, say, the owner of the big SAP application, do I know what I'm using, what sort of service I'm getting, and how much I'm paying?  No clear answer.

Enough on this topic.  The picture was getting clearer, and it was time to move on. 

Where Does It Hurt?

The next topic on the list was the complaint around performance -- where, exactly, were they experiencing performance problems?  Database queries were getting slower, and the business users were starting to complain.  Fine -- did they have any evidence as to where the problems might be -- storage media, storage array, storage path, server, database, query structure?

No clear answer.

OK, I asked the EMC team, had we offered to take a look and see what might be up?  

The EMC presales engineer said that we had spent significant time looking at their setup, and came to the conclusion that the root cause was pretty obvious: the database environment had grown willy-nilly over the years -- it wasn't laid out well, the queries weren't particularly well written, and so on.  

Sure, there were things we could do on the storage side (e.g. faster storage, better layouts, etc.), but it was a bigger issue that just storage performance.  The EMC team had not only offered up a report, but had also shared our extensive library of best practices in an effort to help.

I then turned back to the storage team lead for his view.  He said he had challenges working with the database and server teams to address the problem, and everybody was pointing fingers at the storage team.

The picture here was getting clearer as well.  Time to move on.

Any Outage Is A Serious Outage

Look, at EMC, most of us are storage people at heart.  If you can't get to your data at any time (and for any reason!), that's a really big important deal in our world.  So I wanted to know more -- what the heck was going on here?

The storage team lead talked about "multiple SAN failures".  Really?  Turns out that they had a number of servers that periodically lost connectivity to the array, causing a huge turmoil.  

Wow, I said, how could two paths fail at the same time?  Something really bad must be happening here.

No, I was wrong.  The problem was with servers that were single pathed -- to no-longer-supported operating systems, FC adapters and driver versions, to make the picture more complete.  It got worse.  The EMC presales engineer spoke up and said that every quarter they'd done an inventory of the customer's environment, flagged the problems as a serious concern, and no action had been taken.  

For the last two years.

Another serious outage resulted when the customers' team were moving stuff around, and accidently trashed production data.  EMC got the phone call after the damage was done.  There were problems recovering from backup, and so forth.

I stopped that line of discussion, and decided I'd heard enough.

Time For Some Tough Love

I turned to face the storage lead's immediate boss, and decided to say what I thought needed to be said.  Even if it wasn't going to make me any friends in the process.

"Look", I started "I get to see how hundreds and hundreds of IT organizations run their operations, and especially how they manage the storage function.  So keep that in mind when I say what I'm about to say".

"And, please understand", I continued, "that no vendor is perfect, and there are certiainly many areas where EMC could be doing better, but I am absolutely convinced you'd have the same problems -- or most likely worse -- if you decided to go with another vendor".

"Dealing with rapid storage growth, demanding users and a tough budgeting environment isn't easy.  It requires a partnership: we do our part, you do yours.  Simply blaming your vendor for your troubles might make you feel good in the short term, but it won't solve anything".

I mentioned a friend that had now been unsuccessfully married four times.  He has become quite adept in blaming each and every woman for the failures.  It makes him feel better, but his life still sucks.

"You have a key choice to make here.  Your storage environment is growing fast.  What might have worked in years past clearly isn't going to work going forward.  I think you can see that".

He agreed.

"Here's the choice: you either have to invest in building a modern storage management team that's organized, trained and equipped to deliver storage services, or you need to hand over the storage function to a managed service partner that knows how to do this.  The problem isn't the technology, it's how it's being used".

The storage team lead didn't like where I was going one bit, and started to speak up in his defense.  I ignored him for the time being.  I was on a roll.

"Seven primary storage administrators, and no evidence of tools or processes?  That's a problem.  Poor relationships with your IT co-workers that prevent you from addressing user-visible issues?  That's a problem.  Multiple outages that were clearly preventable?  That's a problem.  No visible use of change control procedures?  That's a problem.  Not being able to act on the advice and help we're offering you?  That's a problem."

"We can dress it all up with happy words and make it more palatable, but -- you asked me for my opinion -- and there it is.  You -- as an IT manager -- have some decisions to make.  We'll be glad to help you explore your options in more detail, and show you what we can do to help -- but I think the next big move is yours".

Long silence in the room.  I think the poor EMC sales rep was going to burst.

The response was a bit awkward from the IT infrastructure manager -- after all, the storage team was sitting right there.  

He sort of acknowledged the points I was making in a roundabout way, and sort of thanked me for my input.  I suppose that's the best I could have hoped for, given the circumstances.

I'm Sure I Didn't Make Any Friends That Day

Sometimes I think I might be weird.  

So many vendors take such extraordinary pains to please everyone all the time.  Even if it's not in their customers' best interests.  As I left the room, I felt sort of bad for the EMC sales rep; after all, I hadn't done the vendor-happy-talk and its-all-our-fault and we'll-try-to-do-better approach as might have been expected.  

In this particular situation, I just couldn't do it.  Something important had to be said.  And I'm not saying that EMC was without fault -- either in this specific situation, or generically.  

But -- if we fail to be transparent and honest with our assessment of the real problem, isn't that a concern as well?  That's what real partners do for each other.

Sometimes I'm a technologist.  Sometimes I'm a marketeer.  Sometimes I'm a visionary.  Sometimes I'm a career coach.

And, once in a while, I'm a therapist doing an intervention.

Comments

[josephmartins]

So, basically Chuck this can be summed up as an all-too-typical customer environment where they're not really very aware of what they have or how they're using it. And more importantly, they do little if anything to improve their situation over time...not through additional technology investments but through good old fashioned organizational and process improvements.

I've seen this dozens of times. A few years ago I wrote, "Chances are excellent that if your requirements can’t be met by available commercial products, your business is either way, way out front or way stuck in the past. Of the more than 100 companies I helped during my engineering career, most fell into the latter group, mired in puzzling, inefficient, and often arcane business processes that should have been replaced—not recreated—in the digital realm."

I'm glad to read you didn't take the "customer is always right" approach which really would not have helped them. They needed and received a little tough love. The customer, contrary to popular belief, is not always right.

[BrandonJRiley]

Excellent story Chuck! I see so many companies stuck in situations just like this. It really is sad to see rampant incompetence impacting a company. It's even worse when YOU get the blame for it.

I think you handled it perfectly.

[Roger Luethy]

Hi Chuck

This i a very good in-depth look into what most of us in the industry face a lot. Sometimes customer expects that we can do Magic do solve there problems (which most of the time are not related to technology but rather to business processes).

Thanks !

[greg schulz]

Great post Chuck

With all of the talk about convergence, that has a focus around technology, until people, processes and their internal organizational issues (politics) can be converged (or at least abstracted), the full benefits of products will not be realized.

Your point about lack of tools is spot on, how can you effectively manage what you do not know about, effectively flying blind, hence need for management tools, situational awareness, metrics and measurements.

Cheers gs

[InvisiTech]

Chuck,

Thanks for the trip back down memory lane. I've spent time on both sides of the table, and am now sitting in the CIO chair.

I particularly appreciate your line 'The problem isn't the technology, it's how it's being used".'

Without a strong knowledge of what technology can do, you wouldn't be qualified to make the statement, but in your case it bore weight. I wish I was in the room.

Now that I'm on the customer side of the equation, I'm working hard to spread the word that IT leaders need to change... it isn't really about the technology, but how it is implemented to add value and provide differentiation for their organizations.

I blog on this topic at TurningTechInvisible.com about the leadership required to turn technology invisible - to make it like oxygen... you don't even have to think about it until it's not there.

You rightly point out that planning, management systems, and leadership need to be layered on top of great technology to make it 'invisible'.

[Richard Hintz]

Entertaining, unless you were the target, but nothing new here. Usually the scenario is that the functional user beats up on the data center people because the app is slow.

The app is slow because it's legacy and the app developers/maintenance people haven't refactored any of it for years, much less tuned to the new hardware/OS environment. And the DB people are struggling, to be kind.

What's odd about this customer/vendor meeting is that the customer didn't come up with a bunch of seemingly credible, but ultimately BS "reasons" to pin on EMC that would have all been new to the sales team and which couldn't be refuted on the spot. On later, further investigation they would have all turned out to be misstatements or plain ingenuous.

I had to laugh at storage/staff ratio. And I'm sure it was predictable that the SAN failures were lack of diversity as the likely cause. I wonder if you went to the customer site how much has changed to improve things.

[Martin G]

Oh I wish more vendors would front-up to senior management and have the guts to tell them where things are going wrong. Many times, you would find the storage team in quiet agreement with the vendor; they will have made many of the observations themselves internally and will have been hushed by senior management because it is perceived that doing things the right way is expensive.

The biggest battle that we fight every day is the tools one; now, it is fair to say that the storage vendors may not have done a great job here and the tools have been poor. But now, the tools are getting better and the estates are so large, that to manage without tools is getting incredibly hard.

But come to think of it, is that really the biggest battle; actually, sometimes I think it’s the technical refresh battle. ‘It aint broke, so don’t fix it’ or ‘Refresh doesn’t add any value, so we won’t do it this year’; key environments which are pretty much completely out of support and now people are scared to change them. So at some point, you end up looking at doing a complete replatform of an environment. Yet again, vendors don’t help matters; certification matrices can be opaque and complex. It is entirely possible to find that a firmware upgrade on an array could trigger a firmware upgrade on a switch which triggers a firmware upgrade on an HBA which triggers a driver upgrade on an operating system which triggers an operating system upgrade which triggers an application platform upgrade which triggers an application retest/verification. This is complex entangled infrastructure; [insert vBlock sales-pitch here] [insert vBlock rebuttal here].

Or is it the fight to instigate process? The customer who demands that they are unique and they should not have to follow change control? The customer who demands that their application goes in untested, unverified?

None of this has to be so but it takes some serious mind-set change.

1) IT is an investment and potentially a foundational investment; would you let your offices fall into rack and ruin? Would let your staff work in an environment with a leaky roof?

2) IT requires leadership but IT leadership requires listening skills. Listen to your teams, they will tell you what is wrong. If a team believe they need to spend money to make things better, work with them to build a business case and plan. Most people don’t go around spending their employer’s money for fun; believe in the people and be open to critique.

3) Be open to challenge from with-in and with-out. Don’t shut the conversation down; I think the biggest barrier to change is not the people on the shop floor, it is the people between the shop floor and the executive suite often.

4) What you can’t measure, you can’t properly and effectively manage. Believe this and invest in measurement.

5) Change your financial models, stop project-based, siloed infrastructure decisions. If you look at your estate and find utilisation is below 20% and your teams are still asking to buy more, ask them why? If the answer is that projects/business units are ring-fencing resources because ‘they paid for them’; don’t beat up the infrastructure team for being ineffective, beat up the CFO for allowing such a stupid model to continue.

6) Build a service-based model and allow an adult conversation. This adult conversation may involve saying ‘No!’ once in whilst.

7) Be serious about DR, Business Continuity.

8) Be honest about risk; if we do it this way and cut this corner, we will risk this system. Or we could do it better and not risk the business.

I was going to reply anonymously but it’s nothing I haven’t said before and it’s nothing I’m not prepared to stand-by.

[Chuck Hollis]

Martin

Excellent thoughts. Thank you.

-- Chuck

[Ed-R]

Good read that. I'm going to circulate it.
I've been in much the same situation - it's often the case that storage is blamed 'by default' as ... well, what amounts to the easiest target. Performance is slow - must be a storage problem.

I've got fairly adept at bouncing back reports that basically say 'not a storage problem' with graphs. But often, I find that the 'storage function' is just not well equipped to do that, can't defend 'their' turf, so get squeezed to pass it on to vendor support. Or otherwise end up having to 'fob it off' because nothing's going to happen without someone making a business case for it.

I also agree with Martin about changing 'internal' storage delivery. Capital expenditure, per project, to buy tin is inefficient at very many levels - it's only very rarely that such an upfront purchase is used immediately, and then with no growth for the entire lifespan.
The vast majority of storage allocations grow, and are overprovisioned at day one.
But you _have_ to switch away from 'project supplies capital, to buy hardware' to a model where the 'storage service' is a product sold by the gig-month and might come in a tiered model, based on how much performance is needed from those gigs.
That represents a substantial change to financial models though, and that can broker resistance. Capital expenditure for project deployment, to operational expenditure for ongoing service. Even if the 'end sum' ends up cheaper...

[David Patton]

Well done. The customer is NOT always right, but they are the customer. You gave them frank, honest feedback in a professional manner with an offer to help them solve their problem. This is much more valuable to them than just rolling over and accepting full blame.

[EQuinn]

Sometimes vendors get what they wished for, or marketed for: All this messaging about "convergence," SaaS and taking the pain-out-of-IT, plus years of outsourcing in the widest sense, well, guess what? It worked in many cases. The oft-berated-themselves IT department, tired of being caught-in-the-middle between not-quite-baked vendor offerings, CEOs asking for lower costs, users asking for more functionality and better support, bought the "depend more on us" pitch from vendors, integrators and service providers. The result? Some IT departments have gone so slender, focusing on procurement and simplified operations, their expertise has eroded significantly.

This isn't just a storage issue, it permeates IT. Rather than sell IT on ease-of-use and haggle on the price premium for that often empty promise, vendors should win on that difficult old concept of "partnership." We are in this together, vendor and IT, and it includes:

* Training, yes, so painful, but so useful long-term; closely related to this is transitional training - when there is turnover to key IT personnel, ensure the replacement is up-to-date

* On-going, formalized, post-sale communication; the drop it and run to meet the next quota technique might buy Larry Ellison the America's Cup, but it puts customers at a huge disadvantage. IT departments that don't keep the vendor in the mix as their conditions change are equally as guilty.

* Transparency in marketing (up to a point of course) but most importantly at point-of-sale and configuration: "This solution works for this context, but if that context changes, we better talk." And keep repeating this mantra.

* And finally, the notion that an SLA is a two-way street, a process. Texting every quarter "did you hit the SLA?" doesn't work and neither of the partners should stand for it. Yes, IT needs to understand, and prepare for the fact that vendors are human too and mistakes are made. IT needs to understand that about themselves as well. The blame game leads to, well, just look at our current U.S. political system. Doesn't work in IT either.

Regarding tools, well, some tools inflict further confusion, some help, but regardless they are not a panacea.

Given the never-ending cycle of quotas and quarterly-focused business models of IT vendors, the cold truth is this all mainly falls at the feet of IT. But the other side of the truth is vendors winning on throw-it-over-the-wall solutions have helped create the phenomena of emasculated IT, therefore shouldn't complain much when it comes back to bite them.

[Richard Hintz]

+EQuinn,
In the specific case that the post mentions, the storage staff level is ludicrously high, given the total storage managed, so I don't think it qualifies in your slender IT category, though I understand what you mean about level of expertise. But, just to take the specific points you raise, are you saying that ensuring that customer technical staff are trained is the responsibility of the vendor?

And as far as ongoing, formalized post-sale communication, in my experience, this is hard to avoid, especially when the vendor has a services arm. If a customer wants more communication, it's usually as simple as making a phone call. Usually at some point the free consulting stops and the meter starts running, but this is typically negotiable if there's a severe performance problem. I can't see that the vendor is responsible for an overall system outage when the architecture didn't include diversity, for example. They are, of course, responsible for outages of their own components according to whatever they represented contractually.

Transparency in marketing means, too, that the customer understands their own environment and, especially, doesn't have any back-level, legacy hardware/software booby traps.

I didn't understand your point about SLA. If something is wrong, one typically doesn't wait for an end of quarter SLA audit, especially for something like storage.

Tools: trying to run a modern environment without appropriate support tools is a recipe for disaster, especially in a multi-vendor environment.

Sorry, but I really don't understand your push back. It seems as if you want to put an inappropriate level of what's supposed to be technical (and general) management responsibility on the vendor. Presumably all the decision makers on the customer team have had things sold to them before and they can apply a sanity check to vendor claims.

(I'm a customer, not a vendor, at least in this context.)

[Jonas]

One of the great things about EBCs is that folks with no direct "ore in the boat" can come in and deliver a clear message that may hurt a bit, then leave. While the account team can blame you for being a "bad guy", the bottom line is you did get through to the customer and probably stopped a dysfunctional cycle that wasn't helping anyone

[Chuck Hollis]

Not to be too controversial here, but I wonder how much of this just All Goes Away when using something like a Vblock.

-- Chuck

[Richard Hintz]

+Jonas,
Except look at the number of "time to move on," "that angle wasn't working well," "Enough on this topic." The message was being transmitted, but only received obliquely. (Yes, I know this isn't a verbatim record, but I've been in these sorts of meetings.)

I'd be amazed if anything changed at the customer site except a bunch of conversations about how EMC was unresponsive and unwilling to accept their responsibility.

[Chuck Hollis]

Richard

Interestingly enough, it turns out that EMC ended up proposing our SMS -- storage managed service -- to this customer. I've heard anecdotally that there's incredibly strong pushback from the storage team.

At least there's a decent alternative on the table to consider -- from EMC.

-- Chuck

[Richard Hintz]

+Chuck,

The storage team is pushing back because they realize they have to work their way out of their legacy tail and can't digest the SMS solution simultaneously. Sounds weird, but I'd bet on it.

[Martin G]

How much goes away when using an Infrastructure stack such as Vblock? It's an interesting one but I think it you put any kind of Infrastructure stack into an organisation such as the in original article; you end up with the same mess and probably a more expensive mess for the vendor and the customer to sort out.

You know as well as I do, infrastructure stacks are not the solution or certainly not the whole solution. Without real cultural change; you'll fail.

Vblock may be a short cut to the infrastructure changes you need to make but you can happily do it with any infrastructure; even the customer you mention could probably do it but they need to want to do it.

And in my experience, if you take a mess and turn in it into a managed service; it'll just be a mess run by other people with a bunch of pretty reports which polish a turd. Unless there is real commit by the customer to invest in fixing said turd, you'll end up with frustrated customer and supplier.

[Chuck Hollis]

Martin

A great point. We have seen a few Vblocks go in without that commitment to change we're both talking about, and it hasn't been pretty in the least :)

As far as managed services, I can only speak to EMC's version, and not others. Part of our proposal is always investment in transformation: process, technology when possible, and always have the right people.

There's also a "hand back" option available, which is popular. EMC invests (and gets paid for) setting things right, and -- once done -- the customer has the option of taking the operation back again.

Thanks again

-- Chuck

[Richard Hintz]

Vblock

Keep in mind this earlier comment:
"Another serious outage resulted when the customers' team were moving stuff around, and accidently trashed production data. EMC got the phone call after the damage was done. There were problems recovering from backup..."

Oops, how to get the running apps from Point A to Point Z, Vblock nirvana.

Given everything else, I also bet they're constrained on power, space, cooling, so bringing in Vblock is problematic, even if it would be a net facilities win once the transition was over.

I also assume that they are substantially pre-virtualized, so that's another barrier.

Last, given everything else, I assume it would take a year for their procurement to work out the software licensing issues. Those are just the highlights without actually knowing anymore than what you've said, much less doing any real analysis.

[Az990tony]

Good post Chuck! I can certainly relate.
-- Tony Pearson (IBM)