By all that is right and just in the world, advanced automation should be the holy grail of modern enterprise IT. You might expect progressive IT organizations to be investing like crazy in a never-ending race to automate, automate, automate — an arms race that never ends.
If that’s happening, I’m not seeing a lot of it. When I talk to enterprise IT organizations, automation is usually on the list of “yeah, we’re working on that too”.
This puzzles me.
On one hand, there’s ample evidence from many non-IT industries that a continual pattern of heavy investment in automation is required to successfully compete. On the other hand, so few enterprise IT organizations seem to be willing to organize and invest appropriately.
Everything exists for a reason, and I think I’ve come up with a handful of reasons why the disparity between opportunity and execution remains so great.
For those of you grappling with these issues in your own environments, see if any of these apply to you.
The Promise Of Automation
There’s a long list of extremely competitive industries that have been fundamentally transformed by successive waves of automation: manufacturing, telecommunications, drug research, financial services, logistics, etc.
Look closely, and you'll see that automation isn’t the icing on the cake, it’s the cake itself: the investment in continually improving and automating operational processes is a critical component of the firm’s competitive advantage.
When we consider the current situation of enterprise IT, it’s ripe for the same investment pattern.
Demand grows, budgets don’t.
The only pragmatic solution to the equation is massive leaps in productivity and efficiency — and that means automation.
By “automation”, I mean a focused, well-funded and never-ending effort to (a) identify core business processes, (b) measure their current effectiveness, (c) propose redesigned processes that are more efficient, (d) test and implement them, and (e) measure their resulting effectiveness.
If you’ve been involved with Six Sigma, this is the familiar DMAIC: discover, measure, analyze, implement, control.
I’m specifically not talking about pouring scripted automation concrete around existing archaic processes.
So, why do we continue to see a gap between theory and practice? I have my suspicions …
#1 — It’s Intellectually Difficult
Getting your head wrapped around a single IT process takes some effort. Getting your head wrapped around dozens — or hundreds — of processes can be mind-melting. Understanding those processes at a deep enough level that common patterns and architectures emerge as a basis for re-engineering -- well, that takes serious intellectual horsepower.
Teams of seriously smart and knowledgeable people who work together well aren’t all that common, and that is what’s required here. Instead, I tend to see well-meaning — but woefully under-resourced — efforts: maybe a semi-dedicated team lead and a bunch of people helping out.
Compare that investment pattern with other industries that have come to accept continually-improving automation as a core business tenet, and there’s a sharp contrast.
#2 — It’s Politically Unpopular
Serious process redesign and subsequent automation always cuts across traditional organizational boundaries: that’s what makes it so darn powerful.
But as you do that, you’re messing with people’s worlds: changing their roles, changing how they get work done, who they work with, etc.
And although we all talk a good game, no one likes to have their world messed with — especially if you’re not in control. I am no exception.
Most people prefer to be well-liked where they work (hey, we're all peer-reviewed!); being seen as a well-intentioned but unpopular troublemaker isn’t everyone’s cup of tea.
#3 — It’s A Never-Ending Journey
“Are we there yet?” No. “When will we be done?”. Probably never.
If we look outside of IT at automation-centric business models, they never want to stop improving, so they never stop investing. The notion that a bunch of money ia going to be spent on something in perpetuity -- and will never finish — well, that’s a tough sell.
Will there be continual and meaningful improvement along the way? Of course. Can we package this as a well-defined project, with a clear beginning and end? Of course not.
It’s generally accepted in the business world that everything must continually improve over time — just to stay competitive. And in a world where so much business depends on IT, it too must continually improve.
And that takes continual investment.
#4 — The Vendors Aren’t Always Helpful
I have waded through many of the different management frameworks and tools, and how they are marketed. At least for me, it takes serious effort to decode what they’re actually saying.
Some clearly fall into the “unicorns and fairy dust” camp: magical and effortless improvements simply through acquiring the right software.
If it was that easy, everyone would be doing it.
Others are far more pragmatic: here is what we do well, here is how we integrate with others -- and here is what you the customer are responsible for. The landscape is littered with automation projects that didn't automagically self-materialize once the purchase was made.
I am reminded of the first wave of SAP implementations back in the 1990s. The winners realized that SAP wasn't simply new software, it was a new way of running your business. The failures could often be traced back to well-intentioned "customizations" that reflected the old way of doing things.
#5 — The Supporting Architectures Can Be Complex
As before, better processes and better automation means crossing traditional boundaries — and lots of them. That means that simply understanding how the various management and automation pieces fit together is also intellectually challenging.
If you’re using a dozen or more tools to automate, that means you’ve also invested in a dozen or more rather specialized skill sets, suites of interfaces, etc.
From what I’ve seen, as IT organizations move from model to model along their automation journey, they tend to frequently discard tools that made sense in the old model, but don't fit the new one.
Getting comfortable with a pattern of continually investing in — and de-investing in — multiple management tools: licenses, skills, interfaces, etc. — well, that’s a really big pill to swallow for many people.
Getting To “Automation First”
Where does that leave us? With a big opportunity — and a correspondingly big set of challenges.
If "virtualization first" was the rallying cry of our recent wave of IT thinking, perhaps "automation first" should be our call sign for the next one.
I tend to think about this through a decidedly Darwinian lens: those IT organizations that fail to invest heavily in automation will likely suffer at the hands that do: be they external service providers, or embedded in companies that compete directly with yours.
In an ideal world, what might a winning strategy look like?
#1 — Invest In An Empowered, Well-Resourced Team
Process change and subsequent automation at scale is not trivial work. Several distinct skills are needed: people who can visualize new workflows and processes, people who understand how all existing bits fit together from both a business and technology perspective, people who can sell change and make it happen, and more.
Whether these are your own people, an outside consultancy, or preferably a mix — without the right set of dedicated people, it’s hard to even get started.
This inarguable requirement comes at a time where many IT leaders realize that they're in a war for talent, and automation just opens up another front in the battle :)
#2 — Create A Model For The Journey
Everyone involved will need to understand three things: where we are today, where we’d like to be at some point in the near future, and how we expect to get there. People can’t execute on what they don’t understand.
I’ve seen a few different approaches, each with their merits. One example is to pick a very focused operational area (e.g. financial reporting, provisioning, etc.) and start there. As success is seen in one area, other areas are considered.
Another apparently successful model is greenfield: when a sizable new environment is being stood up (e.g. private cloud, VDI, big data, etc.) use it as a lab to try out entirely new ideas around process automation. Benchmark the new environment against existing practice, and use the disparity as a lever to drive change.
One model that doesn’t seem to work is the intergalactic approach: mapping out everything that has to change, and all at once. People can’t grasp the complexity; there’s no obvious place to start, by the time the pieces are in place your understanding of the problem has fundamentally changed, etc. It doesn't start to fail, it fails to start.
A model that starts well — but frequently stalls — is the “automate within disciplines” approach: server team automates, network team automates, storage, etc. While there are frequently good improvements to be seen, at some point you realize you need to get good at process design and automation across the silos (err, “cylinders of excellence”) rather than within them.
#3 — Invest In Stuff That Supports Advanced Automation Models
Some people believe that -- theoretically -- any entity in the IT universe can be automated -- all it takes is a little scripting! I would argue that there is a huge difference between products that can be potentially automated -- and those that are designed to be automated.
Driving to successively more productive levels of automation can be directly impacted by how automate-able the components are. Especially if you see yourself on an inevitable journey of progressively deeper and more extensive automation.
Indeed, this is one of the big ideas behind SDDC (software-defined data center) and all of its SD-related disciplines: software-defined networking, software-defined storage, software-defined security et. al.
For example, in my happy little storage world, it’s not enough to say “we have APIs”; I must apply a higher standard: anything a human can do from a console or GUI should be programmatically accessible, along with bindings and adaptors to popular frameworks, sample workflows, etc.
Not coincidentally, that’s how my employer VMware thinks about these things.
Ask yourself: when your team is debating what to buy, how much importance is given to programmability and automate-ability? Pushing the point: would you prefer a solution that’s strong in this regard, but may be less attractive in other aspects?
That's part of the "automation first" mindset.
And every time an Exa-something lands on a data center floor, a kitten dies.
#4 — Defend Against Distractions
Once all of this is in place, be prepared to defend it against the latest crisis-du-jour or recent hot project.
I suspect that these process and automation teams can only make progress if they stay focused on the task at hand, and not brought in as extra resource on the dozens of distractions that are an integral part of the enterprise IT world.
But that's exactly what I see strong leaders do: set the priorities, make the investments and firewall resources so the long-term stuff doesn't suffer at the hands of today's fire alarm.
What Really Makes This Hard
Part of the inherent challenge is that business leaders aren’t directly asking for better process and better automation: they’re asking for better apps, mobility, analytics, lower costs, improved agility, etc.
Not that anyone in IT would think of themselves as an order-taker, but ...
The natural tendency is to give people what they want, when they want it — rather than making the hard investments in the people, process and tools required to create a continually-improving automated environment that does all of this, and much, much more.
Nothing good is easy, nothing easy is good.
------------------
Like this post? Why not subscribe via email?
the biggest problem with automation, the promise of the technology approach and the actual manifestation of it is that it feels too damn hard.
You have vendors out there sell automation / orchestration products with the "of course it will work & plug into your existing processes and not telling the reality of it. You have potential purchasers that thing they are mature and ready enough for automation but they are no way near it and then you have all of the *confused* souls that sit in the middle that just get directed all over the place.
Your comments around get a good team together, understand the inadequacies that exist today and plan for it are well made
My other advice... get a good consultancy in that has done this before to guide you along the way.... people that give you the truthfull answer and tell you that its gonna be bl00dy hard.......
Also - be prepeared to change the way that technology and the business inter-operate together.
I shall carry on this rant later
@stuiesav
Posted by: Stuart Savill | March 11, 2014 at 01:30 PM
Very good overview of the challenges and indeed we're by far not where we actually should be with automation .....
I'd like to add one more hurdle: the inability to stick to the KISS principle: keep everything stupid and simple (at the start and during the implementation). Too much money and effort is spent on supporting and implementing Exceptions and as we all know exceptions cannot be automated !
Posted by: Henk ten Bos | March 12, 2014 at 01:12 AM
Hi Chuck,
Very, very true. Just want to add one thing to the equation of "demand grows, budgets don't.
Part of the buying strategy should not only be to buy solutions that are designed to automate but also to buy simple solution's. Today's solution's are far to complex to operate, think of enterprise storage array's where you almost need a universal study to operate. I think VSAN is heading in this direction. We need dramatically simplified solutions. I think simplify first, then automate complexity.
Rgds, Jerry
Posted by: Jerry Rozeman | March 12, 2014 at 11:51 AM
I'm currently working on a project that is looking at how we transform to a cloud like, or ITaaS model and I think the half the challenge I've noticed a couple of things: that everyone wants it, but they don't want to put the effort into the 'non sexy' bits like documenting and streamlining the processes that they then want to put into the cool automation tool from VMware et al. I've also noticed none of the automation tools are particularly mature. If you want to create VMs and have integration to common things like AD then great, but simply saying these tools work with APIs, REST, XML etc to plug into everything else isn't a great help and it just adds additional complexity. Until certified integration packs are common across the majority of services a typical IT org deploys are introduced, it will not fall into the same product category as vSphere for example which 'just works'. I think its this complexity that will be a major hurdle in terms of adoption - it just isn’t easy enough. If there was ever a niche in a market that was there to be exploited it’s in the creation of integration packs to make a Cloud Management or Automation platform talk to everything else with minimal configuration.
Posted by: David | March 13, 2014 at 05:50 AM
Don't even get me started on how Security needs to be part of this, from the beginning.
Posted by: Mike Foley | March 16, 2014 at 09:12 PM