Most devops discussions are missing the target

I am hearing a lot about the rise of a concept called ‘devops’ – a mashup of ‘development’ and ‘operations’. I am not at all an expert in this area, but from what I can tell, devops is aimed at streamlining rapidly iterative application delivery to allow for greater development and business agility. Devops aims to achieve this by breaking down the barriers – human, process, and technology – between application development and system operations.

Interestingly, the concept is new enough that, as I write this, there is not even an entry for it in Wikipedia yet. I did find a blog by Damon Edwards (on Twitter – @damonedwards) very useful though, as he explains the age-old disconnects between application developers ‘throwing software over the wall’, and ops who are painfully resistant to change. James Urquhart (@jamesurquhart ) blogged very recently on the concept too , and again provided some very helpful content. Conversing online with them and others also helped me to formulate some more concrete ideas about devops – or at least some more concrete questions.

My interest was especially piqued when I understood how closely devops is connected to virtualization, cloud, and automation – my core interests:

• Cloud – Devops has antecedents in ‘rogue’ developers (or developers from smaller shops) using cloud resources (IaaS, PaaS) for new projects, and will benefit greatly from cloud-based development and deployment, as cloud providers do not impose the restrictions of internal change-averse ops teams, and developers can essentially manage their own ops requirements instead.
• Virtualization – In-house devops (which needs more heavy lifting) is greatly assisted by virtualization, as virtual machines become the new base unit for application packaging, avoiding application rollout failures caused by incompatibility between the test and production environments (hardware, OS, middleware, etc.).
• Automation – In-house devops is also greatly facilitated by automation, which can use standard workflows to automatically provision and configure these complete application VMs, as well as backup and restore VMs, allowing complex composite application deployment and rollback at the click of a mouse.

Clearly devops has many very attractive outcomes – drive agile business, reduce delays, smooth application releases, deliver value faster. It is a very seductive idea. Who wouldn’t want it?

However, most of the writings I see about devops are really about dev, not ops. As a result, they don’t really capture the whole story of the application lifecycle. They justify devops as an antidote to the problems that ops are causing – slowing down release cycles, imposing arbitrary rules, screwing up deployments, killing developer productivity, hacking manual scripts and configs, stopping the business from being agile – but fail to recognize both the failings of developers that contribute to the problems, and the role of operations in delivering critical business outcomes during the application delivery lifecycle.

On the contrary, discussions mainly focus on how developers can sideline or change operations, positioning devops as the lone hero in the battle against inefficiency, as application developers fix all the problems (!) by controlling or automating key release management operations like provisioning, deployment, integration, patching, and software update. Meanwhile, ops are marginalised, along with their timesinks and roadblocks, satisfying the needs of an agile and rapidly changing business.

See – seductive, isn’t it?

Yet this seems to me (as a former op) fundamentally flawed, a development-centric neologism based on an incomplete understanding of the real purpose and role of IT operations, or of operations’ history in the development of ‘agile’ IT.

The way I see it, devops misses that target on how IT ops serve business needs too, and seems to gloss over ‘coal face’ realities like:

• Who handles ongoing support, especially software update for the unrestrained sprawl of non-standard systems and components.
• Who ensures each new application doesn’t interfere with existing and especially legacy systems (and networks, storage, etc.)?
• Who handles integration with common production systems that cannot be encapsulated in a VM, like storage arrays (NAS, SAN), networking fabrics, facilities, etc.
• Who handles impact analysis, change control and rollback planning to ensure deployment risk is understood and mitigated?
• Who is responsible for cost containment and asset rationalization, when devops keeps rolling out new systems and applications?
• Who ensures reporting, compliance, data updates, log maintenance, Db administration, etc. are built into the applications, and integrated with standard management tools?
• Who will assure functional isolation, role-based access controls, change auditing, event management, and configuration control to secure applications, data, and compliance?

Because, if you have ever worked with both ops and apps, you know it is not going to be apps.

Now, in defence of devops, I am sure it is being implemented and conferring major benefits, especially in small organizations with little IT management discipline. I am sure the supporters of devops have some positive goals in mind too. What’s more, it is addressing a very real problem – ops really should spend more time on better processes and controls than in ‘daily deployment muck’.

However, devops should be a two-way street. As a former op, I know that the apps team have to pull their weight too, by addressing gaps like:

• Including ops during the design process, so applications are built to work with standard ops tools
• Taking ops input on deployment, so applications will go in cleanly without disrupting other users
• Working with ops on capacity and scalability requirements, so they can keep supporting it when it grows
• Implementing ops’ critical needs for logging, isolation, identity management, configuration needs, and secure interfaces so the app can be secure and compliant
• Giving ops some advance insight into applications, especially during test and QA, so they can start to prepare for them before they come over the wall
• Allowing ops to contribute to better application design, deployment, and management; that ops can do more for the release cycle and ongoing management than just ‘manipulating APIs’

See, ops do enable business – and agile business at that – by ensuring that new applications coming into an existing complex environment are safe, secure, reliable, integrated, and responsive, regardless of how complex IT is, or how many moving parts there are. Devops seems to miss this important detail.

So I am sceptical of how devops will work in large, well-run IT environments with important and necessary operational controls, especially the > 60% of organizations that are committed to ITIL best practices (like formal and integrated management of change, configuration, release, assets, etc.).

After all, ‘agile’ does not magically obviate the need to identify and prevent bad changes, to reject apps that breach operational compliance, to ensure each new application adheres to standards, or to prevent uncontrolled sprawl of heterogeneous software.

I still have a lot to think about on this topic, and am trying to keep an open mind. But my best guess right now is that, for enterprises at least, devops either will not take hold or will not last. It seems most likely to be instead, at best, a transitory state on the path to a ‘new normal’. As with all ‘revolutions’, it has started outside IT ops, yet I expect will eventually co-opt and migrate wholly to operations in some form. Once the revolutionaries in development understand how many business needs besides agility actually require  routine, process, management, and controls, they will back away from devops the same way they backed away from ownership in other IT revolutions – like the deployment of mini computers, desktops, and web applications.

If it does turn out this way – don’t worry. Operations will again dutifully take the reins, and clean up the mess that devops will leave behind. Because that is what ops do – they manage what they are given, and keep the business running, regardless of the mess that gets thrown over the wall at them.

In any case, whether devops takes root or not, hopefully we will all learn something about cooperation, automation, agility, and control. Because all stakeholders in the devops discussion – development, operations, and business owners – could benefit from that.

Microsoft and Opalis

Today Microsoft Corporation (NASD:MSFT) announced a definitive agreement to acquire Opalis Inc., the leading independent vendor of IT Process Automation (ITPA) software.

IT Process Automation (ITPA) is a Data Center Automation (DCA) discipline that EMA defines as “the ability to automate and integrate the workflow of complex, multi-discipline IT management processes.” This automation can replace many manual, resource-intensive, and error-prone activities that typically cross multiple IT components, disciplines, and/or departments. ITPA delivers exceptional results including freeing up 77% more staff for strategic projects, providing more than 60 additional hours of system availability per year, and saving an average $500,000 more per year on staff costs than other Data Center Automation (DCA) disciplines.

This space has been gaining interest, both expanding and consolidating, for some time, as evidenced by significant development and acquisition activity from Novell (ZENworks, PlateSpin), HP (Opsware, iConclude), BMC Software (RealOps, Atrium), NetIQ (Aegis), Symantec (T-Logic, Altiris), and CA (Optinuity, Spectrum).

I think this is an excellent move by Microsoft. It will certainly make customers of both companies very happy. Microsoft and its customers gain an exceptional solution, in a discipline area that Microsoft was clearly lacking, and one which delivers many proven and exceptional benefits. For Opalis customers, it is probably a mixed bag. It will be a major change, but with Microsoft’s strength and stability, it is likely to be a positive outcome overall for Opalis customers.

This is, however, a huge blow for competitors, especially for the few large management vendors that have not yet acquired or built an ITPA solution or components, or whose own ITPA capabilities are less than stellar. For other large mgmt vendors with credible or better ITPA capabilities, this is both an opportunity and a threat. For mid-sized vendors that compete with Opalis or Microsoft Systems Center, and especially smaller vendors, this is a horrible result. Overall, most vendors will have to hustle to respond, although many will be unable to do so.

Meanwhile, Microsoft, Opalis, and their customers should be ecstatic with this deal. Few acquisitions are so clearly positive for the stakeholders as this.

You should be able to check out what the executives from both companies have to say in their blog posts:

• Blog post from Brad Anderson, Microsoft Corporate Vice President: http://blogs.technet.com/systemcenter/default.aspx
• Blog post from Todd DeLaughter, President & CEO of Opalis Software: http://www.opalis.com/blog.asp?id=1

Meanwhile, I will be expanding on the impact of this acquisition very soon with a full EMA Impact Brief. Keep your eyes out for that one – lots of significant implications for customer and competitors, without doubt!

I tweeted that I think such predictions are useless when most CIOs must prove return on investment (ROI) for major IT projects in less than 6 months. Tajeshwar got me thinking more deeply about this idea with his reply:

“cio demanding roi<6 mnths r taking tactical view;3 year tech horizon must for taking strategic view & decisions”

Indeed, this is emblematic of a really interesting challenge for CIOs.

The demand for a rapid ROI, typically less than 6 months, and in some cases shorter, is a fact for today’s CIO – even more so today than before the global economic downturn. I firmly believe that CIOs demanding ROI in less than 6 months are simply realizing and reacting to the modern reality that IT can no longer be a pure cost center. Ask almost any CIO, and you will know that the ‘blue sky’ IT projects that delivered results in 2-3 year timeframes are a thing of the past.

However, as Tajeshwar implied, this demand works directly counter to the mandate for great CIOs to think and act strategically, executing on a long-term corporate vision. The same CIOs that are trying to contain or reduce costs – essentially a ‘cost center’ approach – must also be acting to make IT a strategic asset.

This does not mean that strategic CIOs are dead, or even a dying breed. On the contrary, the ability to accurately envision future trends and get a head start on competitors is perhaps more important than ever, because the rate of change in IT is so much faster, and the barriers to entry for new technology innovations seem to be always decreasing.

So to be a great CIO you need to act tactically, with projects that contain costs and deliver ROI in less than 6 months; yet also provide the business with a strategic launchpad for innovation, competitive advantage, and shareholder value.

What sort of projects can do this?

How about:

Virtualization

This is perhaps low-hanging fruit. My research for EMA clearly shows the key outcomes of virtualization are well divided between short-term ROI and long-term strategic benefits.

For example, in the short-term, virtualization reduces hardware, power, cooling, administration, rent and even software costs. Around 90% of enterprises report that it delivers real, measurable cost savings. Loading up 15 VMs or more on each physical server, allowing admins to manage on average a 10% greater workload, saving an average of $200 per system on administration costs, adding as little as $37 for each new VM in administrator staff costs (up to 28 times less than a physical system), and reducing power costs by an average of 17% are rapid and significant ROI values.

In the long-term, faster system provisioning helps bring products and services to market faster, better DR capabilities provide a strategic defense against disasters and epidemics, and better workload and resource balancing provides faster response times and better customer service – a range of strong strategic opportunities.

IT Process Automation (ITPA)

In the short-term, EMA research has shown that sites with ITPA improve their MTTR, provide almost 65 hours extra availability per year for 24×7 operations, and sites with ITPA (typically larger data centers) save on average around $500,000 more per year on staff costs alone than sites without it (easily offsetting marginally higher staff salaries). These outcomes all provide substantial short-term ROI.

Meanwhile, 95% of enterprises report that ITPA achieves one or more strategic goals, such as improving the ability to adapt to rapid change (like rapidly integrating M&A), freeing up high-level staff, providing better security and compliance, reducing business and IT complexity, reducing human errors, and integrating with best practices. Moreover, 76% report that ITPA helps achieve 2 or more of these goals, and 55% report it helps achieve 3 or more. ITPA also correlates with an overall increase in IT maturity.

Lifecycle Management

In published EMA case studies, automated lifecycle management reduced regular maintenance windows for 50 systems from 2-3 days to just 10 minutes each, and cut the cost of  distribution of a new version of Microsoft Office from $90,000 to just $30,000. It also can help to reduce overall software license costs, allocate and reuse hardware more effectively, improve end user uptime, and reduce or eliminate the (often substantial) travel, staff, and downtime costs of desk-side visits to install new software or fix problems.

EMA research also shows that automated lifecycle management helps to achieve strategic objectives. It provides faster and better service to end users (and ultimately therefore to customers), enables IT and business staff to be more productive, lets business users take advantage of new software and systems much faster, provides essential compliance reporting, and maintains strategic security values.

Part of the reason that these technologies are both tactical and strategic is that they can all be implemented in short, sharp, phases that deliver fast and specific results, while establishing a technology basis that can be leveraged – reused, over and over, in multiple new ways – to deliver strategic benefits with little or no additional cost.

For a great CIO, such technologies are invaluable. They show fast results, justifying budgets and building confidence; yet they deliver technologies they can continue to leverage for better and better strategic outcomes.

All of which meets the needs of today’s CIOs much better than blue-sky, multi-year, technology dreamings.

Cloud Building Blocks

In very simplified form – and heavily out of context – it shows how Infrastructure Automation, Virtualization/Virtual Systems Management, and IT Service Automation form the underpinnings for Cloud Computing, with security and compliance as ever-necessary guardians across all disciplines.

Now, this slide is specific to cloud computing, so I do not claim it to cover all IT or automation requirements. Moreover, it is intended more as architectural maturity model, describing how a lower level of automation must for the basis for, and subsequently extend to, a higher level of IT abstraction.

However, I was challenged by a client to explain why some advanced automation technologies – like IT Process Automation, or Workload Automation – were not in the ‘IT Service Automation’ category, or even in a high-level “Business Automation” category of their own.

Even given the specific limitations, audience, and missing context of this chart (and the broader discussion), I think that is a valid point, and worth addressing more publicly.

Firstly, on the location of IT Process Automation (ITPA, aka ‘run book automation’), I see this technology as predominantly managing machine-to-machine process automation, i.e. connecting individual tasks on separate systems or software engines, integrating and orchestrating these tasks, in order to automate complex IT processes.  By contrast, I see business or service processes as predominantly human-to-human or human-to-machine. This is why I put ITPA into infrastructure automation, not service automation.

Nevertheless, I do think it is reasonable in some circumstances to locate ITPA as both Infrastructure Automation and Service Automation – at least in more mature or emerging use cases. Certainly EMA is starting to see ITPA as a key enabler of business service automation, deeply connected to service automation capabilities like Service Desk and Service Catalog, rather than just for automating  (for example) infrastructure activities like provisioning, change, security administration, or data integration. Arguably this is even a new category of its own (Business Service Automation? IT Service Automation?); arguably it fits into existing categories (Business Process Management?).

In any case, I do see a clear distinction between the low-level use cases illustrated in this chart, and more high-level use cases that are not clearly shown.

Perhaps Workload Automation crosses both areas too, but I would say considerably less so. Indeed, I am not at all convinced that workload automation – which by EMA’s definition differs specifically from job scheduling/batch processing by virtue of being cross-platform, event-driven, reactive & predictive, application-aware, and business-driven – is primarily a top-level business service function. Like ITPA, it does participate partially in higher-level Service Automation; however, I believe it is a technology that is primarily used by business service systems, and only secondarily a technology that uses other technologies. So I do believe it is still most appropriate to categorize Workload Automation – even  more evolutionary varieties of Workload Automation – as Infrastructure Automation, not Service Automation.

In any case, the structure of my diagram does not preclude certain evolutionary technologies existing at multiple layers.

However, I do not think this chart in particular needs a 4th layer to represent Business Automation. Of course, my label of IT Service Automation may be overly simplified (or even plain wrong); maybe it should instead be called Business Service Automation, for example. It certainly aims to incorporate many of the attributes my client thought should be in a separate layer, around enabling and reacting to requirements that have real and clearly visible business impact. However, I specifically wanted to avoid the implication that the building blocks for cloud computing, from an IT perspective, required deep application integration (i.e. beyond batch or IT processing) in business-facing areas like ERP, BI, Data Warehouse, etc. I see these as desirable outcomes, rather than necessary foundations, for cloud computing.

Still, I certainly do agree there is an ‘other’ that is missing in this chart (well, several ‘others’ actually, but one in particular) – the business function that exists primarily outside of IT. It is certainly not clear in this slide, but it is without doubt fundamental to the intelligent automation of IT, that business needs must underpin cloud computing. Any IT department (whether using on-premise cloud or off-premise cloud, or even running a ‘traditional’ data center) that just automates IT with no regard for true business drivers and outcomes, should re-examine what they are doing, and why. While I leave the business inputs and outputs as  assumed requirements in my chart, any IT organization that does not fully incorporate business drivers and outcomes into their processes – or worse, ignores them completely – is doing itself, and its business owners, a severe disservice.

Andi.