Myopic View of DevOps Misses the Mark

March 26, 2010
Missing the target

Most devops discussions are missing the target

I am hearing a lot about the rise of a concept called “devops” – a mashup of “development” and “operations”. I am not at all an expert in this area, but from what I can tell, devops is aimed at streamlining rapidly iterative application delivery to allow for greater development and business agility. Devops aims to achieve this by breaking down the barriers – human, process, and technology – between application development and system operations.

Interestingly, the concept is new enough that, as I write this, there is not even an entry for it in Wikipedia yet. I did find a blog by Damon Edwards (on Twitter – @damonedwards) very useful though, as he explains the age-old disconnects between application developers “throwing software over the wall”, and ops who are painfully resistant to change. James Urquhart (@jamesurquhart ) blogged very recently on the concept too , and again provided some very helpful content. Conversing online with them and others also helped me to formulate some more concrete ideas about devops – or at least some more concrete questions.

My interest was especially piqued when I understood how closely devops is connected to virtualization, cloud, and automation – my core interests:

  • Cloud – Devops has antecedents in “rogue” developers (or developers from smaller shops) using cloud resources (IaaS, PaaS) for new projects, and will benefit greatly from cloud-based development and deployment, as cloud providers do not impose the restrictions of internal change-averse ops teams, and developers can essentially manage their own ops requirements instead.
  • Virtualization – In-house devops (which needs more heavy lifting) is greatly assisted by virtualization, as virtual machines become the new base unit for application packaging, avoiding application rollout failures caused by incompatibility between the test and production environments (hardware, OS, middleware, etc.).
  • Automation – In-house devops is also greatly facilitated by automation, which can use standard workflows to automatically provision and configure these complete application VMs, as well as backup and restore VMs, allowing complex composite application deployment and rollback at the click of a mouse.
“Clearly devops has many very attractive outcomes. It is a very seductive idea.”

Clearly devops has many very attractive outcomes – drive agile business, reduce delays, smooth application releases, deliver value faster. It is a very seductive idea. Who wouldn’t want it?

However, most of the writings I see about devops are really about dev, not ops. As a result, they don’t really capture the whole story of the application lifecycle. They justify devops as an antidote to the problems that ops are causing – slowing down release cycles, imposing arbitrary rules, screwing up deployments, killing developer productivity, hacking manual scripts and configs, stopping the business from being agile – but fail to recognize both the failings of developers that contribute to the problems, and the role of operations in delivering critical business outcomes during the application delivery lifecycle.

On the contrary, discussions mainly focus on how developers can sideline or change operations, positioning devops as the lone hero in the battle against inefficiency, as application developers fix all the problems (!) by controlling or automating key release management operations like provisioning, deployment, integration, patching, and software update. Meanwhile, ops are marginalised, along with their timesinks and roadblocks, satisfying the needs of an agile and rapidly changing business.

See – seductive, isn’t it?

“This seems fundamentally flawed, a development-centric neologism based on an incomplete understanding.”

Yet this seems to me (as a former op) fundamentally flawed, a development-centric neologism based on an incomplete understanding of the real purpose and role of IT operations, or of operations” history in the development of “agile” IT.

The way I see it, devops misses that target on how IT ops serve business needs too, and seems to gloss over “coal face” realities like:

  • Who handles ongoing support, especially software update for the unrestrained sprawl of non-standard systems and components.
  • Who ensures each new application doesn’t interfere with existing and especially legacy systems (and networks, storage, etc.)?
  • Who handles integration with common production systems that cannot be encapsulated in a VM, like storage arrays (NAS, SAN), networking fabrics, facilities, etc.
  • Who handles impact analysis, change control and rollback planning to ensure deployment risk is understood and mitigated?
  • Who is responsible for cost containment and asset rationalization, when devops keeps rolling out new systems and applications?
  • Who ensures reporting, compliance, data updates, log maintenance, Db administration, etc. are built into the applications, and integrated with standard management tools?
  • Who will assure functional isolation, role-based access controls, change auditing, event management, and configuration control to secure applications, data, and compliance?

Because, if you have ever worked with both ops and apps, you know it is not going to be apps. :)

Now, in defence of devops, I am sure it is being implemented and conferring major benefits, especially in small organizations with little IT management discipline. I am sure the supporters of devops have some positive goals in mind too. What’s more, it is addressing a very real problem – ops really should spend more time on better processes and controls than in “daily deployment muck“.

However, devops should be a two-way street. As a former op, I know that the apps team have to pull their weight too, by addressing gaps like:

  • Including ops during the design process, so applications are built to work with standard ops tools
  • Taking ops input on deployment, so applications will go in cleanly without disrupting other users
  • Working with ops on capacity and scalability requirements, so they can keep supporting it when it grows
  • Implementing ops” critical needs for logging, isolation, identity management, configuration needs, and secure interfaces so the app can be secure and compliant
  • Giving ops some advance insight into applications, especially during test and QA, so they can start to prepare for them before they come over the wall
  • Allowing ops to contribute to better application design, deployment, and management; that ops can do more for the release cycle and ongoing management than just “manipulating APIs
“Ops do enable business – and agile business at that.”

See, ops do enable business – and agile business at that – by ensuring that new applications coming into an existing complex environment are safe, secure, reliable, integrated, and responsive, regardless of how complex IT is, or how many moving parts there are. Devops seems to miss this important detail.

So I am sceptical of how devops will work in large, well-run IT environments with important and necessary operational controls, especially the > 60% of organizations that are committed to ITIL best practices (like formal and integrated management of change, configuration, release, assets, etc.).

After all, ‘agile’ does not magically obviate the need to identify and prevent bad changes, to reject apps that breach operational compliance, to ensure each new application adheres to standards, or to prevent uncontrolled sprawl of heterogeneous software.

I still have a lot to think about on this topic, and am trying to keep an open mind. But my best guess right now is that, for enterprises at least, devops either will not take hold or will not last. It seems most likely to be instead, at best, a transitory state on the path to a ‘new normal’. As with all “revolutions”, it has started outside IT ops, yet I expect will eventually co-opt and migrate wholly to operations in some form. Once the revolutionaries in development understand how many business needs besides agility actually require routine, process, management, and controls, they will back away from devops the same way they backed away from ownership in other IT revolutions – like the deployment of mini computers, desktops, and web applications.

If it does turn out this way – don’t worry. Operations will again dutifully take the reins, and clean up the mess that devops will leave behind. Because that is what ops do – they manage what they are given, and keep the business running, regardless of the mess that gets thrown over the wall at them.

In any case, whether devops takes root or not, hopefully we will all learn something about cooperation, automation, agility, and control. Because all stakeholders in the devops discussion – development, operations, and business owners – could benefit from that.

Tags: , , , , , , , , , ,

20 Responses to Myopic View of DevOps Misses the Mark

  1. March 29, 2010 at 12:43

    There’s definitely some confusion about what “this DevOps thing means”. DevOps is a nice catchy terse term but it’s vague. The first times I heard it, it was used in the context of “dev and ops cooperation and hugging.” But at OpsCamp some people were interpreting it as “Ops people who do Dev to code up tools.”

    In my experience, though, the term is originating out of ops groups, however, not dev groups. I think ops people are more frustrated, if anything, with the current generally slow response from ops teams.

    I think there are two major axes of confusion regarding the term.

    One is the differentiation between the (IMO) larger concept of Agile Operations and DevOps specifically. I wrote a somewhat lengthy post on how I see the larger Agile Operations term breaking down, in a way completely parallel to agile development.

    Two is the lack of differentiation of the different roles within system administration – there’s the up front systems engineering of designing the system and then there’s the release, and then the post-release job of operational support. I think “DevOps” tends to assume all three of these items within “Ops” but it needs to be called out more – in a sufficiently large organization these are three different roles! Our DBA team here has a bunch of different sub-groups, the “production support DBAs,” the “release DBAs,” the “project DBAs,” and the “architect DBAs”…

    In closing, I don’t think that ITIL and agile are opposed at all. Sure, when people do agile, either development or operations, sloppily, they forget certain quality steps. However, real agile development, with its test-driven focus, tries to enhance and not abandon testing, performance, security, etc. Similarly, I think agile operations is about how to put in all the needed checks without requiring months to do it. And I think that’s possible. In fact, read Visible Ops (Behr, Kim, Spafford), a book popular among DevOps – they prescribe a phased ITIL implementation and have research showing that top performers that have these processes in place are actually able to accomplish a much higher throughput of changes than ones that don’t. In other words, control vs agility is a false dichotomy, based on doing control wrong.

  2. March 28, 2010 at 11:15

    Some of the semantic aspects of this that seem to be giving people pause got hashed out about a month ago in an exchange of blog posts between myself and Ernest Mueller and comments from James Turnbull, Damon Edwards, and others. My takeaway from those conversations was that, while the concepts that DevOps espouses can be applied almost universally, the people talking about them are often looking at different parts of the elephant and a more nuanced set of terms is going to be necessary for anyone using them outside the rather narrow range of actual development/operations interaction (in other words, the parts you are touching on). Ernest’s last post on the topic may be the most cogent summation: http://www.webadminblog.com/index.php/2010/03/11/defining-agile-operations-and-devops/

    Other related posts (the best bits are in the comments):
    http://agileoperations.net/index.php?/archives/35-DevOps-and-Agile-Operations.html
    http://agileoperations.net/index.php?/archives/38-Worms,-Roxanne!-Worms.html

  3. Dieter Van de Walle
    March 27, 2010 at 10:58

    From your post I believe you feel that DevOps means sacrificing structure and methodology in favor of fast deployment and change.
    The term ‘DevOps’ is still being defined as we speak, but in my view I don’t feel it should symbolise rapid, cowboy-style deployment.

    In my view, DevOps is not about fundamentally changing the way Ops works.
    I definitely agree all the ‘coal face’ realities you mention are the core business responsabilities of operations. I don’t think DevOps means glossing over these tasks.

    I feel that DevOps is about cutting away a lot of overhead by using smart automation and concepts from software development, to free up time to concentrate on the core tasks of operations.

    ["So I am sceptical of how devops will work in large, well-run IT environments with important and necessary operational controls, especially the > 60% of organizations that are committed to ITIL best practices (like formal and integrated management of change, configuration, release, assets, etc.).]

    To me, DevOps is (or definitely should be) perfectly compatible with these operational controls. It’s just about adding an extra layer of automation and control that can accelerate and simplify the processes.
    I don’t think anyone can question the benefit of well defined formal processes and controls.

    I’m still pretty ‘green’ in the operations game, so for me it’s hard to tell what exactly is new about DevOps. Maybe it isn’t so radically new?
    Some ideas for improvement in the way operations operates and a plea for more understanding between ops and devs?

    Many different views on the whole DevOps idea, I wonder what the consensus will be …

  4. March 27, 2010 at 10:42

    I agree with James Dixon, nothing about the concepts are new. And I personally I hate the name devops.

  5. March 27, 2010 at 06:46

    Andi,

    This is a most puzzling post.

    You make excellent posts about how operations needs to improve. You make excellent posts about how development needs to improve. You link to clear and rational explanations about what DevOps is all about (and people, including myself, who all agree that DevOps is about fundamentally about improving how business, dev, and ops work together).

    But then you go an make a claim that DevOps is all about “Dev bypassing Ops” or “Dev replacing Ops”… but you don’t cite where you got this from. You also claim that the people who are discussing DevOps are mostly all Devs… but don’t back that claim up either.

    Andrew Shafer broke down your peculiar argument much more eloquently than I can, but I’d say go and look at or participate in the discussion on the devops toolchain google group (http://groups.google.com/group/devops-toolchain/). I only know who about 1/3 of the 80+ members are who have joined in the 2 week old discussion. However, I know that the majority of those came from Ops backgrounds and got tired of their jobs being full of problems and are demanding a better way of working. Of course, that “better way” is still evolving… but by knocking the whole movement as myopic or a bunch of renegade developers, you are only discouraging people from talking about their experiences and problems. It seems totally counterproductive if you really believe that cooperation and mutual understanding are at the root of DevOps. Your argument sounds a lot like the early naysayers who said agile or clouds were all a bunch of b.s. and that neither would last.

    Then to top it off you go and say that DevOps is doomed and won’t take hold… and then in the same paragraph claim that it will eventually be the “new normal”. Which is it? Is DevOps misguided and doomed or accurate and how things are destined to improve?

    If you really believe your argument that DevOps is somehow misguided or an errant movement, why don’t you come to DevOps Day US on June 25 in Mountain View, CA and make your case? I’m one of the organizers and content chairs. We welcome contrarian views and would welcome a proposal from you to make your case on how those who believe that DevOps is on to something good are wrong, myopic, or misguided. It’s the day after O’Reilly’s Velocity Conference, so it’s worth the trip. DM me on twitter if you are up to the task! :)

    DevOps Day US conference:
    http://www.devopsdays.org/

    -Damon
    http://dev2ops.org

  6. March 26, 2010 at 21:04

    There is nothing new about the concept of “devops”. It’s a catchy name to an existing mindset.

  7. March 26, 2010 at 19:44

    Interesting – actually a lot of “devops” people are Ops – I consider myself somewhere on the “devops” spectrum and have 20 years experience as an Operations persion in enterprise environments. I’ve been a mainframe/midrange Operator, an Ops Analyst, an Ops Manager, Ops Architect and a CIO/CTO.

    “DevOps” isn’t about throwing away the existing concepts, existing controls as you put it, in the Operations space. It’s about potentially introducing, adapting and integrating controls from other domains into Operations. The “why” of this should be obvious to anyone, especially those of us who have worked in large enterprises like financial services, where project-to-BAU/production is often a disaster and speed to market for new products and services is laughable. I struggle not to wince when I see phrases like “safe, secure, reliable, integrated, and responsive” describing enterprise shops. It often simply isn’t true. Operations are frequently managing a mix of 60s, 70s, 80s, 90s and beyond technology in environments where legacy is poorly understood and documented, configuration and asset management questionable, and where technology and the business rarely understand the linkages between their assets, applications and their business services.

    Something needs to be done to fix this and give the business the value they want from technology before they go down the path other businesses have and outsource technology and have us made redundant. “DevOps” is one link in the chain of ideas we should be evaluating on the path to fixing these issues.

    One of the key issues for me personally in the enterprise world is that concepts like “important and necessary operational controls” have become excuses for poor service delivery, poor availability and slow speed to market. I firmly agree concepts like change control are critical to maintaining availability but let’s be sceptical and pragmatic and re-evaluate and reassess some of these controls. One of the ways to do that is to look at other domains like development, where speed to market and quality are equally critical issues, and work out if some of the methods they have used to solve their issues also offer value to Operations.

    P.S. I’ve thrown some of my ideas around “DevOps” into a blog post that might interest you.

Leave a Reply

Your email address will not be published. Required fields are marked *

Notify me of followup comments via e-mail. You can also subscribe without commenting.