Most devops discussions are missing the target

I am hearing a lot about the rise of a concept called ‘devops’ – a mashup of ‘development’ and ‘operations’. I am not at all an expert in this area, but from what I can tell, devops is aimed at streamlining rapidly iterative application delivery to allow for greater development and business agility. Devops aims to achieve this by breaking down the barriers – human, process, and technology – between application development and system operations.

Interestingly, the concept is new enough that, as I write this, there is not even an entry for it in Wikipedia yet. I did find a blog by Damon Edwards (on Twitter – @damonedwards) very useful though, as he explains the age-old disconnects between application developers ‘throwing software over the wall’, and ops who are painfully resistant to change. James Urquhart (@jamesurquhart ) blogged very recently on the concept too , and again provided some very helpful content. Conversing online with them and others also helped me to formulate some more concrete ideas about devops – or at least some more concrete questions.

My interest was especially piqued when I understood how closely devops is connected to virtualization, cloud, and automation – my core interests:

• Cloud – Devops has antecedents in ‘rogue’ developers (or developers from smaller shops) using cloud resources (IaaS, PaaS) for new projects, and will benefit greatly from cloud-based development and deployment, as cloud providers do not impose the restrictions of internal change-averse ops teams, and developers can essentially manage their own ops requirements instead.
• Virtualization – In-house devops (which needs more heavy lifting) is greatly assisted by virtualization, as virtual machines become the new base unit for application packaging, avoiding application rollout failures caused by incompatibility between the test and production environments (hardware, OS, middleware, etc.).
• Automation – In-house devops is also greatly facilitated by automation, which can use standard workflows to automatically provision and configure these complete application VMs, as well as backup and restore VMs, allowing complex composite application deployment and rollback at the click of a mouse.

Clearly devops has many very attractive outcomes – drive agile business, reduce delays, smooth application releases, deliver value faster. It is a very seductive idea. Who wouldn’t want it?

However, most of the writings I see about devops are really about dev, not ops. As a result, they don’t really capture the whole story of the application lifecycle. They justify devops as an antidote to the problems that ops are causing – slowing down release cycles, imposing arbitrary rules, screwing up deployments, killing developer productivity, hacking manual scripts and configs, stopping the business from being agile – but fail to recognize both the failings of developers that contribute to the problems, and the role of operations in delivering critical business outcomes during the application delivery lifecycle.

On the contrary, discussions mainly focus on how developers can sideline or change operations, positioning devops as the lone hero in the battle against inefficiency, as application developers fix all the problems (!) by controlling or automating key release management operations like provisioning, deployment, integration, patching, and software update. Meanwhile, ops are marginalised, along with their timesinks and roadblocks, satisfying the needs of an agile and rapidly changing business.

See – seductive, isn’t it?

Yet this seems to me (as a former op) fundamentally flawed, a development-centric neologism based on an incomplete understanding of the real purpose and role of IT operations, or of operations’ history in the development of ‘agile’ IT.

The way I see it, devops misses that target on how IT ops serve business needs too, and seems to gloss over ‘coal face’ realities like:

• Who handles ongoing support, especially software update for the unrestrained sprawl of non-standard systems and components.
• Who ensures each new application doesn’t interfere with existing and especially legacy systems (and networks, storage, etc.)?
• Who handles integration with common production systems that cannot be encapsulated in a VM, like storage arrays (NAS, SAN), networking fabrics, facilities, etc.
• Who handles impact analysis, change control and rollback planning to ensure deployment risk is understood and mitigated?
• Who is responsible for cost containment and asset rationalization, when devops keeps rolling out new systems and applications?
• Who ensures reporting, compliance, data updates, log maintenance, Db administration, etc. are built into the applications, and integrated with standard management tools?
• Who will assure functional isolation, role-based access controls, change auditing, event management, and configuration control to secure applications, data, and compliance?

Because, if you have ever worked with both ops and apps, you know it is not going to be apps.

Now, in defence of devops, I am sure it is being implemented and conferring major benefits, especially in small organizations with little IT management discipline. I am sure the supporters of devops have some positive goals in mind too. What’s more, it is addressing a very real problem – ops really should spend more time on better processes and controls than in ‘daily deployment muck’.

However, devops should be a two-way street. As a former op, I know that the apps team have to pull their weight too, by addressing gaps like:

• Including ops during the design process, so applications are built to work with standard ops tools
• Taking ops input on deployment, so applications will go in cleanly without disrupting other users
• Working with ops on capacity and scalability requirements, so they can keep supporting it when it grows
• Implementing ops’ critical needs for logging, isolation, identity management, configuration needs, and secure interfaces so the app can be secure and compliant
• Giving ops some advance insight into applications, especially during test and QA, so they can start to prepare for them before they come over the wall
• Allowing ops to contribute to better application design, deployment, and management; that ops can do more for the release cycle and ongoing management than just ‘manipulating APIs’

See, ops do enable business – and agile business at that – by ensuring that new applications coming into an existing complex environment are safe, secure, reliable, integrated, and responsive, regardless of how complex IT is, or how many moving parts there are. Devops seems to miss this important detail.

So I am sceptical of how devops will work in large, well-run IT environments with important and necessary operational controls, especially the > 60% of organizations that are committed to ITIL best practices (like formal and integrated management of change, configuration, release, assets, etc.).

After all, ‘agile’ does not magically obviate the need to identify and prevent bad changes, to reject apps that breach operational compliance, to ensure each new application adheres to standards, or to prevent uncontrolled sprawl of heterogeneous software.

I still have a lot to think about on this topic, and am trying to keep an open mind. But my best guess right now is that, for enterprises at least, devops either will not take hold or will not last. It seems most likely to be instead, at best, a transitory state on the path to a ‘new normal’. As with all ‘revolutions’, it has started outside IT ops, yet I expect will eventually co-opt and migrate wholly to operations in some form. Once the revolutionaries in development understand how many business needs besides agility actually require  routine, process, management, and controls, they will back away from devops the same way they backed away from ownership in other IT revolutions – like the deployment of mini computers, desktops, and web applications.

If it does turn out this way – don’t worry. Operations will again dutifully take the reins, and clean up the mess that devops will leave behind. Because that is what ops do – they manage what they are given, and keep the business running, regardless of the mess that gets thrown over the wall at them.

In any case, whether devops takes root or not, hopefully we will all learn something about cooperation, automation, agility, and control. Because all stakeholders in the devops discussion – development, operations, and business owners – could benefit from that.

Effort vs. Payback is an Everyday Business IT Decision

I read recently a good blog post from Thomas Bittman (@tombitt) of Gartner Group, about how sometimes close enough is good enough. Talking specifically about private cloud, he talked about how an ‘imperfect’ cloud deployment – one that does not have all five essential characteristics, for example – might be enough for some organizations.

I especially appreciated how he highlighted some very specific, real-world examples to sustain his advice. As he shows, sometimes you don’t need a ’100%’ implementation, and for very good business reasons.

Not every IT organization needs a fully self-service interface, and many smaller organizations see no value in usage metering. They simply want to deliver services faster. For them, a 70% private cloud is absolutely good enough … it all comes down to business requirements, return on investment, and future strategy. How far you go is your decision.

via Driving for Imperfection With Your Private Cloud.

If you haven’t seen it yet, you should. It’s a quick read, only 4 paragraphs and less than 300 words. Go ahead. I’ll still be here when you get back.

The theme is very similar to something I wrote in a research report for EMA, ‘The Responsible Cloud‘, also on cloud computing. Regarding the NIST definition of cloud, I cautioned against dogmatic interpretations of cloud computing, and the notion that a ‘real’ cloud must necessarily have all of the essential characteristics, or fit some specific deployment model. Flexibility is key, I advised, and delivering on key business requirements is more important than definitions.

Two other things happened this week that made me think about this in different ways:

• An internal session at CA reviewing some customer-facing materials. All attendees agreed – we can’t preach unattainable dogma; we need to deal with specific requirements and partial deployments, as well as broad requirements that come from  ’100%’ implementations.
• A group discussion on LinkedIn, where an IT practitioner wanted advice on building a small private cloud. He was soon inundated with an unrealistic list of requirements, from hypervisor features to management disciplines, that he *must* have to build a ’100%’ cloud.

The similar inferences in three otherwise unrelated conversations started me thinking more broadly about ’100% adoption’. It IT, as in life, you never really need a Rolls Royce. You can aspire to the quality, appreciate its refinement, and in some cases you may be fortunate enough to actually enjoy it, but there is a point where it simply doesn’t make sense to pursue that level of luxury. Mostly you can get away with a Ford. Sometimes you can even make do with a second-hand Lada.

The same Pareto-like principle applies roughly throughout IT (much to the annoyance of just about every security pro I have ever met) – although the actual ratio may vary wildly, you can often get most of the benefit from less than a ’100%’ implementation.

The phrase that sprang to mind for me was the same conclusion that I published elsewhere in the Responsible Cloud report, and the same notion that many IT pros live by, day in and day out:

It is important to look for opportunities, and do what makes sense

This should not just apply to cloud computing, but across all of IT.

Take, as another example, adherence to the IT Infrastructure Library (ITIL). Now, ITIL is a great framework, and an increasingly definitive reference for best practices in IT management. Data I have seen suggests as many as 60% of all IT organizations are committed to ITIL, and that implementation of ITIL (whatever that actually means) results in measurable and specific benefits in IT costs, staff and server efficiency, operational maturity, and more.

However, I also hear and read somewhat justified rants about how “ITIL just doesn’t work … ITIL is more 1960s than 2010 … it’s useless.” Yet the truth is, as so often, somewhere in the middle. In this too enterprises can definitely benefit from avoiding the dogmatic application of every single prescription. The same is true for other standards such as COBIT and ISO, or prescriptions from standards groups like the DMTF or NIST. All can deliver significant benefits with less than a 100% implementation.

It also applies in internal adoption of standard operating environment (SOE) components, like making singular (and often binding) choices between, for example:

• VMware vs. Hyper-V vs. Xen
• HP vs. Cisco vs. IBM
• HDS vs. NetApp vs. EMC
• Windows vs. Linux vs. UNIX
• iPhone vs. WinMo vs. Blackberry
• Solution suites vs. point products
• Mainframe vs. Commodity
• Physical vs. virtual vs. cloud

In all these cases and more, although standardization can have specific benefits, the greatest benefit to the enterprise does not always accrue from making an exclusionary choice; from committing to a 100% implementation. Most IT practitioners know that heterogeneity is the new standard – whether intuitively or grudgingly. They know that sometimes the best – or at least necessary – outcomes arise from providing multiple choices, fit to support multiple use cases.

Of course some areas are less flexible. You cannot, for example, pick and choose which parts of PCI, HIPAA, or Sarbanes-Oxley compliance would work best for you. Perhaps ‘close’ only matters in horseshoes and hand grenades, but for sure it doesn’t matter in legal compliance.

However, where possible, IT – practitioners, consultants, vendors, and analysts – need to stay away from dogma. We must avoid making any architecture, maturity model, or industry standard a religious ‘all or none’ battle. Important though they may be, these are not religious battles. These are IT decisions. Moreover, these are business decisions. So we need to keep the business goals in mind, and realize that sometimes a ’100%’ implementation simply does not make sense.