Is 'VM Stall' A Stop Sign for Virtualization?

There appears to be a challenger to ‘VM sprawl’ as the scourge of virtualization success – a problem I call ‘VM stall’.

We know about ‘VM sprawl’ – because new virtual machines are so easy to deploy, organizations can end up with more VMs that they can handle, or even use. This has the potential to cause severe problems to availability, performance, compliance, costs, security, and more.

However, I am seeing more and more evidence of this new phenomenon I think of as ‘VM stall’ – the tendency for virtualization deployments to stall once the ‘low-hanging fruit’ has been converted (typically around 20-30% of servers).

I think it happens more or less like this…

In general, organizations start virtualization deployments by converting relatively low-risk, low-impact systems – dev/test servers, Web servers, file servers, internal applications, etc. – to virtualization. With a big impact, great results, and reasonably fast and easy implementation, it is a great hit with IT and business owners. This may even spawn a ‘virtual first’ initiative, where all new server requests are deployed as virtual servers by default.

However, when faced with the next step, converting the remaining existing servers – including tier 1 business services, customer-facing environments, enterprise-wide systems, 3^rd-party applications, multi-platform services, and composite applications – virtualization projects often stall.

I was interested to see the notion of VM stall confirmed again last week (courtesy of eWeek via @JSchroed) in some new research into virtualization (PDF) coming out of Prism Microsystems, a software vendor in the SIEM market.*

One of the most interesting outcomes in this research was again the low penetration of server virtualization within each organization. As the chart below shows, most organizations have still virtualized less than a third of their production servers.

Source: Prism Microsystems, ‘2010 State of Virtualization Security Survey’, April 2010

What’s more, fully 15% have not even started to virtualize their production servers at all!

It might seem that this is really at odds with ‘the common wisdom’ that sees virtualization as mature, ubiquitous, commoditized, and even passé. We hear so much about virtualization, how it has been a top priority for years, about how everyone is deploying virtualization. For example:

• The IBM Global CIO Study 2009 in September showed 76% of 2500 global CIOs are undergoing or planning virtualization projects
• The Gartner 2010 CIO Survey in January reported that virtualization is the top priority for over 1500 global CIOs (up from number 3 the previous year).
• In January, CDW’s Server Virtualization Life Cycle Report (registration required) found that 90% of respondents have implemented server virtualization at some level.
• As far back as 2008, EMA research showed 75% of enterprises were using virtualization for production use cases
• The Prism Microsystems report the chart above comes from states that 85% of their sample have adopted virtualization to some degree

I am even starting to hear that virtualization is set to be irrelevant, becoming nothing more than just a stepping stone to cloud.

However, despite the widespread adoption of virtualization as a percentage of organizations, it is consistently still very low as a percentage of production servers.

Indeed, this is not the only recent (and not so recent) research study to highlight this issue. Over time, CIOs have reported a persistent difficulty in expanding their virtualization deployments beyond the initial 20-30% of servers. For example:

• Around 6 months ago, Gartner reported that “only 16 percent of workloads are running in virtual machines today.”
• Research from EMA has found that the average organization has only virtualized around 25% of servers (and only retired just 17%).
• The CDW Server Virtualization Life Cycle Report cited above showed that just 34% of the average organization’s total server infrastructure consists of virtualized servers
• CIO and HP survey in October 2009 reported that on average just 38% of mission-critical business services have been virtualized by companies with virtualization projects
• Forrester Research from May this year (conducted for CA) shows that the average enterprise has virtualized only around 30% of their servers.

At a time when so many organizations are experiencing VM sprawl, it seems hard to believe that VM stall is such an issue. Yet time and again we see that organizations find it difficult to ‘get over the hump’ of the initial 20-30% of servers, and difficult to move from low-risk/low-impact servers to high-risk/high-impact services.

If this were just a point-in-time observation, then VM stall might not exist. The low penetration rate may just be a point in the deployment cycle. However, VM stall appears to be a longitudinal effect, as it has been holding many deployments at around 20-30% of servers for several years. IIRC, something resembling VM stall was cited as an issue in EMA research as far back as 2008, and again in 2009. The CDW virtualization lifecycle research also reinforces the potential for long-term VM stall. In it, even organizations that self-report as “fully deployed” for server virtualization have only virtualized 37% of their servers. So while many organizations see VM stall as a short-term delay to virtualization rollout, many others are seeing VM stall as a permanent situation.

I see many possible causes for VM stall. For example:

• Risk aversion – high-risk, high-impact services have more stakeholders, more politics, larger and more distributed infrastructures, greater cost of failure and downtime, reduced or non-existent 3^rd-party support, and maximum management attention, among many other risk factors. The risk of failure may be too great, and the newest technology is always blamed for any new problems. Without new ways to address continuity, availability, performance, cost allocation, and other business requirements, conversion risk may be enough to stall virtualization deployment.
• Resourcing – with around 20-30% of servers converted, virtualization staffing starts to become a real challenge. As I talked about recently with my great mate, David Marshall, staff and skills shortages put a real throttle on virtualization deployments, especially as virtualization starts to scale. Not only is demand for virtualization skills still high, but supply continues to lag. Plus, the problem is getting worse, not better. Without the resources and skills to go forward, there is often little alternative to VM stall.
• Scalability – with one (typically small) team trying to manage a quarter of the entire server workload, staff from the virtualization project team simply cannot handle further virtualization deployment. In some cases, the virtualization technology itself does not scale well either; and in others, the management tools do not scale. Throwing more bodies at the problem is rarely the answer – after all, nine women cannot make a baby in one month. So organizations end up with VM stall almost by default, as they find that they need to fundamentally change their processes and technologies to enable further virtualization growth.
• Manageability – new IT management issues come up as the scale and risk of virtualization deployment increases. Enterprise virtualization needs new approaches to performance assurance, process automation, VM mobility, continuity planning, security and audit, software compliance, OEM support, configuration compliance, and more. The importance of manageability is greatly magnified  for high-risk/high-impact services, but few (if any) organizations seem to have the virtualization-aware management tools to scale to handle enterprise-class virtualization deployments. Again, VM stall happens almost by default, as IT tries to figure out enterprise-class manageability.

There may be more or different causes, but whatever the reasons, there is little doubt in my mind that VM stall exists. It is not universal – indeed, every study shows that a decent percentage of organizations are able to power through it – but for the majority of organizations, it appears to be very real. I have personally seen many enterprises going through it. More and more research continues to support it. For affected organizations, it is a significant problem, too, because stalled virtualization deployment means the highly desirable outcomes of virtualization – OpEx reduction, improved continuity, greater IT and business agility, energy cost reduction, ROI, etc. – either stalls as well, or even starts to backslide.

Whether VM stall represents as big a problem as VM sprawl, time will tell; but it is certainly a significant and growing challenge to the success of virtualization – and a fundamental driver for better virtualization management.

(EDIT: This article has been picked up and published on CIO.com! Join in the discussion there, or here.)

Effort vs. Payback is an Everyday Business IT Decision

I read recently a good blog post from Thomas Bittman (@tombitt) of Gartner Group, about how sometimes close enough is good enough. Talking specifically about private cloud, he talked about how an ‘imperfect’ cloud deployment – one that does not have all five essential characteristics, for example – might be enough for some organizations.

I especially appreciated how he highlighted some very specific, real-world examples to sustain his advice. As he shows, sometimes you don’t need a ’100%’ implementation, and for very good business reasons.

Not every IT organization needs a fully self-service interface, and many smaller organizations see no value in usage metering. They simply want to deliver services faster. For them, a 70% private cloud is absolutely good enough … it all comes down to business requirements, return on investment, and future strategy. How far you go is your decision.

via Driving for Imperfection With Your Private Cloud.

If you haven’t seen it yet, you should. It’s a quick read, only 4 paragraphs and less than 300 words. Go ahead. I’ll still be here when you get back.

The theme is very similar to something I wrote in a research report for EMA, ‘The Responsible Cloud‘, also on cloud computing. Regarding the NIST definition of cloud, I cautioned against dogmatic interpretations of cloud computing, and the notion that a ‘real’ cloud must necessarily have all of the essential characteristics, or fit some specific deployment model. Flexibility is key, I advised, and delivering on key business requirements is more important than definitions.

Two other things happened this week that made me think about this in different ways:

• An internal session at CA reviewing some customer-facing materials. All attendees agreed – we can’t preach unattainable dogma; we need to deal with specific requirements and partial deployments, as well as broad requirements that come from  ’100%’ implementations.
• A group discussion on LinkedIn, where an IT practitioner wanted advice on building a small private cloud. He was soon inundated with an unrealistic list of requirements, from hypervisor features to management disciplines, that he *must* have to build a ’100%’ cloud.

The similar inferences in three otherwise unrelated conversations started me thinking more broadly about ’100% adoption’. It IT, as in life, you never really need a Rolls Royce. You can aspire to the quality, appreciate its refinement, and in some cases you may be fortunate enough to actually enjoy it, but there is a point where it simply doesn’t make sense to pursue that level of luxury. Mostly you can get away with a Ford. Sometimes you can even make do with a second-hand Lada.

The same Pareto-like principle applies roughly throughout IT (much to the annoyance of just about every security pro I have ever met) – although the actual ratio may vary wildly, you can often get most of the benefit from less than a ’100%’ implementation.

The phrase that sprang to mind for me was the same conclusion that I published elsewhere in the Responsible Cloud report, and the same notion that many IT pros live by, day in and day out:

It is important to look for opportunities, and do what makes sense

This should not just apply to cloud computing, but across all of IT.

Take, as another example, adherence to the IT Infrastructure Library (ITIL). Now, ITIL is a great framework, and an increasingly definitive reference for best practices in IT management. Data I have seen suggests as many as 60% of all IT organizations are committed to ITIL, and that implementation of ITIL (whatever that actually means) results in measurable and specific benefits in IT costs, staff and server efficiency, operational maturity, and more.

However, I also hear and read somewhat justified rants about how “ITIL just doesn’t work … ITIL is more 1960s than 2010 … it’s useless.” Yet the truth is, as so often, somewhere in the middle. In this too enterprises can definitely benefit from avoiding the dogmatic application of every single prescription. The same is true for other standards such as COBIT and ISO, or prescriptions from standards groups like the DMTF or NIST. All can deliver significant benefits with less than a 100% implementation.

It also applies in internal adoption of standard operating environment (SOE) components, like making singular (and often binding) choices between, for example:

• VMware vs. Hyper-V vs. Xen
• HP vs. Cisco vs. IBM
• HDS vs. NetApp vs. EMC
• Windows vs. Linux vs. UNIX
• iPhone vs. WinMo vs. Blackberry
• Solution suites vs. point products
• Mainframe vs. Commodity
• Physical vs. virtual vs. cloud

In all these cases and more, although standardization can have specific benefits, the greatest benefit to the enterprise does not always accrue from making an exclusionary choice; from committing to a 100% implementation. Most IT practitioners know that heterogeneity is the new standard – whether intuitively or grudgingly. They know that sometimes the best – or at least necessary – outcomes arise from providing multiple choices, fit to support multiple use cases.

Of course some areas are less flexible. You cannot, for example, pick and choose which parts of PCI, HIPAA, or Sarbanes-Oxley compliance would work best for you. Perhaps ‘close’ only matters in horseshoes and hand grenades, but for sure it doesn’t matter in legal compliance.

However, where possible, IT – practitioners, consultants, vendors, and analysts – need to stay away from dogma. We must avoid making any architecture, maturity model, or industry standard a religious ‘all or none’ battle. Important though they may be, these are not religious battles. These are IT decisions. Moreover, these are business decisions. So we need to keep the business goals in mind, and realize that sometimes a ’100%’ implementation simply does not make sense.

For those who have asked, and others who may care to know – the rumours are indeed true. After many happy years leading the fantastic systems and storage management team at one of the very best IT industry analyst and consulting firms, Enterprise Management Associates (EMA), I have moved on to take up an exciting new opportunity.

As of Wednesday this week (2/24), I am now at one of the very best IT management software vendors, CA Inc., where I am leading product marketing for their — our — virtualization management solutions.

In many ways, this was an incredibly difficult decision. EMA is a truly excellent place to work, and the role of an industry analyst was fascinating and fulfilling. The people I worked with and for are some of the best minds in IT – always intellectually stimulating, and straight-out fun to be with. It was truly my privilege to get to know them all, and especially to help my clients and my team to be successful.

Yet this was also one of the easiest decisions I have made. I believe both virtualization and management deliver incredible IT and business benefits, and as virtualization becomes increasingly ubiquitous, management of virtual systems becomes increasingly critical. I have long considered CA a leader in physical and virtual systems management, and believe CA has a great opportunity to extend its leadership in virtualization management, by helping even more IT and business people to be even more successful. As a part of  CA now, I can not only be a part of that opportunity, but can be a significant author of that success.

Moreover, it allows me to indulge my passion for technology and my expertise in marketing in an in-depth, direct, and focused way, rather than the broad, ancillary, and essentially academic role of an industry analyst. I will be able to work directly with some the biggest and most successful companies and technologies, not just in the US, but around the globe. Plus, like EMA, CA also has some incredible minds who are some of the most fun people to hang out with too.

While some will see this a move (back) to ‘the dark side’, I have always considered analysts and vendors to be two sides of the same coin – helping IT to deliver business services in more effective and efficient ways. While some may say that I have ‘sold out’ my integrity as an analyst, I have always considered my integrity to be a core and consistent value — and a non-negotiable one — regardless of my employer. While some may think I can no longer champion the best interests of enterprise IT like I did while I was an analyst, I believe the best software companies, and their best people, succeed and thrive specifically because they do exactly that.

As for this blog (and my Twitter feed), all my reasons for blogging and tweeting, and what I hope to achieve (both personally and professionally) with social media, are still the same as they were when I started. I therefore intend to continue writing and posting my personal opinions and insights about technology and other areas that interest me. After all, the areas I work with haven’t really changed, so I am still going to post about virtualization, systems management, data center operations, and cloud computing.

So although I cannot help but be informed by my current position and experience, my goal is to keep posting interesting and informed ideas, regardless of my employer. No doubt some people will stop reading — which is fine — but I still hope you will keep inspiring, contributing to, reading, commenting on, and arguing about these part-time musings of a full-time technologist.

A few days ago I was pleased to brief again with Reflex Systems. Apart from the fact that they are doing some very cool things with virtualization management, their approach struck me as, if not unique, at least pleasantly rare.

Good for them!

What I liked most was that they are trying to break down the barriers between systems and security management. Certainly this is something that I discuss regularly with enterprises – the need to stop focusing on silo-based management, and perhaps even more importantly, to stop pandering to silo-focused low-level managers. Almost all of the CIOs, VPs, and IT Directors who I talk with are critically aware of the problems these silos cause – including human errors, resource inefficiencies, security problems, and higher costs.

This is also a constant discussion I have within EMA, especially with the lead of our security practice, Scott Crawford – a brilliant mind on security (amongst many other subjects) who constantly thinks about security in ways I never could or would. We work and publish together on this topic frequently. Indeed, it has come up again in our latest research, which shows that security and risk management are a fundamental requirement for cloud computing – or what EMA calls the ‘Responsible Cloud’.

The upshot of all these conversations is simple – security management and systems management are not, cannot, and should not be completely separate. Not in human terms, not in processes, and not in technologies. Without doubt, anyone in a large enterprise who has ever tried to implement a patch, a configuration change, a firewall update, a software release, or a hundred other data center changes will attest to this in a heartbeat.

Of course (as Scott rightly pointed out when I last spoke with him about this), we will always need security experts, and systems experts – the two disciplines are not the same, and we will always need deep domain expertise in each. So I am not advocating complete convergence. But we need more software tools that provide integration and interoperability that allow these professional to work more effectively together.

While multi-function vendors like CA, Symantec, IBM, and others have the product portfolio to approach these cross-silo problems holistically, there are few ‘best of breed’ vendors thinking this way. Of course, Tripwire and the ever-inspiring Gene Kim (who I have sadly never met) spring to mind for me; so would Configuresoft (although now as part of EMC Ionix, hardly a niche vendor), and the indefatigable Dennis Moreau. Both inspire their teams, technologies, and customers by championing a fundamental understanding that systems and security cannot, at their heart, be completely separated.

(As an aside, these two seem like they would have been incredibly compelling arch-enemies in some ubergeek superhero genre – although I would never want to choose which should be the hero and which the villain!)

I must say that, so far at least, I don’t know the product design team from Reflex personally – guys like Hezi Moore, Aaron Bawcom, and Mike Wronski – as well as I do Dennis or Gene. However, I do know that they all have very credible security chops. Plus, one thing is clear to me.

They get it. They really get it.

And that in itself is a thing of rare beauty.

Deliberately designing functionality that addresses both security and systems management – like functional isolation, integrated access control, change segregation, granular audit trails, policy based management, and role-based access – into a systems management toolset is a rare feat, especially in startup and niche products. It is something I look for all the time, because my enterprise clients often demand it. Sadly, all too often I fail to find it – and I am not even a real security wonk! When I do, I am pleasantly surprised. When I see deep thought going into the security value of a systems management product, I am almost ecstatic.

Unfortunately, the challenge for vendors like Reflex and Tripwire (as it was for Configuresoft, and perhaps is still for EMC, Symantec, etc.) is to find customers that value this synergy. While most high-level IT execs understand this imperative, their holistic view frequently does not translate to many of their lower-level managers, or to many functional IT practitioners.

Of course, there are plenty of departmental ops managers and security managers who do get it. They strive to connect their teams with other groups, driving greater business efficiency and effectiveness as a result. However, unfortunately, many do not, instead focusing on protecting their small empires, walling themselves off from integrated management and cross-functional resourcing.

Similarly, many positive-minded individual technicians will actively seek out cross-skilling opportunities, recognizing that it makes them not just more useful but also more valuable, and more indispensable. However, many practitioners (both security and ops) can be just as bad as the most myopic managers (who they often work for), dogmatically eschewing integrated management tools and processes, seeing them as threats to their own personal domains of control.

Sad but true, best practices like breaking down IT management silos are not always adopted.

Fortunately, vendors like Reflex and Tripwire that have expertise and passion in both ops and security (and – shameless plug – trusted advisors like EMA, which is big enough to have experts in both disciplines, yet small enough that we still work together), are trying to break down these barriers.

And more power to them. They serve their clients much better by promoting the undeniable facts that security values are critical to systems management, and systems management is critical to security.