Is 'VM Stall' A Stop Sign for Virtualization?

There appears to be a challenger to ‘VM sprawl’ as the scourge of virtualization success – a problem I call ‘VM stall’.

We know about ‘VM sprawl’ – because new virtual machines are so easy to deploy, organizations can end up with more VMs that they can handle, or even use. This has the potential to cause severe problems to availability, performance, compliance, costs, security, and more.

However, I am seeing more and more evidence of this new phenomenon I think of as ‘VM stall’ – the tendency for virtualization deployments to stall once the ‘low-hanging fruit’ has been converted (typically around 20-30% of servers).

I think it happens more or less like this…

In general, organizations start virtualization deployments by converting relatively low-risk, low-impact systems – dev/test servers, Web servers, file servers, internal applications, etc. – to virtualization. With a big impact, great results, and reasonably fast and easy implementation, it is a great hit with IT and business owners. This may even spawn a ‘virtual first’ initiative, where all new server requests are deployed as virtual servers by default.

However, when faced with the next step, converting the remaining existing servers – including tier 1 business services, customer-facing environments, enterprise-wide systems, 3^rd-party applications, multi-platform services, and composite applications – virtualization projects often stall.

I was interested to see the notion of VM stall confirmed again last week (courtesy of eWeek via @JSchroed) in some new research into virtualization (PDF) coming out of Prism Microsystems, a software vendor in the SIEM market.*

One of the most interesting outcomes in this research was again the low penetration of server virtualization within each organization. As the chart below shows, most organizations have still virtualized less than a third of their production servers.

Source: Prism Microsystems, ‘2010 State of Virtualization Security Survey’, April 2010

What’s more, fully 15% have not even started to virtualize their production servers at all!

It might seem that this is really at odds with ‘the common wisdom’ that sees virtualization as mature, ubiquitous, commoditized, and even passé. We hear so much about virtualization, how it has been a top priority for years, about how everyone is deploying virtualization. For example:

• The IBM Global CIO Study 2009 in September showed 76% of 2500 global CIOs are undergoing or planning virtualization projects
• The Gartner 2010 CIO Survey in January reported that virtualization is the top priority for over 1500 global CIOs (up from number 3 the previous year).
• In January, CDW’s Server Virtualization Life Cycle Report (registration required) found that 90% of respondents have implemented server virtualization at some level.
• As far back as 2008, EMA research showed 75% of enterprises were using virtualization for production use cases
• The Prism Microsystems report the chart above comes from states that 85% of their sample have adopted virtualization to some degree

I am even starting to hear that virtualization is set to be irrelevant, becoming nothing more than just a stepping stone to cloud.

However, despite the widespread adoption of virtualization as a percentage of organizations, it is consistently still very low as a percentage of production servers.

Indeed, this is not the only recent (and not so recent) research study to highlight this issue. Over time, CIOs have reported a persistent difficulty in expanding their virtualization deployments beyond the initial 20-30% of servers. For example:

• Around 6 months ago, Gartner reported that “only 16 percent of workloads are running in virtual machines today.”
• Research from EMA has found that the average organization has only virtualized around 25% of servers (and only retired just 17%).
• The CDW Server Virtualization Life Cycle Report cited above showed that just 34% of the average organization’s total server infrastructure consists of virtualized servers
• CIO and HP survey in October 2009 reported that on average just 38% of mission-critical business services have been virtualized by companies with virtualization projects
• Forrester Research from May this year (conducted for CA) shows that the average enterprise has virtualized only around 30% of their servers.

At a time when so many organizations are experiencing VM sprawl, it seems hard to believe that VM stall is such an issue. Yet time and again we see that organizations find it difficult to ‘get over the hump’ of the initial 20-30% of servers, and difficult to move from low-risk/low-impact servers to high-risk/high-impact services.

If this were just a point-in-time observation, then VM stall might not exist. The low penetration rate may just be a point in the deployment cycle. However, VM stall appears to be a longitudinal effect, as it has been holding many deployments at around 20-30% of servers for several years. IIRC, something resembling VM stall was cited as an issue in EMA research as far back as 2008, and again in 2009. The CDW virtualization lifecycle research also reinforces the potential for long-term VM stall. In it, even organizations that self-report as “fully deployed” for server virtualization have only virtualized 37% of their servers. So while many organizations see VM stall as a short-term delay to virtualization rollout, many others are seeing VM stall as a permanent situation.

I see many possible causes for VM stall. For example:

• Risk aversion – high-risk, high-impact services have more stakeholders, more politics, larger and more distributed infrastructures, greater cost of failure and downtime, reduced or non-existent 3^rd-party support, and maximum management attention, among many other risk factors. The risk of failure may be too great, and the newest technology is always blamed for any new problems. Without new ways to address continuity, availability, performance, cost allocation, and other business requirements, conversion risk may be enough to stall virtualization deployment.
• Resourcing – with around 20-30% of servers converted, virtualization staffing starts to become a real challenge. As I talked about recently with my great mate, David Marshall, staff and skills shortages put a real throttle on virtualization deployments, especially as virtualization starts to scale. Not only is demand for virtualization skills still high, but supply continues to lag. Plus, the problem is getting worse, not better. Without the resources and skills to go forward, there is often little alternative to VM stall.
• Scalability – with one (typically small) team trying to manage a quarter of the entire server workload, staff from the virtualization project team simply cannot handle further virtualization deployment. In some cases, the virtualization technology itself does not scale well either; and in others, the management tools do not scale. Throwing more bodies at the problem is rarely the answer – after all, nine women cannot make a baby in one month. So organizations end up with VM stall almost by default, as they find that they need to fundamentally change their processes and technologies to enable further virtualization growth.
• Manageability – new IT management issues come up as the scale and risk of virtualization deployment increases. Enterprise virtualization needs new approaches to performance assurance, process automation, VM mobility, continuity planning, security and audit, software compliance, OEM support, configuration compliance, and more. The importance of manageability is greatly magnified  for high-risk/high-impact services, but few (if any) organizations seem to have the virtualization-aware management tools to scale to handle enterprise-class virtualization deployments. Again, VM stall happens almost by default, as IT tries to figure out enterprise-class manageability.

There may be more or different causes, but whatever the reasons, there is little doubt in my mind that VM stall exists. It is not universal – indeed, every study shows that a decent percentage of organizations are able to power through it – but for the majority of organizations, it appears to be very real. I have personally seen many enterprises going through it. More and more research continues to support it. For affected organizations, it is a significant problem, too, because stalled virtualization deployment means the highly desirable outcomes of virtualization – OpEx reduction, improved continuity, greater IT and business agility, energy cost reduction, ROI, etc. – either stalls as well, or even starts to backslide.

Whether VM stall represents as big a problem as VM sprawl, time will tell; but it is certainly a significant and growing challenge to the success of virtualization – and a fundamental driver for better virtualization management.

(EDIT: This article has been picked up and published on CIO.com! Join in the discussion there, or here.)

The only significant and unique benefit of KVM for server virtualization (as noted by Sander van Vugt in our (virtual) debate on Xen vs.KVM Linux Virtualization Hypervisors) is that KVM is part of the Linux kernel. This ensures broad standardization, patch compatibility, simpler upgrades, and a low-impact on-ramp for existing Linux IT shops.

Yet this is a solution for a problem that does not really exist.

Large enterprises already run thousands of components, from services/daemons to drivers to applications, all as additions to various kernels. Maintaining one more (or even several more) non-kernel components like Hyper-V, XenServer, ESX, etc., is not a net negative. On the contrary, EMA data shows that virtualization actually improves the productivity of server administrators, and by an average of around 10% – up to 20% or more for best performers. For competent administrators with good lifecycle management tools, the time they spend to learn, test, and maintain hypervisors is a significant effort, but it is time paid back with interest.

On the other hand, many downsides to KVM are all too apparent.

It is easy to point to the lack of technology features and maturity in KVM – areas like live migration, paravirtualization, networking, isolation, performance, security, or a host of other  features which KVM (in some cases arguably) lacks. I have only some doubt that KVM will meet these low-level functional requirements eventually, but it will not be anytime soon. Yet they are essentially table stakes in server virtualization today.

The inherent dependency on Linux would also require a major shift in  platforms for the average datacenter (where Windows outnumbers Linux by  150:1), and a major investment in resourcing, training, and software. This is hardly an attractive proposition for a data center manager. Still, existing Linux staff will be able to pick it up, and could even have some success on their (relatively few) existing Linux platforms.

However, even if these weaknesses are overcome, KVM has a much more strategic problem – the gaping void in the KVM management ecosystem. There is almost no third-party support for KVM from management vendors. Even stated support from key partner vendors like IBM, HP, and of course Red Hat is basic at best. What’s more, EMA data suggests KVM will not foster a significant management ecosystem in the future, either.

EMA’s research on Virtual System Management showed convincingly how important management is to virtualization. Across 18 different management disciplines, almost all correlated with measurably better outcomes in metrics like MTTR, provisioning time, availability, VM density, migration speed, and more.

EMA’s new cloud research shows a similar importance. Applying mature automation and management disciplines to virtual systems is directly correlated with positive cloud outcomes like reduced CapEx, reduced OpEx, improved operational maturity and more.

Not surprising then, that over 80% of enterprises consider manageability an important or very important factor in their virtualization and cloud technology decisions.

Unfortunately, KVM ranks anywhere from 4^th to 10^th in enterprise preferences for virtualization and cloud technology providers. It comes behind first ESX, then Hyper-V or Xen (multiple implementations), often various UNIX hypervisors (PowerVM, Integrity VMs or vPars, Solaris Containers), and even z/VM. No enterprise demand means that management vendors have little incentive to support KVM.

In fact, in my conversations with management software vendors, most generally put KVM around 5th in line for support – which, realistically, means it is not even on the current roadmap. What’s more, for better or worse several of them have a vested interest in not supporting KVM (no points for guessing who).

This means KVM has little or no prospect of gaining third-party support for virtualization management tools like VM-aware backup and restore, VM provisioning, virtual resource management, VM configuration auditing, virtual performance monitoring, VM lab management, VM image control, storage management,network automation and more. The same holds true for integration with higher-level virtual systems management tools for virtual and physical data center automation and service management disciplines.

For any IT group, sophisticated management tools deliver many proven benefits. For larger enterprises especially, they are simply not optional.  Without even the prospect of a robust management ecosystem, KVM is simply a non-starter in most large-scale deployments. For my enterprise clients at least, it is certainly not a credible choice for x86 server virtualization.

Microsoft and Opalis

Today Microsoft Corporation (NASD:MSFT) announced a definitive agreement to acquire Opalis Inc., the leading independent vendor of IT Process Automation (ITPA) software.

IT Process Automation (ITPA) is a Data Center Automation (DCA) discipline that EMA defines as “the ability to automate and integrate the workflow of complex, multi-discipline IT management processes.” This automation can replace many manual, resource-intensive, and error-prone activities that typically cross multiple IT components, disciplines, and/or departments. ITPA delivers exceptional results including freeing up 77% more staff for strategic projects, providing more than 60 additional hours of system availability per year, and saving an average $500,000 more per year on staff costs than other Data Center Automation (DCA) disciplines.

This space has been gaining interest, both expanding and consolidating, for some time, as evidenced by significant development and acquisition activity from Novell (ZENworks, PlateSpin), HP (Opsware, iConclude), BMC Software (RealOps, Atrium), NetIQ (Aegis), Symantec (T-Logic, Altiris), and CA (Optinuity, Spectrum).

I think this is an excellent move by Microsoft. It will certainly make customers of both companies very happy. Microsoft and its customers gain an exceptional solution, in a discipline area that Microsoft was clearly lacking, and one which delivers many proven and exceptional benefits. For Opalis customers, it is probably a mixed bag. It will be a major change, but with Microsoft’s strength and stability, it is likely to be a positive outcome overall for Opalis customers.

This is, however, a huge blow for competitors, especially for the few large management vendors that have not yet acquired or built an ITPA solution or components, or whose own ITPA capabilities are less than stellar. For other large mgmt vendors with credible or better ITPA capabilities, this is both an opportunity and a threat. For mid-sized vendors that compete with Opalis or Microsoft Systems Center, and especially smaller vendors, this is a horrible result. Overall, most vendors will have to hustle to respond, although many will be unable to do so.

Meanwhile, Microsoft, Opalis, and their customers should be ecstatic with this deal. Few acquisitions are so clearly positive for the stakeholders as this.

You should be able to check out what the executives from both companies have to say in their blog posts:

• Blog post from Brad Anderson, Microsoft Corporate Vice President: http://blogs.technet.com/systemcenter/default.aspx
• Blog post from Todd DeLaughter, President & CEO of Opalis Software: http://www.opalis.com/blog.asp?id=1

Meanwhile, I will be expanding on the impact of this acquisition very soon with a full EMA Impact Brief. Keep your eyes out for that one – lots of significant implications for customer and competitors, without doubt!

The survey revealed some very interesting data, with a very well thought out instrument and a quality sample – 300 respondents (100 each from the US, EMEA, and Asia Pacific) with at least 500 employees in the US (250 in the UK, France, Germany, Australia, Singapore and India), and all with a current or planned investment in server virtualization.

A number of data points stand out for me:

• The balance of Test/Dev implementations vs. Production continues to reflect EMA data. While production is still lagging behind test and dev as a use case, virtualization for mission-critical production is only slightly behind test and dev overall – and within the margin of error in most cases. This is good news, as enterprises clearly continue to grow real, production use cases.  It was interesting to see the differences between US and EMEA/APAC on this data point too, something EMA has not broken out in our published reports.
• Microsoft and VMware are neck and neck in enterprises’ plans for server virtualization deployments over the next 18 months. I was called crazy when my 2008 EMA research pointed to a 32% growth rate for Microsoft Hyper-V into 2009, trailing only VMware; yet here we are in 2009, and according to this new study, through 2010/11 that is going up to 49%. So who is crazy now?
• The strong growth for endpoint (desktop, application) virtualization reflects EMA data very well. It also highlights where enterprises and vendors should be heading with management technologies. It is still early days, but there are  a lot of gaps in integrated management for physical and virtual endpoints. So it is clear that this (probably even more than cloud service management) is going to be the next big problem for IT management.
• The percentage of IT services planned to be virtualized over 18 months is growing well. However, just as EMA has predicted, virtualization will remain at only around 50% of service deployments even through 2011, so there will continue to be substantial physical deployments. This reinforces my consistent (and insistent) position that effective management of virtualization must integrate both physical and virtual systems management
• Effective management continues to be elusive. EMA’s research showed this in 2006, 2008, and 2009, and this new data (with some reservations) shows the same. However, while tThe majority (64%) of enterprises rank themselves as extremely or very effective at managing virtualization, and believe they are getting better, I am skeptical. I contend many of those are overestimating their abilities (see my next points).
• Virtualization clearly increases complexity, and is clearly more difficult to manage. I felt like I was tilting at windmills when I published this opinion in 2006 and in 2008, contrary to common perceptions that virtualization made everything easier. I was certainly a lone voice, but as it turned out, a prescient one. It is great to see it being recognized more broadly, finally.
• Human issues continue to be major problems – especially skills and resourcing. EMA has found the same to be true, consistently, for many years. As recently as last week, I spoke with David Marshall of VMBlog and InfoWorld about how this continues to be a problem. This makes management tools even more important – to embed knowledge, define and execute policy, and automate routine work to free up resources.
• These data points all increase my doubt that enterprises are really being better at managing their virtual environments. It seems contradictory to me that this survey shows virtualization is more complex, management is the top inhibitor to ROI, and skills are still lacking, yet most enterprises think they are being very or extremely effective at it. Even though tool usage is more integrated and automated than it has been, this does not make sense. I am instead convinced that enterprises are really overestimating their abilities.

This is just a very small sample of the interesting data in this survey – there are more than 50 pages in the slide deck I reviewed ahead of the webcast. I encourage you to check out the webcast.  If you get in touch with HP, I am sure they will point you in the right direction; or check back here, and I will post the link when it is up.

Andi.

I like the thinking, and agree with a lot of the principles involved. Without doubt, virtualization is not cloud. But I can’t agree with it all. Apart from technical quibbles (like the part about mainframe LPARs not running on a hypervisor), I simply find it unreasonable, if not impossible, to think of implementing cloud computing without virtualization.

My key sticking point in most of these discussions [edit: not necessarily William's post - see comments below] is that they continually assume that ‘virtualization’ is synonymous with ‘hypervisor’, or at best with ’server virtualization’. Neither is true. When EMA first defined virtualization (a definition that has taken hold more or less throughout the industry), we defined it as:

“a technique for abstracting or hiding the physical characteristics of computing resources from the way in which other systems, applications, or end users interact with those resources.”

Even now, Wikipedia defines virtualization as “the abstraction of computer resources” and “hid[ing] the physical characteristics of a computing platform from users.”

No mention of a hypervisor there, and with good reason. Virtualization is much more than a hypervisor, and applies to much more than servers. In fact, EMA’s original definition made this clear by including the following clarifying note:

“This includes making a single physi­cal resource (such as a server, an operating system, an application, or storage device) appear to function as multiple logical resources; or it can include making multiple physical resources (such as storage devices or servers) appear as a single logical resource.”

Indeed, many forms of virtualization (and cloud) are possible without a hypervisor – like OS virtualization, storage virtualization, grid and cluster computing, terminal services, and more. So while it is widely known that Amazon runs its cloud on a classic server virtualization platform (Xen), even a Google-like cloud, which is based (as I understand it) entirely on a fully hardware-based deployment, without any hypervisors, is still using another virtualization technology – grid computing.

So cloud is definitely possible without a hypervisor, but is it possible without virtualization?

Perhaps, but it is far less than ideal.

William cited SoftLayer Technologies  as doing cloud on bare metal; and  Loudcloud as being cloud before it was in vogue. Although I am not sure the latter is true, and Softlayer provide few details about their bare-metal cloud, it seems to be possible to provide cloud computing without virtualization.

Yet with very few exceptions, it is ill-advised at best. In implementation, if not in theory, the many essential characteristics noted in the NIST cloud definition (EMA’s preferred definition) are only barely possible in a purely physical environment.

Sure, you could get rapid elasticity, rapid provisioning, minimal human interaction, dynamic resource assignment, location independence, resource abstraction, etc. with a physical deployment. While they were both substantially unsuccessful with customers, IBM’s On-Demand and HP’s Adaptive Infrastructure both accommodated these elements primarily through automation, and without virtualization (or at least with virtualization as only an optional component). Even without automation, you could imaginably provision and manage physical servers manually to achieve this on-demand, adaptive, cloud infrastructure. In theory, all things are possible.

In practice though, cloud computing without virtualization is barely realistic. It is an edge case at best. Given what virtualization can do – for resource pooling, rapid provisioning, reducing intervention, resource abstraction, workload elasticity, and more – why would you try to implement cloud without it?

And that is just on the server! Given the different types of virtualization – especially network virtualization and storage virtualization – it seems that cloud without virtualization is not just ill-advised, but positively crazy.

For example, would anyone really copy all the data from one DAS drive to another in order to ‘dynamically’ scale a workload onto a bigger machine? Would you uninstall a drive from one server, and put it into another? Would you physically switch or reprovision a network in order to abstract a new server located in a different data center? Even to the biggest skeptic, cloud without any virtualization must seem a ridiculous notion, if not an impossible one.

So yes, William is technically correct (“the best kind of correct!”) – virtualization is not cloud, and it is possible to provide cloud services without virtualization.

But (with apologies to Samuel Johnson) it is like a dog walking on his hind legs – it is not done well; but you are surprised to find it done at all.