What are the best steps to get on the 'red carpet' with virtualization?

Last week I spoke with Pam Baker, a writer with CIO Update, for an article titled The Top 5 Places to Use Virtualization. As you would expect from an experienced professional like Pam, it was a great article, with solid contributions from many others as well.

Pam specifically asked me to provide readers with advice on how to move into production with virtualization, and following our discussion published her article, including this section on ‘Low Risk Services’:

Move the easy stuff — Web servers, print servers, file servers, single-system applications, etc. — first. “Co-locating these environments on virtual machines delivers quick wins in business continuity, agility, resource efficiency, and of course cost savings — both cap-ex and op-ex,” explains Andi Mann, vice president of Product Marketing at CA Technologies Virtualization and Service Automation Business Unit. Moving low-risk services such as HR systems — file servers and Intranet applications, for example, but not payroll or e-mail — onto virtual machines is “a great next step into production virtualization.

However, I wanted to complete the thoughts I had while speaking with Pam, and address some of the other phases of virtualization deployment that we discussed.

What is the first service you should consider using virtualization?

Without doubt, application development is the very first place you should use virtualization. Dev/test – including unit test, system test, quality assurance, and user acceptance – is a great opportunity for virtualization because it is:

• Low-impact – it never touches a customer or even an internal user directly, and so even if you make ‘rookie’ mistakes they cannot damage customer service.
• High-reward – it allows applications to be developed, tested, and delivered both faster and cheaper, driving both agility and cost savings.

Plus, developers are already tech-savvy, so they can learn and deal with virtualization quickly and easily.

This is also a very strategic way to start, with a long tail of positive results. Applications developed on virtual servers can easily be deployed into production on virtual servers too. This gives you an easy route to production, with all the cost, continuity, and availability benefits that delivers.

At this stage you can also start to implement a ‘virtual-first’ policy for new applications – where every new service is deployed on virtual servers unless there is a clear business case – along with authorization, and even additional chargeback penalties – for requesting a physical server.

With this level of experience under your belt, you and your teams will quickly gain a broad, production-quality baseline of skills, knowledge, and ability to handle virtualization, while avoiding negative business impact as you acquire these capabilities in your teams.

This then establishes a solid ‘base camp’ to launch the next phase of virtualization – attacking existing production applications.

How can you move virtualization beyond the initial deployment?

Once you institutionalize virtualization in dev/test, and subsequent production deployments of new applications, as Pam noted in her article, you should look at moving existing low-risk/low-impact production services onto virtual servers next.

As I discussed with Pam, that will often mean virtualizing internal services, like your HR systems, file servers, or Intranet applications. However, just because they are internal systems, does not mean they are low-risk, or low-impact.  That is why I said you should probably leave payroll and e-mail alone in this phase – they are both not only high-risk, but also high-impact if anything fails.

Converting and migrating these low-risk, internal systems establishes another, higher-level  ‘base camp’ from which to expand your virtualization deployments. You can move to a broader virtualization deployment with greater confidence and lower risk, because you have the deeper experience.

Moreover, you have proven to the business how virtualization delivers incremental and substantial gains in CapEx reduction, OpEx reduction, agility, continuity, and time-to-market.

From there, you can then move into more complex, external-facing, mission-critical applications and services.

What are the best uses for virtualization?

Almost everything is a good use case for virtualization! Most organizations should be able to get 80-90% of their server workloads onto virtual machines – far more than the 16% of workloads that analyst firm Gartner says is running in virtual servers today.

The ‘low-hanging fruit’ of virtualization is, as Pam wrote, the “easy stuff” like Web servers, print servers, file servers, and simple, single-system applications. Co-locating these environments on virtual machines delivers quick wins in business continuity, agility, resource efficiency, and of course cost savings – both CapEx and OpEx.

Similarly, it is relatively easy to get new applications onto virtualization, by starting in development and test, and by implementing a ‘virtual first’ policy for new applications.

But even most of the ‘difficult’ applications – mission-critical, tier 1, OLTP, multi-tier, complex composite applications, etc. – can be virtualized with the right approach. These applications will benefit greatly from the improvements to scalability, continuity, performance, and resource efficiency that virtualization delivers.

What are the worst use cases for virtualization?

While it is true that almost all services can and should be virtualized, it is also true that some services are not well suited for a traditional, multi-VM, shared-server virtualization deployment.

The worst use cases for virtualization are where application services saturate one or more physical resources. If, for example, an application regularly uses over 90% of available CPU, memory, or network bandwidth, then there is no headroom left over for another system or service to use these resources. This means that it is not a good option to co-locate this application on a virtual server that shares physical resources with another application.

Typically such services include:

• CPU intensive applications – such as actuarial, modeling, design, or engineering applications
• Memory intensive services – such as database systems, data mining, or business intelligence
• Network intensive services – such as transaction processing or multi-user applications

Some services – such as corporate e-mail servers – may actually be all three.

However, you should never discount the benefits of deploying any application in a virtual server, even if it is deployed all by itself. Even if it will not provide hard ROI through hardware reduction, you can still gain major benefits in improvements to availability and continuity, operational costs, and ease of maintenance, by using virtualization.

How Did You Virtualize?

These are some ways to start with virtualization, some ways to expand virtualization, and some areas that you should probably leave until late in the cycle(if you virtualize them at all).

But I wonder where you started (or where you plan to start)? How did you expand beyond the low-hanging fruit? What types of services have you avoided virtualizing? Why?

Feel free to add your comments below. I would love to hear about your experiences.

(This entry has also been posted as an entry at CA.com – feel free to discuss there, or here)

Is 'VM Stall' A Stop Sign for Virtualization?

There appears to be a challenger to ‘VM sprawl’ as the scourge of virtualization success – a problem I call ‘VM stall’.

We know about ‘VM sprawl’ – because new virtual machines are so easy to deploy, organizations can end up with more VMs that they can handle, or even use. This has the potential to cause severe problems to availability, performance, compliance, costs, security, and more.

However, I am seeing more and more evidence of this new phenomenon I think of as ‘VM stall’ – the tendency for virtualization deployments to stall once the ‘low-hanging fruit’ has been converted (typically around 20-30% of servers).

I think it happens more or less like this…

In general, organizations start virtualization deployments by converting relatively low-risk, low-impact systems – dev/test servers, Web servers, file servers, internal applications, etc. – to virtualization. With a big impact, great results, and reasonably fast and easy implementation, it is a great hit with IT and business owners. This may even spawn a ‘virtual first’ initiative, where all new server requests are deployed as virtual servers by default.

However, when faced with the next step, converting the remaining existing servers – including tier 1 business services, customer-facing environments, enterprise-wide systems, 3^rd-party applications, multi-platform services, and composite applications – virtualization projects often stall.

I was interested to see the notion of VM stall confirmed again last week (courtesy of eWeek via @JSchroed) in some new research into virtualization (PDF) coming out of Prism Microsystems, a software vendor in the SIEM market.*

One of the most interesting outcomes in this research was again the low penetration of server virtualization within each organization. As the chart below shows, most organizations have still virtualized less than a third of their production servers.

Source: Prism Microsystems, ‘2010 State of Virtualization Security Survey’, April 2010

What’s more, fully 15% have not even started to virtualize their production servers at all!

It might seem that this is really at odds with ‘the common wisdom’ that sees virtualization as mature, ubiquitous, commoditized, and even passé. We hear so much about virtualization, how it has been a top priority for years, about how everyone is deploying virtualization. For example:

• The IBM Global CIO Study 2009 in September showed 76% of 2500 global CIOs are undergoing or planning virtualization projects
• The Gartner 2010 CIO Survey in January reported that virtualization is the top priority for over 1500 global CIOs (up from number 3 the previous year).
• In January, CDW’s Server Virtualization Life Cycle Report (registration required) found that 90% of respondents have implemented server virtualization at some level.
• As far back as 2008, EMA research showed 75% of enterprises were using virtualization for production use cases
• The Prism Microsystems report the chart above comes from states that 85% of their sample have adopted virtualization to some degree

I am even starting to hear that virtualization is set to be irrelevant, becoming nothing more than just a stepping stone to cloud.

However, despite the widespread adoption of virtualization as a percentage of organizations, it is consistently still very low as a percentage of production servers.

Indeed, this is not the only recent (and not so recent) research study to highlight this issue. Over time, CIOs have reported a persistent difficulty in expanding their virtualization deployments beyond the initial 20-30% of servers. For example:

• Around 6 months ago, Gartner reported that “only 16 percent of workloads are running in virtual machines today.”
• Research from EMA has found that the average organization has only virtualized around 25% of servers (and only retired just 17%).
• The CDW Server Virtualization Life Cycle Report cited above showed that just 34% of the average organization’s total server infrastructure consists of virtualized servers
• CIO and HP survey in October 2009 reported that on average just 38% of mission-critical business services have been virtualized by companies with virtualization projects
• Forrester Research from May this year (conducted for CA) shows that the average enterprise has virtualized only around 30% of their servers.

At a time when so many organizations are experiencing VM sprawl, it seems hard to believe that VM stall is such an issue. Yet time and again we see that organizations find it difficult to ‘get over the hump’ of the initial 20-30% of servers, and difficult to move from low-risk/low-impact servers to high-risk/high-impact services.

If this were just a point-in-time observation, then VM stall might not exist. The low penetration rate may just be a point in the deployment cycle. However, VM stall appears to be a longitudinal effect, as it has been holding many deployments at around 20-30% of servers for several years. IIRC, something resembling VM stall was cited as an issue in EMA research as far back as 2008, and again in 2009. The CDW virtualization lifecycle research also reinforces the potential for long-term VM stall. In it, even organizations that self-report as “fully deployed” for server virtualization have only virtualized 37% of their servers. So while many organizations see VM stall as a short-term delay to virtualization rollout, many others are seeing VM stall as a permanent situation.

I see many possible causes for VM stall. For example:

• Risk aversion – high-risk, high-impact services have more stakeholders, more politics, larger and more distributed infrastructures, greater cost of failure and downtime, reduced or non-existent 3^rd-party support, and maximum management attention, among many other risk factors. The risk of failure may be too great, and the newest technology is always blamed for any new problems. Without new ways to address continuity, availability, performance, cost allocation, and other business requirements, conversion risk may be enough to stall virtualization deployment.
• Resourcing – with around 20-30% of servers converted, virtualization staffing starts to become a real challenge. As I talked about recently with my great mate, David Marshall, staff and skills shortages put a real throttle on virtualization deployments, especially as virtualization starts to scale. Not only is demand for virtualization skills still high, but supply continues to lag. Plus, the problem is getting worse, not better. Without the resources and skills to go forward, there is often little alternative to VM stall.
• Scalability – with one (typically small) team trying to manage a quarter of the entire server workload, staff from the virtualization project team simply cannot handle further virtualization deployment. In some cases, the virtualization technology itself does not scale well either; and in others, the management tools do not scale. Throwing more bodies at the problem is rarely the answer – after all, nine women cannot make a baby in one month. So organizations end up with VM stall almost by default, as they find that they need to fundamentally change their processes and technologies to enable further virtualization growth.
• Manageability – new IT management issues come up as the scale and risk of virtualization deployment increases. Enterprise virtualization needs new approaches to performance assurance, process automation, VM mobility, continuity planning, security and audit, software compliance, OEM support, configuration compliance, and more. The importance of manageability is greatly magnified  for high-risk/high-impact services, but few (if any) organizations seem to have the virtualization-aware management tools to scale to handle enterprise-class virtualization deployments. Again, VM stall happens almost by default, as IT tries to figure out enterprise-class manageability.

There may be more or different causes, but whatever the reasons, there is little doubt in my mind that VM stall exists. It is not universal – indeed, every study shows that a decent percentage of organizations are able to power through it – but for the majority of organizations, it appears to be very real. I have personally seen many enterprises going through it. More and more research continues to support it. For affected organizations, it is a significant problem, too, because stalled virtualization deployment means the highly desirable outcomes of virtualization – OpEx reduction, improved continuity, greater IT and business agility, energy cost reduction, ROI, etc. – either stalls as well, or even starts to backslide.

Whether VM stall represents as big a problem as VM sprawl, time will tell; but it is certainly a significant and growing challenge to the success of virtualization – and a fundamental driver for better virtualization management.

(EDIT: This article has been picked up and published on CIO.com! Join in the discussion there, or here.)

Is old-school physical security really 'good enough' for virtualization?

Whatever happened to virtualization security?

Back in the day, everyone was talking about blue pills and red pills, about sideways attacks and DOM-0 threats, about security profiles and isolation policies, about perimeter defense and security embedded in the hypervisor.

Then, all of a sudden, the buzz seemed to disappear. It really seems like organizations simply don’t have the time, money, desire, or otherwise to pursue dedicated virtualization security.

Indeed, it seems like most of the pure-play virtualization security vendors have folded, been sold, or reworked their strategy.

For example:

• Blue Lane ended up being sold to VMware, reputedly at a bargain price, after failing to get any traction.
• Third Brigade was rolled up into Trend Micro, and now offers a solution for combined ‘physical, virtual and cloud’ protection.
• Reflex and Catbird have repositioned to highlight their value in configuration, compliance, and/or systems management (in addition to their security value).
• Tripwire and Configuresoft have long promoted some virtualization security values, but were never really pure-play virtualization security vendors.
• Even security specialists like Symantec and RSA do not push virtualization security products, preferring mainly to build on existing security paradigms to support virtualization.

Of course VMware still has vShield Zones and the VMsafe API, but of the ISVs it seems that only Altor Networks still plays strongly in the pure-play virtualization security space.

This barely sustaining demand for pure-play virtualization security was reinforced last week in new research from Prism Microsystems (PDF), a software vendor in the SIEM market* (which I learned about in eWeek via @JSchroed). Possible vendor/sample bias aside, this research showed quite starkly how many respondents are securing their virtual environment using traditional (or no) security, and how few are using virtualization-specific security:

Source: Prism Microsystems, ‘2010 State of Virtualization Security Survey’, April 2010

In confirmation of this ennui, Gartner recently predicted at least a 5 year maturity cycle for virtualization security.

All of this is especially perplexing, because there is no doubt virtualization security is still top-of-mind for some very smart and dedicated people. The Prism Microsystems research, for example, says that 86% of its respondents acknowledge that securing virtualization is as important as securing their physical environment.

So I am unclear as to what is causing this lack of market interest. Perhaps CIOs (and/or CISOs):

• saw virtualization security as unnecessary insurance against threats that have never played out ‘in the wild’
• rated the potential financial impact of any additional risks as low enough that they can simply accept them
• believe that vShield Zones and VMsafe are all that is needed (but what about Hyper-V, Xen, etc.?)
• decided instead to invest in management disciplines with more straightforward ROI (virtualization, automation, configuration management, asset management, etc.)
• have simply been unable to justify virtualization security purchases during the economic downturn

Whatever the reason, it really does focus the question: does virtualization security really matter?

In my opinion, it absolutely does. Yet, it seems to me that decision makers are saying that standalone virtualization security is more important theoretically, from a technology and business perspective, than it is in practice. Most enterprise buyers – for better or worse – apparently believe that their existing security paradigns are at least ‘good enough’. They definitely appear instead to be taking classic intrusion detection, data loss prevention, identity & access, and other entrenched security management disciplines, and adapting them to the new technologies of virtualization (and probably cloud as well).

All of which actually does make sense. Without any major virtualization-specific exploits in evidence, perhaps they are right. While it may be valid to take the view that  it is only a matter of time until they are proven wrong, perhaps extending traditional security capabilities into the virtual world is indeed a good approach, at least for now.Perhaps CISOs are actually ahead of the game, integrating management across virtual and physical domains even while their systems and operations counterparts are maintaining virtualization as a silo.

Regardless of whether it is the right approach or not, one thing is apparent – the heat is off the pure-play virtualization security market, at least for now. As CIOs and CISOs focus on applying traditional physical security paradigms on their virtual environments, a different breed of cross-domain, integrated, and extensible tools are proving superior value – at least for now.

Effort vs. Payback is an Everyday Business IT Decision

I read recently a good blog post from Thomas Bittman (@tombitt) of Gartner Group, about how sometimes close enough is good enough. Talking specifically about private cloud, he talked about how an ‘imperfect’ cloud deployment – one that does not have all five essential characteristics, for example – might be enough for some organizations.

I especially appreciated how he highlighted some very specific, real-world examples to sustain his advice. As he shows, sometimes you don’t need a ‘100%’ implementation, and for very good business reasons.

Not every IT organization needs a fully self-service interface, and many smaller organizations see no value in usage metering. They simply want to deliver services faster. For them, a 70% private cloud is absolutely good enough … it all comes down to business requirements, return on investment, and future strategy. How far you go is your decision.

via Driving for Imperfection With Your Private Cloud.

If you haven’t seen it yet, you should. It’s a quick read, only 4 paragraphs and less than 300 words. Go ahead. I’ll still be here when you get back.

The theme is very similar to something I wrote in a research report for EMA, ‘The Responsible Cloud‘, also on cloud computing. Regarding the NIST definition of cloud, I cautioned against dogmatic interpretations of cloud computing, and the notion that a ‘real’ cloud must necessarily have all of the essential characteristics, or fit some specific deployment model. Flexibility is key, I advised, and delivering on key business requirements is more important than definitions.

Two other things happened this week that made me think about this in different ways:

• An internal session at CA reviewing some customer-facing materials. All attendees agreed – we can’t preach unattainable dogma; we need to deal with specific requirements and partial deployments, as well as broad requirements that come from  ‘100%’ implementations.
• A group discussion on LinkedIn, where an IT practitioner wanted advice on building a small private cloud. He was soon inundated with an unrealistic list of requirements, from hypervisor features to management disciplines, that he *must* have to build a ‘100%’ cloud.

The similar inferences in three otherwise unrelated conversations started me thinking more broadly about ‘100% adoption’. It IT, as in life, you never really need a Rolls Royce. You can aspire to the quality, appreciate its refinement, and in some cases you may be fortunate enough to actually enjoy it, but there is a point where it simply doesn’t make sense to pursue that level of luxury. Mostly you can get away with a Ford. Sometimes you can even make do with a second-hand Lada.

The same Pareto-like principle applies roughly throughout IT (much to the annoyance of just about every security pro I have ever met) – although the actual ratio may vary wildly, you can often get most of the benefit from less than a ‘100%’ implementation.

The phrase that sprang to mind for me was the same conclusion that I published elsewhere in the Responsible Cloud report, and the same notion that many IT pros live by, day in and day out:

It is important to look for opportunities, and do what makes sense

This should not just apply to cloud computing, but across all of IT.

Take, as another example, adherence to the IT Infrastructure Library (ITIL). Now, ITIL is a great framework, and an increasingly definitive reference for best practices in IT management. Data I have seen suggests as many as 60% of all IT organizations are committed to ITIL, and that implementation of ITIL (whatever that actually means) results in measurable and specific benefits in IT costs, staff and server efficiency, operational maturity, and more.

However, I also hear and read somewhat justified rants about how “ITIL just doesn’t work … ITIL is more 1960s than 2010 … it’s useless.” Yet the truth is, as so often, somewhere in the middle. In this too enterprises can definitely benefit from avoiding the dogmatic application of every single prescription. The same is true for other standards such as COBIT and ISO, or prescriptions from standards groups like the DMTF or NIST. All can deliver significant benefits with less than a 100% implementation.

It also applies in internal adoption of standard operating environment (SOE) components, like making singular (and often binding) choices between, for example:

• VMware vs. Hyper-V vs. Xen
• HP vs. Cisco vs. IBM
• HDS vs. NetApp vs. EMC
• Windows vs. Linux vs. UNIX
• iPhone vs. WinMo vs. Blackberry
• Solution suites vs. point products
• Mainframe vs. Commodity
• Physical vs. virtual vs. cloud

In all these cases and more, although standardization can have specific benefits, the greatest benefit to the enterprise does not always accrue from making an exclusionary choice; from committing to a 100% implementation. Most IT practitioners know that heterogeneity is the new standard – whether intuitively or grudgingly. They know that sometimes the best – or at least necessary – outcomes arise from providing multiple choices, fit to support multiple use cases.

Of course some areas are less flexible. You cannot, for example, pick and choose which parts of PCI, HIPAA, or Sarbanes-Oxley compliance would work best for you. Perhaps ‘close’ only matters in horseshoes and hand grenades, but for sure it doesn’t matter in legal compliance.

However, where possible, IT – practitioners, consultants, vendors, and analysts – need to stay away from dogma. We must avoid making any architecture, maturity model, or industry standard a religious ‘all or none’ battle. Important though they may be, these are not religious battles. These are IT decisions. Moreover, these are business decisions. So we need to keep the business goals in mind, and realize that sometimes a ‘100%’ implementation simply does not make sense.

IT is the Magpie of the Business World

I have a request. I hope it is not too onerous, because something is really starting to grind my gears.

Can we in IT please all stop claiming that any technology is going to kill another?

The latest I am reading, for example, is that NoSQL (for want of a better term) will kill off SQL.

No, it won’t.

My hyperbole aside, I know this with complete and utter certainty,  even though I am barely conversant in database technologies. Seriously, SQL hasn’t even killed off VSAM – first released in 1974 – which is still the foundation for a huge volume, perhaps even the majority, of our daily financial, logistics, retail, and government business. In fact, not only are we still storing data in VSAM, we are still programming in COBOL, and even doing it on 20 year old mainframes. So realistically, an upstart like NoSQL has no chance of killing anything.

Similarly, virtualization will not kill the physical computing infrastructures that came before it. Even most early adopters are struggling to get over 50% of their servers virtualized, while the average penetration is, by some reports, as low as 16%. Meanwhile, the percentage of desktops that have been virtualized is still in single digits. In some cases, so-called ‘legacy’ systems are actually becoming their own hypervisors (e.g. Windows, z/VM, Solaris, and Linux).

The same is true of cloud computing. Even if, as Gartner predicts, by 2012, 20 percent of businesses will own no IT assets – which I find highly dubious; and even if the cloud computing market will be worth $160bn by 2011 – also somewhat dubious; then still a vast majority of organizations will continue to own their IT assets. Even allowing for some substantial private cloud deployment (much less dubious), there is no chance cloud computing will kill the on-premise, installed and owned, IT environment.

Historically, this has always been true. Distributed computing never fully replaced mainframe computing. Indeed, the mainframe is actually experiencing record levels of growth (.pdf) in particular among heavy mainframe users (over 500 MIPS). Personal computing never replaced distributed computing either. The Internet did not kill local computing; thin clients did not kill desktops; Firefox did not kill IE (although IE did eventually kill Netscape); Java did not kill COBOL, let alone C; disk did not kill tape; Salesforce.com did not kill Siebel; Google did not kill Yahoo; Gmail did not kill Exchange.

In fact, it is really quite rare that any new technology completely kills off any other. We in IT are the magpies of the business world, collecting and hoarding all the shiny technologies we can. These are not just collector items or museum pieces though; these are real, mission-critical systems and applications. So we end up with a hybrid of critical technologies spanning not just years, but decades.

(Which is why I am such a strong proponent of heterogeneous IT management … but that is another article)

Perhaps it is just semantics, or a philosophical distaste for absolutes. Perhaps the rampant pace of IT development just makes it seem like we don’t replace technology (when of course we do).

But I would still be really happy if we could all refrain from declaring the death of any technology.

Because chances are it is simply never going to happen.