More than four reasons count towards VM stall

I recently saw a great article in IT World Canada titled “Virtual stall: What it is and why you have it,” written by Jay Litkey, that took up my idea of VM stall, which I first came up with in my blog from May ‘Is “VM Stall” the Next Big Virtualization Challenge?‘.

Though they barely acknowledge my blog as their inspiration (and as a competitor to CA Technologies – my employer – why would they?), it seems Jay and his team have wholeheartedly taken up my concern with VM stall, and not just in the IT World Canada article. Marketing lead David Lynch was quoted on the topic in a post by Bruce Hoard of Virtualization Review, and in a recent Tech Target article on ‘ISV stall’. Several posts on their corporate blog also address the issue as if it was their own baby.

In my past life at EMA, I have spoken with both Jay and David a number of times, and had a lot of time for what they were doing in the management space. For a small startup with limited resources, it is great that they can take the time to pick up my idea and run with it.

The IT World Canada article is really worthwhile, because it zeroes in on some important concepts. It helps to expand the thought around VM stall, and specifically on a couple of additional causes, as it notes:

Virtual stall has four main causes:

• Scalability issues:  A single IT team often finds it difficult to scale beyond the 25-30 per cent penetration range. This is due to the combination of lack of automation and reporting in virtualization management tools, creating time-consuming manual processes that are a particular problem when there is a lack of experienced and trained staff.
• Management issues: The data centre is not a place that can be managed manually; there are too many elements to be checked, and too many independencies [sic]. And, while there are levels of automation built into the virtualization platform, they can be difficult to define and implement. The lack of automated monitoring, alerting and control becomes more and more of a problem as the overall level of virtualization in the data centre increases.
• Process issues:  Enterprise virtualization impacts a wide range of existing data centre processes, all of which need to be modified, replaced, or augmented. As long as the virtual environments are small and self-contained, these processes can be manipulated or ignored. But as the environment grows, it reaches a point when they have to be dealt with before real efficiencies can be reached. The more “process-mature” an organization is, the more quickly this point is reached.
• Co-ordination issues: Virtualization crosses multiple silos and ultimately requires a level of co-operation and integration that is impossible to achieve with the traditional silo management structure. In addition, the first workloads to be virtualized tend to be less critical ones.  However, as environments grow, higher-risk, higher-impact services are virtualized. These tend to have more stakeholders, more politics, more distributed infrastructures, and a greater cost of failure and downtime. Consequently, they require more coordination.

This is great insight, and offers a number of important causes. However, I don’t think it is reasonable to say there are just “four main causes.” Not to pick on Jay, as it is probably just unfortunate phrasing, but I think it is important to see that the issues of VM stall are much more varied, complex, and numerous.

I am not entirely without fault either. To start with, when I first identified the issue of VM stall in my blog post back in May, I said that “I see many possible causes for VM stall,” but like Jay I only identified four examples. As Jay recounts in his analysis, I saw scalability and manageability as key issues; but unlike Jay, I chose to highlight risk aversion and resourcing as two more of my examples.

However, even these six are just a part of the problem. As I said when I spoke with my great mate (and one of the industry’s great virtualization gurus, observers, and commentators), David Marshall of Hyper9 and InfoWorld in his article, “VM stall: Breaking through the second phase of virtualization“:

“… many organizations strike a ‘perfect storm’ of challenges that slows their virtualization rollout, or stops it entirely. Some causes at this stage include greater complexity of services and applications, higher demand on scarce virtualization skills, limited visibility into a growing deployment, increasingly heterogeneous systems, and greater resistance from risk-averse application owners and recalcitrant application vendors.”

In the same article, David spoke with Dave Bartoletti, formerly of automation vendor Enigmatec and now a leading light showing the way through the virtualization darkness with research and advisory analyst firm, the Taneja Group:

“The second wave of issues is always harder when a core technology matures. Server virtualization essentially paid for itself in CAPEX savings, but when we virtualize Tier 1 business-critical applications, or user desktops, CAPEX savings take a backseat to application performance and IT efficiency, and this is why we’re stalling.”

My former editor at Tech Target and another keen virtualization observer, Colin Steele, highlighted another core element of VM stall, in his article “ISV stall makes virtualizing applications a challenge“:

By now, the benefits of virtualizing applications are clear, but the goal of 100% virtualization remains elusive. One reason is that some independent software vendors (ISVs) don’t support their server-based applications — databases, telecom apps, healthcare programs, etc. — on virtual servers.

Moreover, I talk a lot with customers about their real world concerns, so I can quickly pinpoint many other causes. They talk to me about issues like vendor licensing, facilities constraints, capacity blindness, service prioritization, deployment costs, line-of-business resistance, internal politics, a lack of skills, and even senior management resistance.

In fact, last week at CA Expo in Australia, I talked with CA Technologies customers about seven significant issues in virtualization that are contributing to (among other things) VM stall, as you can see from one of the slides from my presentation:

Virtualization is not clear sailing - from CA Expo Australia

(You can see the whole deck at the CA Expo site)

To be fair to Jay and his team, other posts on his corporate blog agree with me, citing issues like mission-critical apps, management skepticism, bureaucracy, poor project vetting, and more.

I am really glad to see my thoughts around VM stall have captured the imagination of the market. Thanks to Jay for taking this up, and to his team for joining me and CA Technologies in raising awareness of issues causing VM stall.

However, I think we all need to be careful about being categorical about VM stall. It is important to be clear that VM stall – like most enterprise IT issues, and indeed most organizations – is both complex and varied, so trying to categorically define four (or six, or seven, or really any number) of causes for VM stall is underestimating this important problem.

But if we can all contribute new ideas to the community, we will all learn more, and our enterprise customers will benefit from our combined wisdom.

What are the best steps to get on the 'red carpet' with virtualization?

Last week I spoke with Pam Baker, a writer with CIO Update, for an article titled The Top 5 Places to Use Virtualization. As you would expect from an experienced professional like Pam, it was a great article, with solid contributions from many others as well.

Pam specifically asked me to provide readers with advice on how to move into production with virtualization, and following our discussion published her article, including this section on ‘Low Risk Services’:

Move the easy stuff — Web servers, print servers, file servers, single-system applications, etc. — first. “Co-locating these environments on virtual machines delivers quick wins in business continuity, agility, resource efficiency, and of course cost savings — both cap-ex and op-ex,” explains Andi Mann, vice president of Product Marketing at CA Technologies Virtualization and Service Automation Business Unit. Moving low-risk services such as HR systems — file servers and Intranet applications, for example, but not payroll or e-mail — onto virtual machines is “a great next step into production virtualization.

However, I wanted to complete the thoughts I had while speaking with Pam, and address some of the other phases of virtualization deployment that we discussed.

What is the first service you should consider using virtualization?

Without doubt, application development is the very first place you should use virtualization. Dev/test – including unit test, system test, quality assurance, and user acceptance – is a great opportunity for virtualization because it is:

• Low-impact – it never touches a customer or even an internal user directly, and so even if you make ‘rookie’ mistakes they cannot damage customer service.
• High-reward – it allows applications to be developed, tested, and delivered both faster and cheaper, driving both agility and cost savings.

Plus, developers are already tech-savvy, so they can learn and deal with virtualization quickly and easily.

This is also a very strategic way to start, with a long tail of positive results. Applications developed on virtual servers can easily be deployed into production on virtual servers too. This gives you an easy route to production, with all the cost, continuity, and availability benefits that delivers.

At this stage you can also start to implement a ‘virtual-first’ policy for new applications – where every new service is deployed on virtual servers unless there is a clear business case – along with authorization, and even additional chargeback penalties – for requesting a physical server.

With this level of experience under your belt, you and your teams will quickly gain a broad, production-quality baseline of skills, knowledge, and ability to handle virtualization, while avoiding negative business impact as you acquire these capabilities in your teams.

This then establishes a solid ‘base camp’ to launch the next phase of virtualization – attacking existing production applications.

How can you move virtualization beyond the initial deployment?

Once you institutionalize virtualization in dev/test, and subsequent production deployments of new applications, as Pam noted in her article, you should look at moving existing low-risk/low-impact production services onto virtual servers next.

As I discussed with Pam, that will often mean virtualizing internal services, like your HR systems, file servers, or Intranet applications. However, just because they are internal systems, does not mean they are low-risk, or low-impact.  That is why I said you should probably leave payroll and e-mail alone in this phase – they are both not only high-risk, but also high-impact if anything fails.

Converting and migrating these low-risk, internal systems establishes another, higher-level  ‘base camp’ from which to expand your virtualization deployments. You can move to a broader virtualization deployment with greater confidence and lower risk, because you have the deeper experience.

Moreover, you have proven to the business how virtualization delivers incremental and substantial gains in CapEx reduction, OpEx reduction, agility, continuity, and time-to-market.

From there, you can then move into more complex, external-facing, mission-critical applications and services.

What are the best uses for virtualization?

Almost everything is a good use case for virtualization! Most organizations should be able to get 80-90% of their server workloads onto virtual machines – far more than the 16% of workloads that analyst firm Gartner says is running in virtual servers today.

The ‘low-hanging fruit’ of virtualization is, as Pam wrote, the “easy stuff” like Web servers, print servers, file servers, and simple, single-system applications. Co-locating these environments on virtual machines delivers quick wins in business continuity, agility, resource efficiency, and of course cost savings – both CapEx and OpEx.

Similarly, it is relatively easy to get new applications onto virtualization, by starting in development and test, and by implementing a ‘virtual first’ policy for new applications.

But even most of the ‘difficult’ applications – mission-critical, tier 1, OLTP, multi-tier, complex composite applications, etc. – can be virtualized with the right approach. These applications will benefit greatly from the improvements to scalability, continuity, performance, and resource efficiency that virtualization delivers.

What are the worst use cases for virtualization?

While it is true that almost all services can and should be virtualized, it is also true that some services are not well suited for a traditional, multi-VM, shared-server virtualization deployment.

The worst use cases for virtualization are where application services saturate one or more physical resources. If, for example, an application regularly uses over 90% of available CPU, memory, or network bandwidth, then there is no headroom left over for another system or service to use these resources. This means that it is not a good option to co-locate this application on a virtual server that shares physical resources with another application.

Typically such services include:

• CPU intensive applications – such as actuarial, modeling, design, or engineering applications
• Memory intensive services – such as database systems, data mining, or business intelligence
• Network intensive services – such as transaction processing or multi-user applications

Some services – such as corporate e-mail servers – may actually be all three.

However, you should never discount the benefits of deploying any application in a virtual server, even if it is deployed all by itself. Even if it will not provide hard ROI through hardware reduction, you can still gain major benefits in improvements to availability and continuity, operational costs, and ease of maintenance, by using virtualization.

How Did You Virtualize?

These are some ways to start with virtualization, some ways to expand virtualization, and some areas that you should probably leave until late in the cycle(if you virtualize them at all).

But I wonder where you started (or where you plan to start)? How did you expand beyond the low-hanging fruit? What types of services have you avoided virtualizing? Why?

Feel free to add your comments below. I would love to hear about your experiences.

(This entry has also been posted as an entry at CA.com – feel free to discuss there, or here)

How should vendors and others fight FUD?

I, along with many others, witnessed this week (or was it last week?) a public squabble between two well-known vendors in the virtualization market. Of course, this is nothing new. The whole world has been watching as Adobe attacked Apple (and Apple responded) over Flash support on the iPad. Before that, of course, America was regaled by the amusing Verizon campaign attacking AT&T (‘There’s a map for that’). Last year the gloves were well and truly off between VMware and Microsoft over Hyper-V bluescreens. Apple and Microsoft were at it last year when both ran their ‘I’m a PC’ ads (each side taking very different interpretations), and they were at it again just this week.

Now, a lot of these ‘fights’ seem to be what is referred to in rugby circles as ‘handbags at 10 paces’ – a long-distance squabble with a lot of pushing and posturing and preening, but little actual contact, and no actual damage. But occasionally these fights – like in rugby – get very serious, with real hits on both sides, and a lot of very real damage (to reputations, customers, influence, sales, and more). Unfortunately, in both cases, the damage seems to be to both sides of the stoush.

This impact is especially apparent when it is influencers (bloggers, analysts, media, etc.) who are (re)publishing the attacks. After all, many influencers are driven by a marginally slower version of the ’24-hour news cycle’ – the need to get content to print quickly so they get more eyeballs as the first to ‘break’ a story. This means that they can end up accepting any reasonable story at face value. Some of the more provocative authors seem unfortunately to do this more consciously, and have led to the sad resurgence of ‘yellow journalism’ (especially driven by the pay-per-click model for Web site advertising). However, even the most honest and scrupulous of authors can occasionally publish excitedly about unconfirmed future releases, saucy though unfounded rumours, or interesting secrets and leaks. Or they may simply end up for one of many reasons publishing content that is not untrue, per se, but is simply not evenly balanced.

All of this has spurred me to think more deeply about how to react to vendor FUD (Fear, Uncertainty, Doubt). The way I see it, there are three core stakeholders in these mix-ups – the vendor(s), the enterprise customers/buyers, and various commentators/influencers.

So how should these three constituents react to unbridled FUD from software vendors?

Vendor Responses

The first decision for a vendor is whether to respond at all. We should never underestimate this option – depending on the case, it is frequently better to simply take the high road. After all, as the old saying goes, ‘never wrestle with a pig – you both end up covered in s**t, but the pig enjoys it’.

If a vendor does choose to respond, the next question is, how strong should their response be? They must decide between a subtle approach, and attacking the issue head-on. It is one thing to whisper snide remarks in dark alleys, another to come straight out and state the facts as you see them, and something entirely different again to really go in ‘boots and all’. If they decide to go big, they can choose anything from an attack blog to a full-blow PR campaign – but each has significant issues.

Social media is clearly an option, but is a vendor blog or tweet really going to be effective, or is it just singing to the choir? A full-blown PR campaign attracts big attention, but gives oxygen to the fire, and risks both putting the competition on a pedestal and making the responder look petty. Using personal relationships to bring up the issues one-on-one with customers is more subtle and ‘high-road’, but will never reach everyone that has already heard the FUD.

Getting influencers to balance the table for you is an excellent outcome if it can be achieved. However, it is not always an easy option. Some influencers have already made up their mind by the time they publish; some may simply have moved on from that story and have no desire to go back to it; still others may just not be interested in talking to a vendor. How a vendor works with influencers to help provide balance after they have already published a damaging story is a perennial issue. If anyone figures this out definitively, let me know!

Buyer Responses

For the CIO and other enterprise buyers, the real problem is not so much the fight itself, as much as knowing what to believe. So the first response should always be to try to find alternative viewpoints from a wide variety of opinions.

Certainly gathering multiple analyst opinions is a great option. If you are a subscriber with one or more analysts, and the issues raised are important enough to burn through your subscription time & dollars, set up a quick call to discuss directly. If it is not a burning issue, wait and see if they publish a research note on the topic (handy hint: you can often get these for notes free if a vendor decides to license them – but be aware that when vendors do license analyst reports, it is mostly because they are positive about that vendor, so take it all with a grain of salt). If you are not a subscriber, then try just reaching out on Twitter. Some analysts will give up 140 characters (or even more) for free, although of course many will not.

Beyond the independent analyst community, look toward different media sources so you are not just reading one side of the story, or even just what one source is publishing. Some stories build a life of their own, some journals and blogs are part of a single network (and may be influenced collectively). Some individual authors or outlets will have a decided bias – whether permanently or temporarily, intentionally or accidentally – so look for another source with a different bias. And remember that almost no human communication can be without any bias at all.

Another excellent source for alternate viewpoints is your peers both in IT and business. Reach out to your user groups, contacts from conferences, internal colleagues and former colleagues/peers at other companies. To find new peers and new opinions, try using social media like LinkedIn groups, Facebook, Google groups, Twitter etc. Post invitations to discuss the issues on your own blog if you have one; if you don’t, then comment on other people’s blogs with your own opinions and questions.

You can also reach out to the vendor themselves, as well as their competitors, and get them all to respond directly. Try connecting on LinkedIn, Twitter, or their blog; or call or e-mail a sales rep or their marketing people. Believe me, they will love to hear from you, and be more than happy to give an alternate opinion – perhaps over lunch or a beer! You will at least then get both sides of the story. Even though both sides may be horribly biased, you can normally figure out some middle ground for yourself.

Influencer Responses

It is simply a fact of life that influencers – including analysts, media, bloggers, tweeters, and more – can be just as vulnerable to a well-crafted message as any other humans, and can buy into well-crafted vendor FUD just like the rest of us. However, most influencers also realize that it is critically important to their credibility and livelihood to present a balanced view.

The most important requirement then is probably the most basic, and a matter or course for most professional writers – check your facts with reliable sources. Go beyond an initial or single source, even if they are unimpeachable – there is always the chance that a single source has been honestly misled themselves, or even that they are simply not well-informed of all sides of an issue.

When writing about one vendor – especially based on their own releases or references – you should also actively  try to find out what their competitors are saying. Again, their sales and marketing people will be more than happy to talk with you.

Influencers especially need to make sure their content is defendable from all angles, as this independence is fundamental to their credibility and reputation. When I wrote as an analyst, I always made sure I was able to defend my content against the most rigorous accusations from all sides – the enterprise users I wrote for, the vendors I worked with, the vendors they competed with, my peers within the firm, and the broader influencer community. Many writers even have a formal process for this peer review, and it is certainly a best practice among the larger analyst firms.

If, despite your honest efforts and intentions, you find that you have inadvertently published some content that lacks balance or independence, then please make the effort to redress the balance. It is only fair to show the opposite opinion. That does not always mean re-writing or publishing a correction; on the contrary, it may be a very positive opportunity to publish a new article, blog, or research note as a follow-up, which can even magnify the number of readers, hits, and content sales being driven by the one issue.

Influencers should especially treat vendor leaks, rumours, and exclusives very cautiously. When given references, you should try to find references that are not recommended by the vendor, and make sure to verify their information with other independent third parties.

There Must Be Other Options

These are just my top-of-mind thoughts on the topic. To be honest, I am really not sure what options are best for any of these stakeholders (or even if there are other stakeholders that I am not focused on). Moreover, I am sure I have missed some other good options. I have discussed some of this online, but I am far from convinced of the efficacy of any one approach.

Moreover, there is one unfortunate caveat to all the above: when you are inside the IT echo chamber, it can be difficult to even find alternate opinions. As we all talk amongst ourselves, we repeat to each other what we have all heard from each other, so we all risk just going with the flow. No matter where you stand in this regard, it is always important to keep looking. Other voices are almost always out there.

Of course, the best responses to FUD will vary from one situation to another, but are there some that are always appropriate? Are there some that will never work? Are some responses just fundamentally wrong? Even if some options are odious and unattractive, are they nevertheless fair game if they are still effective (as some believe about political attack ads)?

Feel free to comment here, or just hit me up on Twitter (@AndiMann), and add your own ideas. Maybe I will run a survey to see if I can get more input that way.

Because there has to be a better way to deal with FUD than handbags at 10 paces.

Federal CIO Vivek Kundra and the CIO Council

If there was still any doubt about the real world use cases for cloud computing, the US Federal Government last week published a 38-page report  entitled “State of Public Sector Cloud Computing” (link to PDF at CIO.gov). Attributed to the Federal CIO Vivek Kundra, it is stamped with the seal/logo of the CIO Council, which comprises the CIOs of some 28 federal government agencies.

The report details 30 case studies in public sector cloud computing (for both state and federal governments), covering IaaS, PaaS, and SaaS service models; using private, public, community, and hybrid cloud deployment models; with both on-premise and off-premise implementations.

Measurable Benefits from Key Case Studies

After perfunctorily reciting what it calls “the broadly recognized and adopted NIST Definition of Cloud Computing,” and using the opportunity to briefly push its own barrow on cloud standards (a subject I plan to blog about in more detail at another time), the report cites several projects with ‘soft’ outcomes – improved productivity, better efficiency, higher reliability – as well as several planned cloud projects that are yet to bear fruit.

However, most of the report is given over to demonstrating solid and measurable outcomes from over a dozen current cloud deployment case studies involving multiple state and federal government agencies, with cloud success stories such as:

• The US Army is piloting a customized version of Salesforce.com to update its 10 year old recruiting systems for Web 2.0, social media, mobile devices, marketing integration, real-time data interchange, and engagement tracking. At an annual cost of $54,000, this pilot compares to bids from traditional IT vendors ranging from $500K to over $1 million, and has already replaced five traditional recruiting centers.
• The Department of Health and Human Services is also using Salesforce.com to support the implementation of Electronic Health Records systems. This new CRM system for working with participating healthcare providers was deployed in just 3 months, instead of the full year estimated for an internally delivered system.
• The General Services Administration (GSA) moved to a Terremark Enterprise Cloud service, to take advantage of on-demand scalability for Web sites like USA.gov. As a result, GSA accelerated its site upgrade time from nine months to a maximum of one day, reduced monthly downtime from roughly two hours to near zero (99.9% availability), and reduced annual costs for USA.gov by $1.7 million, from $2.35 million to $650,000, or 72%.
• The Defense Information Systems Agency (DISA) is using virtualization with a self-service portal to provide on-demand server space for development teams. With just an approved Government credit card, these end users can set up new environments (with DoD-compliant security guaranteed) in just 24 hours – down from three to six weeks – and at a “reasonable” cost.
“DISA estimates PaaS cloud savings between $200,000 and $500,000 per project.”
• DISA also used cloud provider CollabNet to set up Forge.mil, a private PaaS cloud development environment with a heavy focus on collaboration and code sharing/reuse. DISA estimates this saves between $200,000 and $500,000 per project – not including the estimated $15 million in cost avoidance by utilizing an open source philosophy.
• The Lawrence Berkeley National Labs (LBL), part of the Dept of Energy, is using Google Apps for 2,300 e-mail users, and planning to more than double that by August. LBL estimates they will save $1.5 million over five years “in hardware, software and labor costs from the deployments they have already made.”
• NASA’s Jet Propulsion Laboratory used a Microsoft Azure development platform “to excite the public about Mars” with the website, BeAMartian.jpl.nasa.gov. This site has generated over 2,000 pieces of social media, inspired 200 traditional media stories, responded up 2.5 million API queries, gathered  40,000 votes in its ‘Town Hall’ polls, and attracted 5,000 registrations from individuals and teams.
• The Federal Labor Relations Authority recently replaced its underperforming, decade-old case management system, switching to Intuit’s Quickbase system. As a result, it was able to go from requirements-definition to completed development in 10 months – a quarter of the original deployment time – and expects a TCO reduction of nearly $600,000 over five years.
“Moving Recovery.gov to Amazon EC2 will drive cost savings of $750,000”
• Less than a month ago, the Recovery Accountability and Transparency Board moved Recovery.gov to a “fully scalable site” in the Amazon EC2 infrastructure cloud, delivering “added security” and “nearly 100 percent uptime.” The Board is projecting that this move will drive cost savings of $750,000 through FY2011 (4% of its $18 million budget) – while allowing it to reallocate more than $1 million worth of hardware and software.
• The New Jersey Transit Authority also used Salesforce.com (alongside some organizational change) to improve its customer service system. The new cloud-based processes allowed the same number of staff to handle 5 times the number of enquires (from 8354 in 2004 to 42,323 in 2006), reduced response time for enquiries by 35%, and improved productivity by 31%.
• Wisconsin’s Department of Natural Resources replaced its aging video conferencing systems with Microsoft LiveMeeting as an alternative to server-based collaboration software. Since migration in 2009, this has saved an estimated $320,000, with ROI expected to grow from 270% for the first year to over 400% in future years.
• The State of Utah uses several public cloud services (Force.com, Google Earth Pro, and Wikispaces), and has completed 70% of its private cloud project to move 1,800 physical servers in over 35 locations to a virtual platform of just 400 servers. The private cloud project alone is expected to the state save $4 million annually – over 2.5% of its $150m IT budget.
• Facing a $400 million deficit, the City of Los Angeles has been transitioning to Google Apps cloud-based e-mail, with all employees to be cut over by June 30 this year. The City’s CTO estimates a direct savings of $5.5 million over 5 years, and a total ROI (including increased productivity) of $20-30m.
  “Colorado estimates annual savings of $8m, and up to $20m in expense avoidance”
• The City of Orlando rolled out a similar Google Mail project for all 3,000 city employees in January this year. The City has realized a 65% reduction in e-mail costs, not including benefits from improved productivity, increased storage allocation (from 100MB to 25GB per user), improved security/malware detection, and enhanced mobile device support.
• The State of Colorado is shifting to a hybrid cloud model, mixing private cloud (an existing data center leveraging server virtualization), a virtual private cloud (for additional pay-as-you-go scalability), and public cloud (Google Apps for e-mail and office productivity). Just by shifting 122 servers running Lotus Notes, Microsoft Exchange, and Novell GroupWise to the cloud, Colorado estimates annual savings of $8 million, and up to $20 million in expense avoidance over 3 years.

Set SMART Goals, But Be Pragmatic

Kundra does not shy away from clearly stating his ongoing cloud computing goals in this report. By 2011, all business cases for new federal IT investment must include cloud alternatives; by 2012, all enhancements to existing systems must do the same; by 2013, all IT investments, even on legacy systems, must be justified against a cloud alternative. These SMART (Specific, Measurable, Attainable, Relevant, and Timed) goals are important to overcome the all-too-frequent adoption of disruptive technologies almost as a fad, unrelated to business goals and without a clear and realistic timeline.

However, these case studies show an essential pragmatism about the public sector approach to cloud computing. Kundra and the CIO Council recognize (as I have previously published) that the cloud will not completely replace on-premise IT, stipulating:

“Federal agencies are to deploy cloud computing solutions to improve the delivery of IT services, where the cloud computing solution has demonstrable benefits versus the status quo.”

So while cloud must be increasingly evaluated, actual cloud adoption must be justified by “demonstrable benefits” that improve IT service delivery, not just reduce costs. As I have stated in EMA research and blogged about here, it is important for enterprises (public or private) to “look for opportunities, and do what makes sense” when it comes to cloud computing. This is reflected by thought-leaders like Gartner’s Thomas Bittman (@tombitt), who explains that for some organizations “a 70% private cloud is absolutely good enough.”

Cloud Lessons For Other CIOs?

These case studies have a lot of lessons to offer other business and IT leaders, both private and public sector, in everything from mid-sized businesses to the largest enterprises. They detail many clear and realistic case studies; provide insight into achieving both specific ROI and soft benefits; show how cloud can be applied to both business- and IT-oriented goals; and give ideas for how CIOs might address real problems with cloud alternatives.

Moreover, more than any set of self-published corporate case studies, this is incredibly significant, because, as the report points out:

“The United States Government is the world’s largest consumer of information technology, spending over $76 billion annually on more than 10,000 different systems.”

This level of influence from the world’s largest consumer of IT will drive a solid and relentless march to cloud computing, a juggernaut that will likely carry the rest of us along, whether we like it or not.

However, it reads almost like promotional material from a cloud provider – which, in a way, it is – because it does not deal directly with any of the potential problems of cloud computing. It mentions security only very briefly, and then only how certain cloud implementations actually improve security (with no details). It does not give any details of how federal clouds have ensured compliance with regulations like the Federal Rules of Disclosure and DOD 5015, and industry requirements like PCI-DSS. It does not talk about if, or how, they overcame the endemic  problems of performance assurance and continuity in the cloud. Perhaps most ironically of all, it does not even mention how it overcame the tough political and departmental challenges that are cited by analysts as one of the top barriers to both virtualization and cloud adoption.

So for CIOs, this report really needs to be taken with a grain of salt. Be informed and educated by these case studies; use them to be set pragmatic expectations and SMART goals; but be wary that as much as it says about the upside of cloud computing, it avoids saying just as much – if not more – about the potential for deleterious, or even disastrous, downsides.

Is 'VM Stall' A Stop Sign for Virtualization?

There appears to be a challenger to ‘VM sprawl’ as the scourge of virtualization success – a problem I call ‘VM stall’.

We know about ‘VM sprawl’ – because new virtual machines are so easy to deploy, organizations can end up with more VMs that they can handle, or even use. This has the potential to cause severe problems to availability, performance, compliance, costs, security, and more.

However, I am seeing more and more evidence of this new phenomenon I think of as ‘VM stall’ – the tendency for virtualization deployments to stall once the ‘low-hanging fruit’ has been converted (typically around 20-30% of servers).

I think it happens more or less like this…

In general, organizations start virtualization deployments by converting relatively low-risk, low-impact systems – dev/test servers, Web servers, file servers, internal applications, etc. – to virtualization. With a big impact, great results, and reasonably fast and easy implementation, it is a great hit with IT and business owners. This may even spawn a ‘virtual first’ initiative, where all new server requests are deployed as virtual servers by default.

However, when faced with the next step, converting the remaining existing servers – including tier 1 business services, customer-facing environments, enterprise-wide systems, 3^rd-party applications, multi-platform services, and composite applications – virtualization projects often stall.

I was interested to see the notion of VM stall confirmed again last week (courtesy of eWeek via @JSchroed) in some new research into virtualization (PDF) coming out of Prism Microsystems, a software vendor in the SIEM market.*

One of the most interesting outcomes in this research was again the low penetration of server virtualization within each organization. As the chart below shows, most organizations have still virtualized less than a third of their production servers.

Source: Prism Microsystems, ‘2010 State of Virtualization Security Survey’, April 2010

What’s more, fully 15% have not even started to virtualize their production servers at all!

It might seem that this is really at odds with ‘the common wisdom’ that sees virtualization as mature, ubiquitous, commoditized, and even passé. We hear so much about virtualization, how it has been a top priority for years, about how everyone is deploying virtualization. For example:

• The IBM Global CIO Study 2009 in September showed 76% of 2500 global CIOs are undergoing or planning virtualization projects
• The Gartner 2010 CIO Survey in January reported that virtualization is the top priority for over 1500 global CIOs (up from number 3 the previous year).
• In January, CDW’s Server Virtualization Life Cycle Report (registration required) found that 90% of respondents have implemented server virtualization at some level.
• As far back as 2008, EMA research showed 75% of enterprises were using virtualization for production use cases
• The Prism Microsystems report the chart above comes from states that 85% of their sample have adopted virtualization to some degree

I am even starting to hear that virtualization is set to be irrelevant, becoming nothing more than just a stepping stone to cloud.

However, despite the widespread adoption of virtualization as a percentage of organizations, it is consistently still very low as a percentage of production servers.

Indeed, this is not the only recent (and not so recent) research study to highlight this issue. Over time, CIOs have reported a persistent difficulty in expanding their virtualization deployments beyond the initial 20-30% of servers. For example:

• Around 6 months ago, Gartner reported that “only 16 percent of workloads are running in virtual machines today.”
• Research from EMA has found that the average organization has only virtualized around 25% of servers (and only retired just 17%).
• The CDW Server Virtualization Life Cycle Report cited above showed that just 34% of the average organization’s total server infrastructure consists of virtualized servers
• CIO and HP survey in October 2009 reported that on average just 38% of mission-critical business services have been virtualized by companies with virtualization projects
• Forrester Research from May this year (conducted for CA) shows that the average enterprise has virtualized only around 30% of their servers.

At a time when so many organizations are experiencing VM sprawl, it seems hard to believe that VM stall is such an issue. Yet time and again we see that organizations find it difficult to ‘get over the hump’ of the initial 20-30% of servers, and difficult to move from low-risk/low-impact servers to high-risk/high-impact services.

If this were just a point-in-time observation, then VM stall might not exist. The low penetration rate may just be a point in the deployment cycle. However, VM stall appears to be a longitudinal effect, as it has been holding many deployments at around 20-30% of servers for several years. IIRC, something resembling VM stall was cited as an issue in EMA research as far back as 2008, and again in 2009. The CDW virtualization lifecycle research also reinforces the potential for long-term VM stall. In it, even organizations that self-report as “fully deployed” for server virtualization have only virtualized 37% of their servers. So while many organizations see VM stall as a short-term delay to virtualization rollout, many others are seeing VM stall as a permanent situation.

I see many possible causes for VM stall. For example:

• Risk aversion – high-risk, high-impact services have more stakeholders, more politics, larger and more distributed infrastructures, greater cost of failure and downtime, reduced or non-existent 3^rd-party support, and maximum management attention, among many other risk factors. The risk of failure may be too great, and the newest technology is always blamed for any new problems. Without new ways to address continuity, availability, performance, cost allocation, and other business requirements, conversion risk may be enough to stall virtualization deployment.
• Resourcing – with around 20-30% of servers converted, virtualization staffing starts to become a real challenge. As I talked about recently with my great mate, David Marshall, staff and skills shortages put a real throttle on virtualization deployments, especially as virtualization starts to scale. Not only is demand for virtualization skills still high, but supply continues to lag. Plus, the problem is getting worse, not better. Without the resources and skills to go forward, there is often little alternative to VM stall.
• Scalability – with one (typically small) team trying to manage a quarter of the entire server workload, staff from the virtualization project team simply cannot handle further virtualization deployment. In some cases, the virtualization technology itself does not scale well either; and in others, the management tools do not scale. Throwing more bodies at the problem is rarely the answer – after all, nine women cannot make a baby in one month. So organizations end up with VM stall almost by default, as they find that they need to fundamentally change their processes and technologies to enable further virtualization growth.
• Manageability – new IT management issues come up as the scale and risk of virtualization deployment increases. Enterprise virtualization needs new approaches to performance assurance, process automation, VM mobility, continuity planning, security and audit, software compliance, OEM support, configuration compliance, and more. The importance of manageability is greatly magnified  for high-risk/high-impact services, but few (if any) organizations seem to have the virtualization-aware management tools to scale to handle enterprise-class virtualization deployments. Again, VM stall happens almost by default, as IT tries to figure out enterprise-class manageability.

There may be more or different causes, but whatever the reasons, there is little doubt in my mind that VM stall exists. It is not universal – indeed, every study shows that a decent percentage of organizations are able to power through it – but for the majority of organizations, it appears to be very real. I have personally seen many enterprises going through it. More and more research continues to support it. For affected organizations, it is a significant problem, too, because stalled virtualization deployment means the highly desirable outcomes of virtualization – OpEx reduction, improved continuity, greater IT and business agility, energy cost reduction, ROI, etc. – either stalls as well, or even starts to backslide.

Whether VM stall represents as big a problem as VM sprawl, time will tell; but it is certainly a significant and growing challenge to the success of virtualization – and a fundamental driver for better virtualization management.

(EDIT: This article has been picked up and published on CIO.com! Join in the discussion there, or here.)

Is old-school physical security really 'good enough' for virtualization?

Whatever happened to virtualization security?

Back in the day, everyone was talking about blue pills and red pills, about sideways attacks and DOM-0 threats, about security profiles and isolation policies, about perimeter defense and security embedded in the hypervisor.

Then, all of a sudden, the buzz seemed to disappear. It really seems like organizations simply don’t have the time, money, desire, or otherwise to pursue dedicated virtualization security.

Indeed, it seems like most of the pure-play virtualization security vendors have folded, been sold, or reworked their strategy.

For example:

• Blue Lane ended up being sold to VMware, reputedly at a bargain price, after failing to get any traction.
• Third Brigade was rolled up into Trend Micro, and now offers a solution for combined ‘physical, virtual and cloud’ protection.
• Reflex and Catbird have repositioned to highlight their value in configuration, compliance, and/or systems management (in addition to their security value).
• Tripwire and Configuresoft have long promoted some virtualization security values, but were never really pure-play virtualization security vendors.
• Even security specialists like Symantec and RSA do not push virtualization security products, preferring mainly to build on existing security paradigms to support virtualization.

Of course VMware still has vShield Zones and the VMsafe API, but of the ISVs it seems that only Altor Networks still plays strongly in the pure-play virtualization security space.

This barely sustaining demand for pure-play virtualization security was reinforced last week in new research from Prism Microsystems (PDF), a software vendor in the SIEM market* (which I learned about in eWeek via @JSchroed). Possible vendor/sample bias aside, this research showed quite starkly how many respondents are securing their virtual environment using traditional (or no) security, and how few are using virtualization-specific security:

Source: Prism Microsystems, ‘2010 State of Virtualization Security Survey’, April 2010

In confirmation of this ennui, Gartner recently predicted at least a 5 year maturity cycle for virtualization security.

All of this is especially perplexing, because there is no doubt virtualization security is still top-of-mind for some very smart and dedicated people. The Prism Microsystems research, for example, says that 86% of its respondents acknowledge that securing virtualization is as important as securing their physical environment.

So I am unclear as to what is causing this lack of market interest. Perhaps CIOs (and/or CISOs):

• saw virtualization security as unnecessary insurance against threats that have never played out ‘in the wild’
• rated the potential financial impact of any additional risks as low enough that they can simply accept them
• believe that vShield Zones and VMsafe are all that is needed (but what about Hyper-V, Xen, etc.?)
• decided instead to invest in management disciplines with more straightforward ROI (virtualization, automation, configuration management, asset management, etc.)
• have simply been unable to justify virtualization security purchases during the economic downturn

Whatever the reason, it really does focus the question: does virtualization security really matter?

In my opinion, it absolutely does. Yet, it seems to me that decision makers are saying that standalone virtualization security is more important theoretically, from a technology and business perspective, than it is in practice. Most enterprise buyers – for better or worse – apparently believe that their existing security paradigns are at least ‘good enough’. They definitely appear instead to be taking classic intrusion detection, data loss prevention, identity & access, and other entrenched security management disciplines, and adapting them to the new technologies of virtualization (and probably cloud as well).

All of which actually does make sense. Without any major virtualization-specific exploits in evidence, perhaps they are right. While it may be valid to take the view that  it is only a matter of time until they are proven wrong, perhaps extending traditional security capabilities into the virtual world is indeed a good approach, at least for now.Perhaps CISOs are actually ahead of the game, integrating management across virtual and physical domains even while their systems and operations counterparts are maintaining virtualization as a silo.

Regardless of whether it is the right approach or not, one thing is apparent – the heat is off the pure-play virtualization security market, at least for now. As CIOs and CISOs focus on applying traditional physical security paradigms on their virtual environments, a different breed of cross-domain, integrated, and extensible tools are proving superior value – at least for now.

Many claims are made to support this so called Bring Your Own PC (BYOPC) approach – although they seem mostly, if not only, to originate from vendors (notably desktop virtualization and application virtualization vendors) that have a vested commercial interest in its success. I disagree with many of these claims (especially the questionable claims of cost reduction), but I do agree that BYOPC can have some benefits.

However, one of the many claims in support of BYOPC is that it will help organizations to attract and retain an important demographic of young, technologically sophisticated employees – the so-called ‘millennials’,  ‘echo boomers’, ‘generation next’, or ‘generation Y’, all loose terms generally used to describe people aged between 18 and 35 years old.

Personally, I find this claim to be absurd.

My main issue with this claim is the implicit assumption that millennials have such a strong choice in their employment options that issues like what type of operating system they use, or what device they work on, can be significant decision factors in whether or not to accept a job offer.

This flies in the face of unemployment statistics that suggest, today more than ever, millennials simply do not have this level of choice. Put plainly, in the near term millennials should be happy to simply get a job offer, let alone one that comes with over $2000 to buy themselves a shiny new MacBook Air.

Specifically, data from the US Bureau of Labor Statistics (see chart below) shows that in fourth quarter of 2009, unemployment for 18-35 year olds was on average 17%. For males specifically, this was on average 25%, and as high as 30%  (for males, 18-19 y.o). Compare this to the national average for ages 35 and above – 8% for the general population, and 9% for males – and you can see that this demographic does not exactly have abundant bargaining power on the job market.

Unemployment by age, sex, marital status 2006-2009 (Source: BLS)

Perhaps this will change over time, but as the chart above shows, unemployment for millennials has been trending up, not just since the onset of the recession around the end of 2008, but at least since 2006. It may (and hopefully will) come down dramatically, improving millenials’ bargaining power for employment,  but there is no sign that this is happening, or that it will happen anytime soon. And remember, when we are looking at BYOPC it is not only for technology workers,  but also (perhaps primarily) for knowledge workers across many fields – sales, finance, management, R&D, etc. – so any specific skill shortages in IT that may skew millennials’ bargaining power do not really come into play.

Of course, there are probably nuances here that a behavioural statistician would find and explain much better than I can – granular variations by  age, education, location, industry, and more. For example, unemployment among slightly older millennials aged 25 to 34,  many of whom presumably have college degrees, is much lower than millennials aged 18 to 24. Yet at over 10% unemployment, even this group still has an unemployment rate several percentage points higher than ages 35 and up (just under 8%).

However, I do not see any promoters of BYOPC incorporating such detailed demographic analysis to substantiate their claims for BYOPC; rather, most seem to just be making unsubstantiated claims using baseless assumptions about millennials’ employment ‘needs’ without even considering widely available independent data that substantially undermines their position.

Sure, there are studies that suggest, for example, that millennials consider “state-of-the-art technology is an important consideration in selecting an employer.” However, the most credible of these studies was conducted prior to the global economic downturn, when unemployment among 18-35 year olds was just 9-10% – almost half what it is today. In any case, an organization does not need a BYOPC program simply to provide state-of-the-art technology. If an employer gives a millennial employee a top-of-the-line company-owned Dell, HP, Lenovo, Sony, or Apple laptop, would they turn the job down just because they cannot buy it themselves?

So based on a cursory analysis of recent, credible, and available data, the idea that a young person would turn down a job in this economic climate, simply because they can’t get their preferred laptop or mobile device seems to me quite ridiculous. At best, it may help companies attract the very top tier of millennial graduates who do have multiple job offers to choose from, but I expect this would still be the least of the considerations of 18-35 year olds. Even for the best of them, this likely pales compared to significant concerns about compensation, vacation and holidays, health care, education support, flexible hours, corporate ethics, retirement funding, work-life balance, telecommuting, career opportunities, and more.

This then has significant implications for CIOs and others looking at BYOPC.  Despite the validity (or otherwise) of any other claims in favour of BYOPC, no organization should be looking to BYOPC to attract and retain staff from the echo boom generation. They would be better off looking at a dozen or more other important factors than spending the significant time, effort, and money on implementing a BYOPC program.

I tweeted that I think such predictions are useless when most CIOs must prove return on investment (ROI) for major IT projects in less than 6 months. Tajeshwar got me thinking more deeply about this idea with his reply:

“cio demanding roi<6 mnths r taking tactical view;3 year tech horizon must for taking strategic view & decisions”

Indeed, this is emblematic of a really interesting challenge for CIOs.

The demand for a rapid ROI, typically less than 6 months, and in some cases shorter, is a fact for today’s CIO – even more so today than before the global economic downturn. I firmly believe that CIOs demanding ROI in less than 6 months are simply realizing and reacting to the modern reality that IT can no longer be a pure cost center. Ask almost any CIO, and you will know that the ‘blue sky’ IT projects that delivered results in 2-3 year timeframes are a thing of the past.

However, as Tajeshwar implied, this demand works directly counter to the mandate for great CIOs to think and act strategically, executing on a long-term corporate vision. The same CIOs that are trying to contain or reduce costs – essentially a ‘cost center’ approach – must also be acting to make IT a strategic asset.

This does not mean that strategic CIOs are dead, or even a dying breed. On the contrary, the ability to accurately envision future trends and get a head start on competitors is perhaps more important than ever, because the rate of change in IT is so much faster, and the barriers to entry for new technology innovations seem to be always decreasing.

So to be a great CIO you need to act tactically, with projects that contain costs and deliver ROI in less than 6 months; yet also provide the business with a strategic launchpad for innovation, competitive advantage, and shareholder value.

What sort of projects can do this?

How about:

Virtualization

This is perhaps low-hanging fruit. My research for EMA clearly shows the key outcomes of virtualization are well divided between short-term ROI and long-term strategic benefits.

For example, in the short-term, virtualization reduces hardware, power, cooling, administration, rent and even software costs. Around 90% of enterprises report that it delivers real, measurable cost savings. Loading up 15 VMs or more on each physical server, allowing admins to manage on average a 10% greater workload, saving an average of $200 per system on administration costs, adding as little as $37 for each new VM in administrator staff costs (up to 28 times less than a physical system), and reducing power costs by an average of 17% are rapid and significant ROI values.

In the long-term, faster system provisioning helps bring products and services to market faster, better DR capabilities provide a strategic defense against disasters and epidemics, and better workload and resource balancing provides faster response times and better customer service – a range of strong strategic opportunities.

IT Process Automation (ITPA)

In the short-term, EMA research has shown that sites with ITPA improve their MTTR, provide almost 65 hours extra availability per year for 24×7 operations, and sites with ITPA (typically larger data centers) save on average around $500,000 more per year on staff costs alone than sites without it (easily offsetting marginally higher staff salaries). These outcomes all provide substantial short-term ROI.

Meanwhile, 95% of enterprises report that ITPA achieves one or more strategic goals, such as improving the ability to adapt to rapid change (like rapidly integrating M&A), freeing up high-level staff, providing better security and compliance, reducing business and IT complexity, reducing human errors, and integrating with best practices. Moreover, 76% report that ITPA helps achieve 2 or more of these goals, and 55% report it helps achieve 3 or more. ITPA also correlates with an overall increase in IT maturity.

Lifecycle Management

In published EMA case studies, automated lifecycle management reduced regular maintenance windows for 50 systems from 2-3 days to just 10 minutes each, and cut the cost of  distribution of a new version of Microsoft Office from $90,000 to just $30,000. It also can help to reduce overall software license costs, allocate and reuse hardware more effectively, improve end user uptime, and reduce or eliminate the (often substantial) travel, staff, and downtime costs of desk-side visits to install new software or fix problems.

EMA research also shows that automated lifecycle management helps to achieve strategic objectives. It provides faster and better service to end users (and ultimately therefore to customers), enables IT and business staff to be more productive, lets business users take advantage of new software and systems much faster, provides essential compliance reporting, and maintains strategic security values.

Part of the reason that these technologies are both tactical and strategic is that they can all be implemented in short, sharp, phases that deliver fast and specific results, while establishing a technology basis that can be leveraged – reused, over and over, in multiple new ways – to deliver strategic benefits with little or no additional cost.

For a great CIO, such technologies are invaluable. They show fast results, justifying budgets and building confidence; yet they deliver technologies they can continue to leverage for better and better strategic outcomes.

All of which meets the needs of today’s CIOs much better than blue-sky, multi-year, technology dreamings.

The survey revealed some very interesting data, with a very well thought out instrument and a quality sample – 300 respondents (100 each from the US, EMEA, and Asia Pacific) with at least 500 employees in the US (250 in the UK, France, Germany, Australia, Singapore and India), and all with a current or planned investment in server virtualization.

A number of data points stand out for me:

• The balance of Test/Dev implementations vs. Production continues to reflect EMA data. While production is still lagging behind test and dev as a use case, virtualization for mission-critical production is only slightly behind test and dev overall – and within the margin of error in most cases. This is good news, as enterprises clearly continue to grow real, production use cases.  It was interesting to see the differences between US and EMEA/APAC on this data point too, something EMA has not broken out in our published reports.
• Microsoft and VMware are neck and neck in enterprises’ plans for server virtualization deployments over the next 18 months. I was called crazy when my 2008 EMA research pointed to a 32% growth rate for Microsoft Hyper-V into 2009, trailing only VMware; yet here we are in 2009, and according to this new study, through 2010/11 that is going up to 49%. So who is crazy now?
• The strong growth for endpoint (desktop, application) virtualization reflects EMA data very well. It also highlights where enterprises and vendors should be heading with management technologies. It is still early days, but there are  a lot of gaps in integrated management for physical and virtual endpoints. So it is clear that this (probably even more than cloud service management) is going to be the next big problem for IT management.
• The percentage of IT services planned to be virtualized over 18 months is growing well. However, just as EMA has predicted, virtualization will remain at only around 50% of service deployments even through 2011, so there will continue to be substantial physical deployments. This reinforces my consistent (and insistent) position that effective management of virtualization must integrate both physical and virtual systems management
• Effective management continues to be elusive. EMA’s research showed this in 2006, 2008, and 2009, and this new data (with some reservations) shows the same. However, while tThe majority (64%) of enterprises rank themselves as extremely or very effective at managing virtualization, and believe they are getting better, I am skeptical. I contend many of those are overestimating their abilities (see my next points).
• Virtualization clearly increases complexity, and is clearly more difficult to manage. I felt like I was tilting at windmills when I published this opinion in 2006 and in 2008, contrary to common perceptions that virtualization made everything easier. I was certainly a lone voice, but as it turned out, a prescient one. It is great to see it being recognized more broadly, finally.
• Human issues continue to be major problems – especially skills and resourcing. EMA has found the same to be true, consistently, for many years. As recently as last week, I spoke with David Marshall of VMBlog and InfoWorld about how this continues to be a problem. This makes management tools even more important – to embed knowledge, define and execute policy, and automate routine work to free up resources.
• These data points all increase my doubt that enterprises are really being better at managing their virtual environments. It seems contradictory to me that this survey shows virtualization is more complex, management is the top inhibitor to ROI, and skills are still lacking, yet most enterprises think they are being very or extremely effective at it. Even though tool usage is more integrated and automated than it has been, this does not make sense. I am instead convinced that enterprises are really overestimating their abilities.

This is just a very small sample of the interesting data in this survey – there are more than 50 pages in the slide deck I reviewed ahead of the webcast. I encourage you to check out the webcast.  If you get in touch with HP, I am sure they will point you in the right direction; or check back here, and I will post the link when it is up.

Andi.