IBM Z10 Mainframe

IT loves analogies.

Seriously, will the computer-as-a-car analogy ever die (please)? It has been over 10 years since we first heard jokes about if Microsoft built cars:

At a computer expo (COMDEX) Bill Gates reportedly compared the computer industry with the auto industry and stated “If GM had kept up with technology like the computer industry has, we would all be driving twenty-five dollar cars that got 1000 miles/gallon.” Recently General Motors addressed this comment by releasing the statement : “Yeah, but would you want your car to crash twice a day?”

It has been popular ever since.

Citrix stretched the car analogy significantly last year, comparing VDI to a truck, XenDesktop (or was it XenApp?) to a Prius (or was it an SUV?), and XenServer to a Porsche (with Xen as the engine, ‘natch). This year Citrix again used some kind of car analogy, but the compact car was apparently no longer a Prius. Only a couple of months ago, Ballmer and Jobs were going after each other again, with Jobs comparing PCs to trucks, and Ballmer riffing on a questionable ‘Mac(k) truck’ analogy.

The latest and greatest example (depending on your reference point) is, of course, computing as a cloud – for many years as no more than a network icon, but mostly recently as a metaphor for a network-based on-demand computing model.

The analogy that has been bugging me recently though is virtualization (or cloud) as a ‘software mainframe’.

It was almost 18 months ago when VMware’s CEO, Paul Maritz, used the term ‘software mainframe’ at VMworld Europe. I bridled at it even then. Stephen Herrod soon followed, and both have used it periodically ever since. At Citrix’s annual Synergy event in May this year, Microsoft’s Brad Anderson used it too.

The thing is, with my experience in virtualization, cloud, and mainframe, the whole ‘software mainframe’ thing simply isn’t working for me.

Despite Maritz’s claims at the time that the analogy “proved especially useful in describing vSphere to people age 45 and over,” almost all the people I know with actual mainframe experience (both over and under 45) scoff at it. For them, even vSphere fails to live up to an actual mainframe in so many areas – uptime, throughput, manageability, security, scalability, standardization, lifespan, interoperability – the list goes on.

Meanwhile, I consistently hear most people without mainframe experience – including many CIOs, even those over 45 – want nothing to do with mainframes. “That old junk?” they say. After all, who really longs for the world of green screens, CICS and IMS, SNA/VTAM, COBOL and VSAM, transaction processing, DB2, and on and on?

I simply cannot see how the analogy is appealing for anyone. Indeed, in my experience, the message of a ‘software mainframe’ appeals to exactly no one.

In any case, VMware should really be careful what it wishes for – it may just come true. After all, if IBM ever decides to be more aggressive in its virtualization strategy, they might just enable their zSeries mainframe to run Microsoft Windows (and I for one do think they should). If they did, the real mainframe would make a very strong server virtualization option, especially for mid to large enterprises.

Remember, IBM didn’t just invent the mainframe, they invented virtualization. And if they delivered a real virtualization mainframe, you know that VMware would stop talking about mainframes pretty quickly.

And I for one would applaud, not least because I am heartily sick of the ‘software mainframe’ analogy.

More than four reasons count towards VM stall

I recently saw a great article in IT World Canada titled “Virtual stall: What it is and why you have it,” written by Jay Litkey, that took up my idea of VM stall, which I first came up with in my blog from May ‘Is “VM Stall” the Next Big Virtualization Challenge?‘.

Though they barely acknowledge my blog as their inspiration (and as a competitor to CA Technologies – my employer – why would they?), it seems Jay and his team have wholeheartedly taken up my concern with VM stall, and not just in the IT World Canada article. Marketing lead David Lynch was quoted on the topic in a post by Bruce Hoard of Virtualization Review, and in a recent Tech Target article on ‘ISV stall’. Several posts on their corporate blog also address the issue as if it was their own baby.

In my past life at EMA, I have spoken with both Jay and David a number of times, and had a lot of time for what they were doing in the management space. For a small startup with limited resources, it is great that they can take the time to pick up my idea and run with it.

The IT World Canada article is really worthwhile, because it zeroes in on some important concepts. It helps to expand the thought around VM stall, and specifically on a couple of additional causes, as it notes:

Virtual stall has four main causes:

• Scalability issues:  A single IT team often finds it difficult to scale beyond the 25-30 per cent penetration range. This is due to the combination of lack of automation and reporting in virtualization management tools, creating time-consuming manual processes that are a particular problem when there is a lack of experienced and trained staff.
• Management issues: The data centre is not a place that can be managed manually; there are too many elements to be checked, and too many independencies [sic]. And, while there are levels of automation built into the virtualization platform, they can be difficult to define and implement. The lack of automated monitoring, alerting and control becomes more and more of a problem as the overall level of virtualization in the data centre increases.
• Process issues:  Enterprise virtualization impacts a wide range of existing data centre processes, all of which need to be modified, replaced, or augmented. As long as the virtual environments are small and self-contained, these processes can be manipulated or ignored. But as the environment grows, it reaches a point when they have to be dealt with before real efficiencies can be reached. The more “process-mature” an organization is, the more quickly this point is reached.
• Co-ordination issues: Virtualization crosses multiple silos and ultimately requires a level of co-operation and integration that is impossible to achieve with the traditional silo management structure. In addition, the first workloads to be virtualized tend to be less critical ones.  However, as environments grow, higher-risk, higher-impact services are virtualized. These tend to have more stakeholders, more politics, more distributed infrastructures, and a greater cost of failure and downtime. Consequently, they require more coordination.

This is great insight, and offers a number of important causes. However, I don’t think it is reasonable to say there are just “four main causes.” Not to pick on Jay, as it is probably just unfortunate phrasing, but I think it is important to see that the issues of VM stall are much more varied, complex, and numerous.

I am not entirely without fault either. To start with, when I first identified the issue of VM stall in my blog post back in May, I said that “I see many possible causes for VM stall,” but like Jay I only identified four examples. As Jay recounts in his analysis, I saw scalability and manageability as key issues; but unlike Jay, I chose to highlight risk aversion and resourcing as two more of my examples.

However, even these six are just a part of the problem. As I said when I spoke with my great mate (and one of the industry’s great virtualization gurus, observers, and commentators), David Marshall of Hyper9 and InfoWorld in his article, “VM stall: Breaking through the second phase of virtualization“:

“… many organizations strike a ‘perfect storm’ of challenges that slows their virtualization rollout, or stops it entirely. Some causes at this stage include greater complexity of services and applications, higher demand on scarce virtualization skills, limited visibility into a growing deployment, increasingly heterogeneous systems, and greater resistance from risk-averse application owners and recalcitrant application vendors.”

In the same article, David spoke with Dave Bartoletti, formerly of automation vendor Enigmatec and now a leading light showing the way through the virtualization darkness with research and advisory analyst firm, the Taneja Group:

“The second wave of issues is always harder when a core technology matures. Server virtualization essentially paid for itself in CAPEX savings, but when we virtualize Tier 1 business-critical applications, or user desktops, CAPEX savings take a backseat to application performance and IT efficiency, and this is why we’re stalling.”

My former editor at Tech Target and another keen virtualization observer, Colin Steele, highlighted another core element of VM stall, in his article “ISV stall makes virtualizing applications a challenge“:

By now, the benefits of virtualizing applications are clear, but the goal of 100% virtualization remains elusive. One reason is that some independent software vendors (ISVs) don’t support their server-based applications — databases, telecom apps, healthcare programs, etc. — on virtual servers.

Moreover, I talk a lot with customers about their real world concerns, so I can quickly pinpoint many other causes. They talk to me about issues like vendor licensing, facilities constraints, capacity blindness, service prioritization, deployment costs, line-of-business resistance, internal politics, a lack of skills, and even senior management resistance.

In fact, last week at CA Expo in Australia, I talked with CA Technologies customers about seven significant issues in virtualization that are contributing to (among other things) VM stall, as you can see from one of the slides from my presentation:

Virtualization is not clear sailing - from CA Expo Australia

(You can see the whole deck at the CA Expo site)

To be fair to Jay and his team, other posts on his corporate blog agree with me, citing issues like mission-critical apps, management skepticism, bureaucracy, poor project vetting, and more.

I am really glad to see my thoughts around VM stall have captured the imagination of the market. Thanks to Jay for taking this up, and to his team for joining me and CA Technologies in raising awareness of issues causing VM stall.

However, I think we all need to be careful about being categorical about VM stall. It is important to be clear that VM stall – like most enterprise IT issues, and indeed most organizations – is both complex and varied, so trying to categorically define four (or six, or seven, or really any number) of causes for VM stall is underestimating this important problem.

But if we can all contribute new ideas to the community, we will all learn more, and our enterprise customers will benefit from our combined wisdom.

What are the best steps to get on the 'red carpet' with virtualization?

Last week I spoke with Pam Baker, a writer with CIO Update, for an article titled The Top 5 Places to Use Virtualization. As you would expect from an experienced professional like Pam, it was a great article, with solid contributions from many others as well.

Pam specifically asked me to provide readers with advice on how to move into production with virtualization, and following our discussion published her article, including this section on ‘Low Risk Services’:

Move the easy stuff — Web servers, print servers, file servers, single-system applications, etc. — first. “Co-locating these environments on virtual machines delivers quick wins in business continuity, agility, resource efficiency, and of course cost savings — both cap-ex and op-ex,” explains Andi Mann, vice president of Product Marketing at CA Technologies Virtualization and Service Automation Business Unit. Moving low-risk services such as HR systems — file servers and Intranet applications, for example, but not payroll or e-mail — onto virtual machines is “a great next step into production virtualization.

However, I wanted to complete the thoughts I had while speaking with Pam, and address some of the other phases of virtualization deployment that we discussed.

What is the first service you should consider using virtualization?

Without doubt, application development is the very first place you should use virtualization. Dev/test – including unit test, system test, quality assurance, and user acceptance – is a great opportunity for virtualization because it is:

• Low-impact – it never touches a customer or even an internal user directly, and so even if you make ‘rookie’ mistakes they cannot damage customer service.
• High-reward – it allows applications to be developed, tested, and delivered both faster and cheaper, driving both agility and cost savings.

Plus, developers are already tech-savvy, so they can learn and deal with virtualization quickly and easily.

This is also a very strategic way to start, with a long tail of positive results. Applications developed on virtual servers can easily be deployed into production on virtual servers too. This gives you an easy route to production, with all the cost, continuity, and availability benefits that delivers.

At this stage you can also start to implement a ‘virtual-first’ policy for new applications – where every new service is deployed on virtual servers unless there is a clear business case – along with authorization, and even additional chargeback penalties – for requesting a physical server.

With this level of experience under your belt, you and your teams will quickly gain a broad, production-quality baseline of skills, knowledge, and ability to handle virtualization, while avoiding negative business impact as you acquire these capabilities in your teams.

This then establishes a solid ‘base camp’ to launch the next phase of virtualization – attacking existing production applications.

How can you move virtualization beyond the initial deployment?

Once you institutionalize virtualization in dev/test, and subsequent production deployments of new applications, as Pam noted in her article, you should look at moving existing low-risk/low-impact production services onto virtual servers next.

As I discussed with Pam, that will often mean virtualizing internal services, like your HR systems, file servers, or Intranet applications. However, just because they are internal systems, does not mean they are low-risk, or low-impact.  That is why I said you should probably leave payroll and e-mail alone in this phase – they are both not only high-risk, but also high-impact if anything fails.

Converting and migrating these low-risk, internal systems establishes another, higher-level  ‘base camp’ from which to expand your virtualization deployments. You can move to a broader virtualization deployment with greater confidence and lower risk, because you have the deeper experience.

Moreover, you have proven to the business how virtualization delivers incremental and substantial gains in CapEx reduction, OpEx reduction, agility, continuity, and time-to-market.

From there, you can then move into more complex, external-facing, mission-critical applications and services.

What are the best uses for virtualization?

Almost everything is a good use case for virtualization! Most organizations should be able to get 80-90% of their server workloads onto virtual machines – far more than the 16% of workloads that analyst firm Gartner says is running in virtual servers today.

The ‘low-hanging fruit’ of virtualization is, as Pam wrote, the “easy stuff” like Web servers, print servers, file servers, and simple, single-system applications. Co-locating these environments on virtual machines delivers quick wins in business continuity, agility, resource efficiency, and of course cost savings – both CapEx and OpEx.

Similarly, it is relatively easy to get new applications onto virtualization, by starting in development and test, and by implementing a ‘virtual first’ policy for new applications.

But even most of the ‘difficult’ applications – mission-critical, tier 1, OLTP, multi-tier, complex composite applications, etc. – can be virtualized with the right approach. These applications will benefit greatly from the improvements to scalability, continuity, performance, and resource efficiency that virtualization delivers.

What are the worst use cases for virtualization?

While it is true that almost all services can and should be virtualized, it is also true that some services are not well suited for a traditional, multi-VM, shared-server virtualization deployment.

The worst use cases for virtualization are where application services saturate one or more physical resources. If, for example, an application regularly uses over 90% of available CPU, memory, or network bandwidth, then there is no headroom left over for another system or service to use these resources. This means that it is not a good option to co-locate this application on a virtual server that shares physical resources with another application.

Typically such services include:

• CPU intensive applications – such as actuarial, modeling, design, or engineering applications
• Memory intensive services – such as database systems, data mining, or business intelligence
• Network intensive services – such as transaction processing or multi-user applications

Some services – such as corporate e-mail servers – may actually be all three.

However, you should never discount the benefits of deploying any application in a virtual server, even if it is deployed all by itself. Even if it will not provide hard ROI through hardware reduction, you can still gain major benefits in improvements to availability and continuity, operational costs, and ease of maintenance, by using virtualization.

How Did You Virtualize?

These are some ways to start with virtualization, some ways to expand virtualization, and some areas that you should probably leave until late in the cycle(if you virtualize them at all).

But I wonder where you started (or where you plan to start)? How did you expand beyond the low-hanging fruit? What types of services have you avoided virtualizing? Why?

Feel free to add your comments below. I would love to hear about your experiences.

(This entry has also been posted as an entry at CA.com – feel free to discuss there, or here)

Federal CIO Vivek Kundra and the CIO Council

If there was still any doubt about the real world use cases for cloud computing, the US Federal Government last week published a 38-page report  entitled “State of Public Sector Cloud Computing” (link to PDF at CIO.gov). Attributed to the Federal CIO Vivek Kundra, it is stamped with the seal/logo of the CIO Council, which comprises the CIOs of some 28 federal government agencies.

The report details 30 case studies in public sector cloud computing (for both state and federal governments), covering IaaS, PaaS, and SaaS service models; using private, public, community, and hybrid cloud deployment models; with both on-premise and off-premise implementations.

Measurable Benefits from Key Case Studies

After perfunctorily reciting what it calls “the broadly recognized and adopted NIST Definition of Cloud Computing,” and using the opportunity to briefly push its own barrow on cloud standards (a subject I plan to blog about in more detail at another time), the report cites several projects with ‘soft’ outcomes – improved productivity, better efficiency, higher reliability – as well as several planned cloud projects that are yet to bear fruit.

However, most of the report is given over to demonstrating solid and measurable outcomes from over a dozen current cloud deployment case studies involving multiple state and federal government agencies, with cloud success stories such as:

• The US Army is piloting a customized version of Salesforce.com to update its 10 year old recruiting systems for Web 2.0, social media, mobile devices, marketing integration, real-time data interchange, and engagement tracking. At an annual cost of $54,000, this pilot compares to bids from traditional IT vendors ranging from $500K to over $1 million, and has already replaced five traditional recruiting centers.
• The Department of Health and Human Services is also using Salesforce.com to support the implementation of Electronic Health Records systems. This new CRM system for working with participating healthcare providers was deployed in just 3 months, instead of the full year estimated for an internally delivered system.
• The General Services Administration (GSA) moved to a Terremark Enterprise Cloud service, to take advantage of on-demand scalability for Web sites like USA.gov. As a result, GSA accelerated its site upgrade time from nine months to a maximum of one day, reduced monthly downtime from roughly two hours to near zero (99.9% availability), and reduced annual costs for USA.gov by $1.7 million, from $2.35 million to $650,000, or 72%.
• The Defense Information Systems Agency (DISA) is using virtualization with a self-service portal to provide on-demand server space for development teams. With just an approved Government credit card, these end users can set up new environments (with DoD-compliant security guaranteed) in just 24 hours – down from three to six weeks – and at a “reasonable” cost.
“DISA estimates PaaS cloud savings between $200,000 and $500,000 per project.”
• DISA also used cloud provider CollabNet to set up Forge.mil, a private PaaS cloud development environment with a heavy focus on collaboration and code sharing/reuse. DISA estimates this saves between $200,000 and $500,000 per project – not including the estimated $15 million in cost avoidance by utilizing an open source philosophy.
• The Lawrence Berkeley National Labs (LBL), part of the Dept of Energy, is using Google Apps for 2,300 e-mail users, and planning to more than double that by August. LBL estimates they will save $1.5 million over five years “in hardware, software and labor costs from the deployments they have already made.”
• NASA’s Jet Propulsion Laboratory used a Microsoft Azure development platform “to excite the public about Mars” with the website, BeAMartian.jpl.nasa.gov. This site has generated over 2,000 pieces of social media, inspired 200 traditional media stories, responded up 2.5 million API queries, gathered  40,000 votes in its ‘Town Hall’ polls, and attracted 5,000 registrations from individuals and teams.
• The Federal Labor Relations Authority recently replaced its underperforming, decade-old case management system, switching to Intuit’s Quickbase system. As a result, it was able to go from requirements-definition to completed development in 10 months – a quarter of the original deployment time – and expects a TCO reduction of nearly $600,000 over five years.
“Moving Recovery.gov to Amazon EC2 will drive cost savings of $750,000”
• Less than a month ago, the Recovery Accountability and Transparency Board moved Recovery.gov to a “fully scalable site” in the Amazon EC2 infrastructure cloud, delivering “added security” and “nearly 100 percent uptime.” The Board is projecting that this move will drive cost savings of $750,000 through FY2011 (4% of its $18 million budget) – while allowing it to reallocate more than $1 million worth of hardware and software.
• The New Jersey Transit Authority also used Salesforce.com (alongside some organizational change) to improve its customer service system. The new cloud-based processes allowed the same number of staff to handle 5 times the number of enquires (from 8354 in 2004 to 42,323 in 2006), reduced response time for enquiries by 35%, and improved productivity by 31%.
• Wisconsin’s Department of Natural Resources replaced its aging video conferencing systems with Microsoft LiveMeeting as an alternative to server-based collaboration software. Since migration in 2009, this has saved an estimated $320,000, with ROI expected to grow from 270% for the first year to over 400% in future years.
• The State of Utah uses several public cloud services (Force.com, Google Earth Pro, and Wikispaces), and has completed 70% of its private cloud project to move 1,800 physical servers in over 35 locations to a virtual platform of just 400 servers. The private cloud project alone is expected to the state save $4 million annually – over 2.5% of its $150m IT budget.
• Facing a $400 million deficit, the City of Los Angeles has been transitioning to Google Apps cloud-based e-mail, with all employees to be cut over by June 30 this year. The City’s CTO estimates a direct savings of $5.5 million over 5 years, and a total ROI (including increased productivity) of $20-30m.
  “Colorado estimates annual savings of $8m, and up to $20m in expense avoidance”
• The City of Orlando rolled out a similar Google Mail project for all 3,000 city employees in January this year. The City has realized a 65% reduction in e-mail costs, not including benefits from improved productivity, increased storage allocation (from 100MB to 25GB per user), improved security/malware detection, and enhanced mobile device support.
• The State of Colorado is shifting to a hybrid cloud model, mixing private cloud (an existing data center leveraging server virtualization), a virtual private cloud (for additional pay-as-you-go scalability), and public cloud (Google Apps for e-mail and office productivity). Just by shifting 122 servers running Lotus Notes, Microsoft Exchange, and Novell GroupWise to the cloud, Colorado estimates annual savings of $8 million, and up to $20 million in expense avoidance over 3 years.

Set SMART Goals, But Be Pragmatic

Kundra does not shy away from clearly stating his ongoing cloud computing goals in this report. By 2011, all business cases for new federal IT investment must include cloud alternatives; by 2012, all enhancements to existing systems must do the same; by 2013, all IT investments, even on legacy systems, must be justified against a cloud alternative. These SMART (Specific, Measurable, Attainable, Relevant, and Timed) goals are important to overcome the all-too-frequent adoption of disruptive technologies almost as a fad, unrelated to business goals and without a clear and realistic timeline.

However, these case studies show an essential pragmatism about the public sector approach to cloud computing. Kundra and the CIO Council recognize (as I have previously published) that the cloud will not completely replace on-premise IT, stipulating:

“Federal agencies are to deploy cloud computing solutions to improve the delivery of IT services, where the cloud computing solution has demonstrable benefits versus the status quo.”

So while cloud must be increasingly evaluated, actual cloud adoption must be justified by “demonstrable benefits” that improve IT service delivery, not just reduce costs. As I have stated in EMA research and blogged about here, it is important for enterprises (public or private) to “look for opportunities, and do what makes sense” when it comes to cloud computing. This is reflected by thought-leaders like Gartner’s Thomas Bittman (@tombitt), who explains that for some organizations “a 70% private cloud is absolutely good enough.”

Cloud Lessons For Other CIOs?

These case studies have a lot of lessons to offer other business and IT leaders, both private and public sector, in everything from mid-sized businesses to the largest enterprises. They detail many clear and realistic case studies; provide insight into achieving both specific ROI and soft benefits; show how cloud can be applied to both business- and IT-oriented goals; and give ideas for how CIOs might address real problems with cloud alternatives.

Moreover, more than any set of self-published corporate case studies, this is incredibly significant, because, as the report points out:

“The United States Government is the world’s largest consumer of information technology, spending over $76 billion annually on more than 10,000 different systems.”

This level of influence from the world’s largest consumer of IT will drive a solid and relentless march to cloud computing, a juggernaut that will likely carry the rest of us along, whether we like it or not.

However, it reads almost like promotional material from a cloud provider – which, in a way, it is – because it does not deal directly with any of the potential problems of cloud computing. It mentions security only very briefly, and then only how certain cloud implementations actually improve security (with no details). It does not give any details of how federal clouds have ensured compliance with regulations like the Federal Rules of Disclosure and DOD 5015, and industry requirements like PCI-DSS. It does not talk about if, or how, they overcame the endemic  problems of performance assurance and continuity in the cloud. Perhaps most ironically of all, it does not even mention how it overcame the tough political and departmental challenges that are cited by analysts as one of the top barriers to both virtualization and cloud adoption.

So for CIOs, this report really needs to be taken with a grain of salt. Be informed and educated by these case studies; use them to be set pragmatic expectations and SMART goals; but be wary that as much as it says about the upside of cloud computing, it avoids saying just as much – if not more – about the potential for deleterious, or even disastrous, downsides.

Is 'VM Stall' A Stop Sign for Virtualization?

There appears to be a challenger to ‘VM sprawl’ as the scourge of virtualization success – a problem I call ‘VM stall’.

We know about ‘VM sprawl’ – because new virtual machines are so easy to deploy, organizations can end up with more VMs that they can handle, or even use. This has the potential to cause severe problems to availability, performance, compliance, costs, security, and more.

However, I am seeing more and more evidence of this new phenomenon I think of as ‘VM stall’ – the tendency for virtualization deployments to stall once the ‘low-hanging fruit’ has been converted (typically around 20-30% of servers).

I think it happens more or less like this…

In general, organizations start virtualization deployments by converting relatively low-risk, low-impact systems – dev/test servers, Web servers, file servers, internal applications, etc. – to virtualization. With a big impact, great results, and reasonably fast and easy implementation, it is a great hit with IT and business owners. This may even spawn a ‘virtual first’ initiative, where all new server requests are deployed as virtual servers by default.

However, when faced with the next step, converting the remaining existing servers – including tier 1 business services, customer-facing environments, enterprise-wide systems, 3^rd-party applications, multi-platform services, and composite applications – virtualization projects often stall.

I was interested to see the notion of VM stall confirmed again last week (courtesy of eWeek via @JSchroed) in some new research into virtualization (PDF) coming out of Prism Microsystems, a software vendor in the SIEM market.*

One of the most interesting outcomes in this research was again the low penetration of server virtualization within each organization. As the chart below shows, most organizations have still virtualized less than a third of their production servers.

Source: Prism Microsystems, ‘2010 State of Virtualization Security Survey’, April 2010

What’s more, fully 15% have not even started to virtualize their production servers at all!

It might seem that this is really at odds with ‘the common wisdom’ that sees virtualization as mature, ubiquitous, commoditized, and even passé. We hear so much about virtualization, how it has been a top priority for years, about how everyone is deploying virtualization. For example:

• The IBM Global CIO Study 2009 in September showed 76% of 2500 global CIOs are undergoing or planning virtualization projects
• The Gartner 2010 CIO Survey in January reported that virtualization is the top priority for over 1500 global CIOs (up from number 3 the previous year).
• In January, CDW’s Server Virtualization Life Cycle Report (registration required) found that 90% of respondents have implemented server virtualization at some level.
• As far back as 2008, EMA research showed 75% of enterprises were using virtualization for production use cases
• The Prism Microsystems report the chart above comes from states that 85% of their sample have adopted virtualization to some degree

I am even starting to hear that virtualization is set to be irrelevant, becoming nothing more than just a stepping stone to cloud.

However, despite the widespread adoption of virtualization as a percentage of organizations, it is consistently still very low as a percentage of production servers.

Indeed, this is not the only recent (and not so recent) research study to highlight this issue. Over time, CIOs have reported a persistent difficulty in expanding their virtualization deployments beyond the initial 20-30% of servers. For example:

• Around 6 months ago, Gartner reported that “only 16 percent of workloads are running in virtual machines today.”
• Research from EMA has found that the average organization has only virtualized around 25% of servers (and only retired just 17%).
• The CDW Server Virtualization Life Cycle Report cited above showed that just 34% of the average organization’s total server infrastructure consists of virtualized servers
• CIO and HP survey in October 2009 reported that on average just 38% of mission-critical business services have been virtualized by companies with virtualization projects
• Forrester Research from May this year (conducted for CA) shows that the average enterprise has virtualized only around 30% of their servers.

At a time when so many organizations are experiencing VM sprawl, it seems hard to believe that VM stall is such an issue. Yet time and again we see that organizations find it difficult to ‘get over the hump’ of the initial 20-30% of servers, and difficult to move from low-risk/low-impact servers to high-risk/high-impact services.

If this were just a point-in-time observation, then VM stall might not exist. The low penetration rate may just be a point in the deployment cycle. However, VM stall appears to be a longitudinal effect, as it has been holding many deployments at around 20-30% of servers for several years. IIRC, something resembling VM stall was cited as an issue in EMA research as far back as 2008, and again in 2009. The CDW virtualization lifecycle research also reinforces the potential for long-term VM stall. In it, even organizations that self-report as “fully deployed” for server virtualization have only virtualized 37% of their servers. So while many organizations see VM stall as a short-term delay to virtualization rollout, many others are seeing VM stall as a permanent situation.

I see many possible causes for VM stall. For example:

• Risk aversion – high-risk, high-impact services have more stakeholders, more politics, larger and more distributed infrastructures, greater cost of failure and downtime, reduced or non-existent 3^rd-party support, and maximum management attention, among many other risk factors. The risk of failure may be too great, and the newest technology is always blamed for any new problems. Without new ways to address continuity, availability, performance, cost allocation, and other business requirements, conversion risk may be enough to stall virtualization deployment.
• Resourcing – with around 20-30% of servers converted, virtualization staffing starts to become a real challenge. As I talked about recently with my great mate, David Marshall, staff and skills shortages put a real throttle on virtualization deployments, especially as virtualization starts to scale. Not only is demand for virtualization skills still high, but supply continues to lag. Plus, the problem is getting worse, not better. Without the resources and skills to go forward, there is often little alternative to VM stall.
• Scalability – with one (typically small) team trying to manage a quarter of the entire server workload, staff from the virtualization project team simply cannot handle further virtualization deployment. In some cases, the virtualization technology itself does not scale well either; and in others, the management tools do not scale. Throwing more bodies at the problem is rarely the answer – after all, nine women cannot make a baby in one month. So organizations end up with VM stall almost by default, as they find that they need to fundamentally change their processes and technologies to enable further virtualization growth.
• Manageability – new IT management issues come up as the scale and risk of virtualization deployment increases. Enterprise virtualization needs new approaches to performance assurance, process automation, VM mobility, continuity planning, security and audit, software compliance, OEM support, configuration compliance, and more. The importance of manageability is greatly magnified  for high-risk/high-impact services, but few (if any) organizations seem to have the virtualization-aware management tools to scale to handle enterprise-class virtualization deployments. Again, VM stall happens almost by default, as IT tries to figure out enterprise-class manageability.

There may be more or different causes, but whatever the reasons, there is little doubt in my mind that VM stall exists. It is not universal – indeed, every study shows that a decent percentage of organizations are able to power through it – but for the majority of organizations, it appears to be very real. I have personally seen many enterprises going through it. More and more research continues to support it. For affected organizations, it is a significant problem, too, because stalled virtualization deployment means the highly desirable outcomes of virtualization – OpEx reduction, improved continuity, greater IT and business agility, energy cost reduction, ROI, etc. – either stalls as well, or even starts to backslide.

Whether VM stall represents as big a problem as VM sprawl, time will tell; but it is certainly a significant and growing challenge to the success of virtualization – and a fundamental driver for better virtualization management.

(EDIT: This article has been picked up and published on CIO.com! Join in the discussion there, or here.)

Is old-school physical security really 'good enough' for virtualization?

Whatever happened to virtualization security?

Back in the day, everyone was talking about blue pills and red pills, about sideways attacks and DOM-0 threats, about security profiles and isolation policies, about perimeter defense and security embedded in the hypervisor.

Then, all of a sudden, the buzz seemed to disappear. It really seems like organizations simply don’t have the time, money, desire, or otherwise to pursue dedicated virtualization security.

Indeed, it seems like most of the pure-play virtualization security vendors have folded, been sold, or reworked their strategy.

For example:

• Blue Lane ended up being sold to VMware, reputedly at a bargain price, after failing to get any traction.
• Third Brigade was rolled up into Trend Micro, and now offers a solution for combined ‘physical, virtual and cloud’ protection.
• Reflex and Catbird have repositioned to highlight their value in configuration, compliance, and/or systems management (in addition to their security value).
• Tripwire and Configuresoft have long promoted some virtualization security values, but were never really pure-play virtualization security vendors.
• Even security specialists like Symantec and RSA do not push virtualization security products, preferring mainly to build on existing security paradigms to support virtualization.

Of course VMware still has vShield Zones and the VMsafe API, but of the ISVs it seems that only Altor Networks still plays strongly in the pure-play virtualization security space.

This barely sustaining demand for pure-play virtualization security was reinforced last week in new research from Prism Microsystems (PDF), a software vendor in the SIEM market* (which I learned about in eWeek via @JSchroed). Possible vendor/sample bias aside, this research showed quite starkly how many respondents are securing their virtual environment using traditional (or no) security, and how few are using virtualization-specific security:

Source: Prism Microsystems, ‘2010 State of Virtualization Security Survey’, April 2010

In confirmation of this ennui, Gartner recently predicted at least a 5 year maturity cycle for virtualization security.

All of this is especially perplexing, because there is no doubt virtualization security is still top-of-mind for some very smart and dedicated people. The Prism Microsystems research, for example, says that 86% of its respondents acknowledge that securing virtualization is as important as securing their physical environment.

So I am unclear as to what is causing this lack of market interest. Perhaps CIOs (and/or CISOs):

• saw virtualization security as unnecessary insurance against threats that have never played out ‘in the wild’
• rated the potential financial impact of any additional risks as low enough that they can simply accept them
• believe that vShield Zones and VMsafe are all that is needed (but what about Hyper-V, Xen, etc.?)
• decided instead to invest in management disciplines with more straightforward ROI (virtualization, automation, configuration management, asset management, etc.)
• have simply been unable to justify virtualization security purchases during the economic downturn

Whatever the reason, it really does focus the question: does virtualization security really matter?

In my opinion, it absolutely does. Yet, it seems to me that decision makers are saying that standalone virtualization security is more important theoretically, from a technology and business perspective, than it is in practice. Most enterprise buyers – for better or worse – apparently believe that their existing security paradigns are at least ‘good enough’. They definitely appear instead to be taking classic intrusion detection, data loss prevention, identity & access, and other entrenched security management disciplines, and adapting them to the new technologies of virtualization (and probably cloud as well).

All of which actually does make sense. Without any major virtualization-specific exploits in evidence, perhaps they are right. While it may be valid to take the view that  it is only a matter of time until they are proven wrong, perhaps extending traditional security capabilities into the virtual world is indeed a good approach, at least for now.Perhaps CISOs are actually ahead of the game, integrating management across virtual and physical domains even while their systems and operations counterparts are maintaining virtualization as a silo.

Regardless of whether it is the right approach or not, one thing is apparent – the heat is off the pure-play virtualization security market, at least for now. As CIOs and CISOs focus on applying traditional physical security paradigms on their virtual environments, a different breed of cross-domain, integrated, and extensible tools are proving superior value – at least for now.

Most devops discussions are missing the target

I am hearing a lot about the rise of a concept called ‘devops’ – a mashup of ‘development’ and ‘operations’. I am not at all an expert in this area, but from what I can tell, devops is aimed at streamlining rapidly iterative application delivery to allow for greater development and business agility. Devops aims to achieve this by breaking down the barriers – human, process, and technology – between application development and system operations.

Interestingly, the concept is new enough that, as I write this, there is not even an entry for it in Wikipedia yet. I did find a blog by Damon Edwards (on Twitter – @damonedwards) very useful though, as he explains the age-old disconnects between application developers ‘throwing software over the wall’, and ops who are painfully resistant to change. James Urquhart (@jamesurquhart ) blogged very recently on the concept too , and again provided some very helpful content. Conversing online with them and others also helped me to formulate some more concrete ideas about devops – or at least some more concrete questions.

My interest was especially piqued when I understood how closely devops is connected to virtualization, cloud, and automation – my core interests:

• Cloud – Devops has antecedents in ‘rogue’ developers (or developers from smaller shops) using cloud resources (IaaS, PaaS) for new projects, and will benefit greatly from cloud-based development and deployment, as cloud providers do not impose the restrictions of internal change-averse ops teams, and developers can essentially manage their own ops requirements instead.
• Virtualization – In-house devops (which needs more heavy lifting) is greatly assisted by virtualization, as virtual machines become the new base unit for application packaging, avoiding application rollout failures caused by incompatibility between the test and production environments (hardware, OS, middleware, etc.).
• Automation – In-house devops is also greatly facilitated by automation, which can use standard workflows to automatically provision and configure these complete application VMs, as well as backup and restore VMs, allowing complex composite application deployment and rollback at the click of a mouse.

Clearly devops has many very attractive outcomes – drive agile business, reduce delays, smooth application releases, deliver value faster. It is a very seductive idea. Who wouldn’t want it?

However, most of the writings I see about devops are really about dev, not ops. As a result, they don’t really capture the whole story of the application lifecycle. They justify devops as an antidote to the problems that ops are causing – slowing down release cycles, imposing arbitrary rules, screwing up deployments, killing developer productivity, hacking manual scripts and configs, stopping the business from being agile – but fail to recognize both the failings of developers that contribute to the problems, and the role of operations in delivering critical business outcomes during the application delivery lifecycle.

On the contrary, discussions mainly focus on how developers can sideline or change operations, positioning devops as the lone hero in the battle against inefficiency, as application developers fix all the problems (!) by controlling or automating key release management operations like provisioning, deployment, integration, patching, and software update. Meanwhile, ops are marginalised, along with their timesinks and roadblocks, satisfying the needs of an agile and rapidly changing business.

See – seductive, isn’t it?

Yet this seems to me (as a former op) fundamentally flawed, a development-centric neologism based on an incomplete understanding of the real purpose and role of IT operations, or of operations’ history in the development of ‘agile’ IT.

The way I see it, devops misses that target on how IT ops serve business needs too, and seems to gloss over ‘coal face’ realities like:

• Who handles ongoing support, especially software update for the unrestrained sprawl of non-standard systems and components.
• Who ensures each new application doesn’t interfere with existing and especially legacy systems (and networks, storage, etc.)?
• Who handles integration with common production systems that cannot be encapsulated in a VM, like storage arrays (NAS, SAN), networking fabrics, facilities, etc.
• Who handles impact analysis, change control and rollback planning to ensure deployment risk is understood and mitigated?
• Who is responsible for cost containment and asset rationalization, when devops keeps rolling out new systems and applications?
• Who ensures reporting, compliance, data updates, log maintenance, Db administration, etc. are built into the applications, and integrated with standard management tools?
• Who will assure functional isolation, role-based access controls, change auditing, event management, and configuration control to secure applications, data, and compliance?

Because, if you have ever worked with both ops and apps, you know it is not going to be apps.

Now, in defence of devops, I am sure it is being implemented and conferring major benefits, especially in small organizations with little IT management discipline. I am sure the supporters of devops have some positive goals in mind too. What’s more, it is addressing a very real problem – ops really should spend more time on better processes and controls than in ‘daily deployment muck’.

However, devops should be a two-way street. As a former op, I know that the apps team have to pull their weight too, by addressing gaps like:

• Including ops during the design process, so applications are built to work with standard ops tools
• Taking ops input on deployment, so applications will go in cleanly without disrupting other users
• Working with ops on capacity and scalability requirements, so they can keep supporting it when it grows
• Implementing ops’ critical needs for logging, isolation, identity management, configuration needs, and secure interfaces so the app can be secure and compliant
• Giving ops some advance insight into applications, especially during test and QA, so they can start to prepare for them before they come over the wall
• Allowing ops to contribute to better application design, deployment, and management; that ops can do more for the release cycle and ongoing management than just ‘manipulating APIs’

See, ops do enable business – and agile business at that – by ensuring that new applications coming into an existing complex environment are safe, secure, reliable, integrated, and responsive, regardless of how complex IT is, or how many moving parts there are. Devops seems to miss this important detail.

So I am sceptical of how devops will work in large, well-run IT environments with important and necessary operational controls, especially the > 60% of organizations that are committed to ITIL best practices (like formal and integrated management of change, configuration, release, assets, etc.).

After all, ‘agile’ does not magically obviate the need to identify and prevent bad changes, to reject apps that breach operational compliance, to ensure each new application adheres to standards, or to prevent uncontrolled sprawl of heterogeneous software.

I still have a lot to think about on this topic, and am trying to keep an open mind. But my best guess right now is that, for enterprises at least, devops either will not take hold or will not last. It seems most likely to be instead, at best, a transitory state on the path to a ‘new normal’. As with all ‘revolutions’, it has started outside IT ops, yet I expect will eventually co-opt and migrate wholly to operations in some form. Once the revolutionaries in development understand how many business needs besides agility actually require  routine, process, management, and controls, they will back away from devops the same way they backed away from ownership in other IT revolutions – like the deployment of mini computers, desktops, and web applications.

If it does turn out this way – don’t worry. Operations will again dutifully take the reins, and clean up the mess that devops will leave behind. Because that is what ops do – they manage what they are given, and keep the business running, regardless of the mess that gets thrown over the wall at them.

In any case, whether devops takes root or not, hopefully we will all learn something about cooperation, automation, agility, and control. Because all stakeholders in the devops discussion – development, operations, and business owners – could benefit from that.

Effort vs. Payback is an Everyday Business IT Decision

I read recently a good blog post from Thomas Bittman (@tombitt) of Gartner Group, about how sometimes close enough is good enough. Talking specifically about private cloud, he talked about how an ‘imperfect’ cloud deployment – one that does not have all five essential characteristics, for example – might be enough for some organizations.

I especially appreciated how he highlighted some very specific, real-world examples to sustain his advice. As he shows, sometimes you don’t need a ‘100%’ implementation, and for very good business reasons.

Not every IT organization needs a fully self-service interface, and many smaller organizations see no value in usage metering. They simply want to deliver services faster. For them, a 70% private cloud is absolutely good enough … it all comes down to business requirements, return on investment, and future strategy. How far you go is your decision.

via Driving for Imperfection With Your Private Cloud.

If you haven’t seen it yet, you should. It’s a quick read, only 4 paragraphs and less than 300 words. Go ahead. I’ll still be here when you get back.

The theme is very similar to something I wrote in a research report for EMA, ‘The Responsible Cloud‘, also on cloud computing. Regarding the NIST definition of cloud, I cautioned against dogmatic interpretations of cloud computing, and the notion that a ‘real’ cloud must necessarily have all of the essential characteristics, or fit some specific deployment model. Flexibility is key, I advised, and delivering on key business requirements is more important than definitions.

Two other things happened this week that made me think about this in different ways:

• An internal session at CA reviewing some customer-facing materials. All attendees agreed – we can’t preach unattainable dogma; we need to deal with specific requirements and partial deployments, as well as broad requirements that come from  ‘100%’ implementations.
• A group discussion on LinkedIn, where an IT practitioner wanted advice on building a small private cloud. He was soon inundated with an unrealistic list of requirements, from hypervisor features to management disciplines, that he *must* have to build a ‘100%’ cloud.

The similar inferences in three otherwise unrelated conversations started me thinking more broadly about ‘100% adoption’. It IT, as in life, you never really need a Rolls Royce. You can aspire to the quality, appreciate its refinement, and in some cases you may be fortunate enough to actually enjoy it, but there is a point where it simply doesn’t make sense to pursue that level of luxury. Mostly you can get away with a Ford. Sometimes you can even make do with a second-hand Lada.

The same Pareto-like principle applies roughly throughout IT (much to the annoyance of just about every security pro I have ever met) – although the actual ratio may vary wildly, you can often get most of the benefit from less than a ‘100%’ implementation.

The phrase that sprang to mind for me was the same conclusion that I published elsewhere in the Responsible Cloud report, and the same notion that many IT pros live by, day in and day out:

It is important to look for opportunities, and do what makes sense

This should not just apply to cloud computing, but across all of IT.

Take, as another example, adherence to the IT Infrastructure Library (ITIL). Now, ITIL is a great framework, and an increasingly definitive reference for best practices in IT management. Data I have seen suggests as many as 60% of all IT organizations are committed to ITIL, and that implementation of ITIL (whatever that actually means) results in measurable and specific benefits in IT costs, staff and server efficiency, operational maturity, and more.

However, I also hear and read somewhat justified rants about how “ITIL just doesn’t work … ITIL is more 1960s than 2010 … it’s useless.” Yet the truth is, as so often, somewhere in the middle. In this too enterprises can definitely benefit from avoiding the dogmatic application of every single prescription. The same is true for other standards such as COBIT and ISO, or prescriptions from standards groups like the DMTF or NIST. All can deliver significant benefits with less than a 100% implementation.

It also applies in internal adoption of standard operating environment (SOE) components, like making singular (and often binding) choices between, for example:

• VMware vs. Hyper-V vs. Xen
• HP vs. Cisco vs. IBM
• HDS vs. NetApp vs. EMC
• Windows vs. Linux vs. UNIX
• iPhone vs. WinMo vs. Blackberry
• Solution suites vs. point products
• Mainframe vs. Commodity
• Physical vs. virtual vs. cloud

In all these cases and more, although standardization can have specific benefits, the greatest benefit to the enterprise does not always accrue from making an exclusionary choice; from committing to a 100% implementation. Most IT practitioners know that heterogeneity is the new standard – whether intuitively or grudgingly. They know that sometimes the best – or at least necessary – outcomes arise from providing multiple choices, fit to support multiple use cases.

Of course some areas are less flexible. You cannot, for example, pick and choose which parts of PCI, HIPAA, or Sarbanes-Oxley compliance would work best for you. Perhaps ‘close’ only matters in horseshoes and hand grenades, but for sure it doesn’t matter in legal compliance.

However, where possible, IT – practitioners, consultants, vendors, and analysts – need to stay away from dogma. We must avoid making any architecture, maturity model, or industry standard a religious ‘all or none’ battle. Important though they may be, these are not religious battles. These are IT decisions. Moreover, these are business decisions. So we need to keep the business goals in mind, and realize that sometimes a ‘100%’ implementation simply does not make sense.

IT is the Magpie of the Business World

I have a request. I hope it is not too onerous, because something is really starting to grind my gears.

Can we in IT please all stop claiming that any technology is going to kill another?

The latest I am reading, for example, is that NoSQL (for want of a better term) will kill off SQL.

No, it won’t.

My hyperbole aside, I know this with complete and utter certainty,  even though I am barely conversant in database technologies. Seriously, SQL hasn’t even killed off VSAM – first released in 1974 – which is still the foundation for a huge volume, perhaps even the majority, of our daily financial, logistics, retail, and government business. In fact, not only are we still storing data in VSAM, we are still programming in COBOL, and even doing it on 20 year old mainframes. So realistically, an upstart like NoSQL has no chance of killing anything.

Similarly, virtualization will not kill the physical computing infrastructures that came before it. Even most early adopters are struggling to get over 50% of their servers virtualized, while the average penetration is, by some reports, as low as 16%. Meanwhile, the percentage of desktops that have been virtualized is still in single digits. In some cases, so-called ‘legacy’ systems are actually becoming their own hypervisors (e.g. Windows, z/VM, Solaris, and Linux).

The same is true of cloud computing. Even if, as Gartner predicts, by 2012, 20 percent of businesses will own no IT assets – which I find highly dubious; and even if the cloud computing market will be worth $160bn by 2011 – also somewhat dubious; then still a vast majority of organizations will continue to own their IT assets. Even allowing for some substantial private cloud deployment (much less dubious), there is no chance cloud computing will kill the on-premise, installed and owned, IT environment.

Historically, this has always been true. Distributed computing never fully replaced mainframe computing. Indeed, the mainframe is actually experiencing record levels of growth (.pdf) in particular among heavy mainframe users (over 500 MIPS). Personal computing never replaced distributed computing either. The Internet did not kill local computing; thin clients did not kill desktops; Firefox did not kill IE (although IE did eventually kill Netscape); Java did not kill COBOL, let alone C; disk did not kill tape; Salesforce.com did not kill Siebel; Google did not kill Yahoo; Gmail did not kill Exchange.

In fact, it is really quite rare that any new technology completely kills off any other. We in IT are the magpies of the business world, collecting and hoarding all the shiny technologies we can. These are not just collector items or museum pieces though; these are real, mission-critical systems and applications. So we end up with a hybrid of critical technologies spanning not just years, but decades.

(Which is why I am such a strong proponent of heterogeneous IT management … but that is another article)

Perhaps it is just semantics, or a philosophical distaste for absolutes. Perhaps the rampant pace of IT development just makes it seem like we don’t replace technology (when of course we do).

But I would still be really happy if we could all refrain from declaring the death of any technology.

Because chances are it is simply never going to happen.

As of Wednesday this week (2/24), I am now at one of the very best IT management software vendors, CA Inc., where I am leading product marketing for their — our — virtualization management solutions.

In many ways, this was an incredibly difficult decision. EMA is a truly excellent place to work, and the role of an industry analyst was fascinating and fulfilling. The people I worked with and for are some of the best minds in IT – always intellectually stimulating, and straight-out fun to be with. It was truly my privilege to get to know them all, and especially to help my clients and my team to be successful.

Yet this was also one of the easiest decisions I have made. I believe both virtualization and management deliver incredible IT and business benefits, and as virtualization becomes increasingly ubiquitous, management of virtual systems becomes increasingly critical. I have long considered CA a leader in physical and virtual systems management, and believe CA has a great opportunity to extend its leadership in virtualization management, by helping even more IT and business people to be even more successful. As a part of  CA now, I can not only be a part of that opportunity, but can be a significant author of that success.

Moreover, it allows me to indulge my passion for technology and my expertise in marketing in an in-depth, direct, and focused way, rather than the broad, ancillary, and essentially academic role of an industry analyst. I will be able to work directly with some the biggest and most successful companies and technologies, not just in the US, but around the globe. Plus, like EMA, CA also has some incredible minds who are some of the most fun people to hang out with too.

While some will see this a move (back) to ‘the dark side’, I have always considered analysts and vendors to be two sides of the same coin – helping IT to deliver business services in more effective and efficient ways. While some may say that I have ’sold out’ my integrity as an analyst, I have always considered my integrity to be a core and consistent value — and a non-negotiable one — regardless of my employer. While some may think I can no longer champion the best interests of enterprise IT like I did while I was an analyst, I believe the best software companies, and their best people, succeed and thrive specifically because they do exactly that.

As for this blog (and my Twitter feed), all my reasons for blogging and tweeting, and what I hope to achieve (both personally and professionally) with social media, are still the same as they were when I started. I therefore intend to continue writing and posting my personal opinions and insights about technology and other areas that interest me. After all, the areas I work with haven’t really changed, so I am still going to post about virtualization, systems management, data center operations, and cloud computing.

So although I cannot help but be informed by my current position and experience, my goal is to keep posting interesting and informed ideas, regardless of my employer. No doubt some people will stop reading — which is fine — but I still hope you will keep inspiring, contributing to, reading, commenting on, and arguing about these part-time musings of a full-time technologist.