I had a great time at VMworld 2011 Las Vegas this year. As I predicted in my last blog post, I met with loads of amazing people – too many to list out here, let alone in 140 on Twitter! I also saw some great technology in the solutions exchange, dropped in on some fascinating sessions, and of course enjoyed some excellent meals, drinks, and parties!

I was also very pleased to present on Extending the Value of Your VMware Solutions to Design, Deliver and Maintain Reliable, Mission-critical Virtualization and Cloud Services. I certainly was not there to ‘pitch’ any CA Technologies products or solutions (after all, I know that no one wants a sales pitch at a tradeshow like VMworld). Instead, I tried to provide strategic advice to the audience on how to look at their migration to cloud, and especially how to extend VMware’s excellent virtualization and cloud technologies.

My VMworld 2011 Las Vegas Presentation Agenda

With a smattering of additional tips and content from ‘Visible Ops – Private Cloud: From Virtualization to Private Cloud in 4 Practical Steps’, I talked about:

• how to match services with the right cloud using project and portfolio analysis based on models from Visible Ops – Private Cloud, a CA Technologies quadrant framework, Forrester Research’s Evaluating Application Fit With Cloud model, and Freeform Dynamics’ model from Applied Cloud Computing: A practical guide to identifying the potential in your environment
• how to think about your service portfolio, whether considering migrating existing services to a private VMware cloud, building new services on a public VMware cloud, dealing with business users who buy into 3^rd-party cloud themselves, or even services that you may never migrate to the cloud
• how to leverage VMware to deliver both evolutionary cloud models built with virtualization, optimization, automation, orchestration, and dynamic IT; and with revolutionary models that deliver exponential benefits with a virtual business service, built on a virtual service fabric
• how to integrate complex service workflows, skillsets, and technologies, as well as incorporating NIST best practices including cloud service management and service-aware end-to-end application assurance to continually improve service quality, predictability, and costs
• how to apply critical security disciplines including Identity Management & Provisioning, Identity Federation & Single Sign-On, Web Access Management, Privileged User Management, Identity Compliance, and User activity reporting, whether to, from or for the cloud
• how to approach cloud as a transformation opportunity, so you don’t just do the same things in different ways, but fundamentally transform business and IT, delivering a ‘cloud of clouds’ with a broad technology ecosystem stocked with key VMware partners (like CA Technologies!)

You can check out my slides at the CA.com communities site, or over at SlideShare.

Overall, my session seemed to be very well received. A lot of people came up to me there and afterwards and told me how much they enjoyed my presentation, and how useful it was for them. I also enjoyed a great set of questions from the attendees immediately after the session. In fact, we were chatting so much we had to be ushered out so the next session could start.

Immediately afterwards I headed down to the CA Technologies booth, and really enjoyed talking with various practitioners and others at the book signing for ‘Visible Ops – Private Cloud: From Virtualization to Private Cloud in 4 Practical Steps‘ afterwards (with co-authors Jeanne Morain and Kurt Milne). I even had a professor in IT from NYU ask for a copy of my book! Cool!

All in all, I had a great time, made new friends, enjoyed great food, and even managed to avoid the possible downsides of VMworld!

I hope VMware Europe Copenhagen will be just as good – and I hope to see you there!

So, here is something interesting I discovered today, courtesy of a tweet from Christian Reilly (@ReillyUSA) – the US federal agency, the National Institute of Standards and Technology (NIST), today released Version 1 of their Cloud Computing Reference Architecture (PDF). It is free and, like all US Federal Government content, it is open.

I have written about NIST before – both in my research work at EMA and in my personal blog – and wholeheartedly endorse their excellent definitions for cloud computing. If we can trust them to define time – and a thousand more standards besides – we can trust them to define cloud.

So I am more than willing to let them have a go at describing a cloud reference architecture.

The document essentially provides a brief outline of the five key actors:

• Cloud Consumer – Person or organization that maintains a business relationship with, and uses service from, Cloud Providers.
• Cloud Provider – Person, organization or entity responsible for making a service available to Cloud Consumers.
• Cloud Auditor – A party that can conduct independent assessment of cloud services, information system operations, performance and security of the cloud implementation.
• Cloud Broker – An entity manages the use, performance and delivery of cloud services, and negotiates relationships between Cloud Providers and Cloud Consumers.
• Cloud Carrier – The intermediary that provides connectivity and transport of cloud services

Then through a combination of definition, example, and illustration, it places these actors into a big picture end state ‘reference architecture’:

NIST Cloud Reference Architecture V1

Despite some clear flaws, I think this is a great document. More than just a series of definitions, far less than a ‘true’ technical reference architecture, it is advisory and high-level, but practical and usable.

Some key standouts for me include:

‘Grown-up’ management finally takes center stage

I am particularly excited that such a powerful voice in cloud computing is finally highlighting the primary importance of management in their cloud documentation. Almost half this document is focused in cloud management – something I have been deeply committed to for many years. It does not just rehash simplistic notions of cloud – that it is just live migration, capacity management, or an orchestration engine. It shows that you need to maintain many mature enterprise management disciplines – even as ‘old school’ as performance management and SLM – as you grow your cloud maturity. All actors – including consumers and providers – must mature as well. You can call it names like ‘legacy’, or pretend ‘enterprise’ is code for ‘mainframe’ – like that’s a bad thing – but NIST clearly believes a cloud computing environment needs mature management discipline.

It’s all about the service

Of the eleven management slides, six are devoted specifically to a concept NIST calls Cloud Service Management (CSM) – something I first wrote about in 2008, and which likely has been around for longer than that. NIST defines CSM as:

all the service-related functions that are necessary for the management and operations of those services required by or proposed to cloud consumers.

It breaks these down into three main management areas as follows:

NIST Architecture for Cloud Service Management

This is a huge step forward in pragmatic (dare I say, responsible?) cloud service delivery. Many vendors are trying to define cloud as advanced virtualization, or rapid provisioning, or service catalog, or automation – or a proprietary ‘cloud in a box’. Others claim public cloud vendors will do it all, as though there is no need to deal with performance assurance, incident reporting, or bandwidth management. A rational, independent, authoritative body explaining the breadth of integrated enterprise service management required to deliver a high quality cloud service is important information for many CIOs who have been led to believe a more simplistic vision.

Service orchestration needs breadth and deep

This cloud reference architecture devotes special attention to service orchestration across multiple layers of the cloud environment:

• physical resources – including hardware  (memory, storage networking, etc.) and facilities (HVAC, power, comms, etc.)
• virtual systems – hypervisors, virtual machines, virtual data storage, and VM platform tools
• physical systems – NIST specifically accommodates non-virtual resources for cloud delivery
• application delivery – top-down delivery of end-user software clients or other programs
• platform delivery – various development environments, databases, app servers, etc
• infrastructure services – processing, storage, networks, and other fundamental resources

In clearly delineating the need for sophisticated process automation, real-time management, and integration, it shows the need to orchestrate not just a single platform or silo, but end to end across multiple layers, platforms, technologies, and vendors.

The value of an independent judge

I also really like the idea of specifically including an independent arbiter – the Cloud Auditor – that is empowered to “evaluate the services provided by a Cloud Provider in terms of security controls, privacy impact, performance, etc.” The need for an independent reviewer is already well overdue. Today, even cloud leaders like Amazon, WordPress, Salesforce, and Netflix can be down for hours with no reporting or explanation, and with no more payback than a sorry letter and few pennies in credit for time lost. They are also killing off any expectation of security, compliance, or privacy by hiding away fine print like the right to “access, retain, use and disclose your account information and your files … as [they] determine is necessary”. In this climate, we already desperately need an independent agent to adjudge the operations, performance and security of all cloud providers, especially public cloud providers.

However, the reference architecture is not all good, and some significant issues also stood out for me:

Security is a one-sided activity

The reference architecture hangs the responsibility for security almost entirely on the Cloud Provider, which is poor advice. Unlike cookies, security is not a ‘sometime’ food, and active participation in security cannot be attributed to any one actor or interaction. For example, two-factor authentication necessarily requires active participation by both service provider and service consumer. Cloud Auditors and Cloud Brokers also have significant responsibilities for security.

Everyone is a ‘Cloud Carrier’

The ‘Cloud Carrier’ actor essentially elevates all telcos to the role of ‘Cloud Carrier’ with no change in business model or technology. It also actually classifies cabs, couriers, and even the UPS as ‘Cloud Carriers’, as this actor includes any provider of “physical transport of storage media such as high-capacity hard drives.” A requirement for Cloud Providers to set SLAs with Cloud Carriers is especially unlikely for a public cloud, though it makes more sense in the context of a private cloud. It also leads to difficult questions of carrier interoperability, quality of service, traffic shaping, and even ‘Net neutrality.

Encryption is optional

Encryption is included as an optional (!) activity, which of itself is unacceptable for mission-critical enterprise applications. Even then it is ascribed to the Cloud Carrier. The idea sounds great – carriers provide “dedicated and encrypted connections” for the “connectivity and transport of cloud services.” However, it is unrealistic for carriers to implement interoperable encryption for ‘cloud traffic’ (whatever that is). It also forgoes the current, quite logical, de facto standard – encryption directly between the provider and the consumer, regardless of carrier. Again, much of this architecture seems to be more directed at private cloud networks, rather than public networks including the Internet.

Privacy is not having to say you’re sorry

Privacy is included as a single line item without much meat on the bone:

Protect the assured, proper, and consistent collection, processing, communication, use and disposition of personal and personally identifiable information (PII) information on the cloud.

This is so neutral as to be unhelpful. Sharing personal data with advertisers, handing over corporate data to warrantless investigations, or even selling your customer database on eBay, may all be ‘assured, proper, and consistent’ according to some so-called ‘privacy policies’. The document does allow the Cloud Auditor to “evaluate the services provided by a cloud provider in terms of … privacy impact,” but beyond this it has no advice on what privacy actually means. Perhaps this is asking too much of a high-level document, but personal privacy and data loss prevention are critical issues in cloud computing. From controversy over Facebook’s exposure of personal details to cloud providers cutting off legitimate businesses, there is significant concern over privacy. I expected more prescriptive advice, rather than a neutral academic definition, especially from a public body setting policy for the IRS, the Pentagon, Department of Social Security, and other sensitive departments.

Management is entirely a provider activity

NIST attributes cloud management – including security and service management – entirely to the cloud provider. This is rare (if it exists at all) among public cloud providers today, and is unlikely to ever be acceptable for most enterprises. With providers doing all the management the fox is watching the hen house. Consumers will require at least some participation. We learned that it is bad to give total control to third-party providers when we did things like IT outsourcing. Just as with cloud computing itself, the majority of enterprises will probably always want a hybrid model for cloud management.

The bottom line

I do see this as a very useful document. It is really quite good – as far as it goes. It is also a very important document – for what it gets right, for what it gets wrong, and for where it comes from, as NIST is helping to shape cloud standards for the world’s largest consumer of information technology. It is far from perfect, and I believe has some truly fundamental flaws, but it is only a Version 1, and who among us has delivered a perfect product on the first release?

So ultimately, it is good enough for now, but I am very much looking forward to the ongoing development of this document. To quote Raymond Umerley (@SecJitsu):

'Close Enough for Government Work'

Federal CIO Vivek Kundra and the CIO Council

If there was still any doubt about the real world use cases for cloud computing, the US Federal Government last week published a 38-page report  entitled “State of Public Sector Cloud Computing” (link to PDF at CIO.gov). Attributed to the Federal CIO Vivek Kundra, it is stamped with the seal/logo of the CIO Council, which comprises the CIOs of some 28 federal government agencies.

The report details 30 case studies in public sector cloud computing (for both state and federal governments), covering IaaS, PaaS, and SaaS service models; using private, public, community, and hybrid cloud deployment models; with both on-premise and off-premise implementations.

Measurable Benefits from Key Case Studies

After perfunctorily reciting what it calls “the broadly recognized and adopted NIST Definition of Cloud Computing,” and using the opportunity to briefly push its own barrow on cloud standards (a subject I plan to blog about in more detail at another time), the report cites several projects with ‘soft’ outcomes – improved productivity, better efficiency, higher reliability – as well as several planned cloud projects that are yet to bear fruit.

However, most of the report is given over to demonstrating solid and measurable outcomes from over a dozen current cloud deployment case studies involving multiple state and federal government agencies, with cloud success stories such as:

• The US Army is piloting a customized version of Salesforce.com to update its 10 year old recruiting systems for Web 2.0, social media, mobile devices, marketing integration, real-time data interchange, and engagement tracking. At an annual cost of $54,000, this pilot compares to bids from traditional IT vendors ranging from $500K to over $1 million, and has already replaced five traditional recruiting centers.
• The Department of Health and Human Services is also using Salesforce.com to support the implementation of Electronic Health Records systems. This new CRM system for working with participating healthcare providers was deployed in just 3 months, instead of the full year estimated for an internally delivered system.
• The General Services Administration (GSA) moved to a Terremark Enterprise Cloud service, to take advantage of on-demand scalability for Web sites like USA.gov. As a result, GSA accelerated its site upgrade time from nine months to a maximum of one day, reduced monthly downtime from roughly two hours to near zero (99.9% availability), and reduced annual costs for USA.gov by $1.7 million, from $2.35 million to $650,000, or 72%.
• The Defense Information Systems Agency (DISA) is using virtualization with a self-service portal to provide on-demand server space for development teams. With just an approved Government credit card, these end users can set up new environments (with DoD-compliant security guaranteed) in just 24 hours – down from three to six weeks – and at a “reasonable” cost.
“DISA estimates PaaS cloud savings between $200,000 and $500,000 per project.”
• DISA also used cloud provider CollabNet to set up Forge.mil, a private PaaS cloud development environment with a heavy focus on collaboration and code sharing/reuse. DISA estimates this saves between $200,000 and $500,000 per project – not including the estimated $15 million in cost avoidance by utilizing an open source philosophy.
• The Lawrence Berkeley National Labs (LBL), part of the Dept of Energy, is using Google Apps for 2,300 e-mail users, and planning to more than double that by August. LBL estimates they will save $1.5 million over five years “in hardware, software and labor costs from the deployments they have already made.”
• NASA’s Jet Propulsion Laboratory used a Microsoft Azure development platform “to excite the public about Mars” with the website, BeAMartian.jpl.nasa.gov. This site has generated over 2,000 pieces of social media, inspired 200 traditional media stories, responded up 2.5 million API queries, gathered  40,000 votes in its ‘Town Hall’ polls, and attracted 5,000 registrations from individuals and teams.
• The Federal Labor Relations Authority recently replaced its underperforming, decade-old case management system, switching to Intuit’s Quickbase system. As a result, it was able to go from requirements-definition to completed development in 10 months – a quarter of the original deployment time – and expects a TCO reduction of nearly $600,000 over five years.
“Moving Recovery.gov to Amazon EC2 will drive cost savings of $750,000”
• Less than a month ago, the Recovery Accountability and Transparency Board moved Recovery.gov to a “fully scalable site” in the Amazon EC2 infrastructure cloud, delivering “added security” and “nearly 100 percent uptime.” The Board is projecting that this move will drive cost savings of $750,000 through FY2011 (4% of its $18 million budget) – while allowing it to reallocate more than $1 million worth of hardware and software.
• The New Jersey Transit Authority also used Salesforce.com (alongside some organizational change) to improve its customer service system. The new cloud-based processes allowed the same number of staff to handle 5 times the number of enquires (from 8354 in 2004 to 42,323 in 2006), reduced response time for enquiries by 35%, and improved productivity by 31%.
• Wisconsin’s Department of Natural Resources replaced its aging video conferencing systems with Microsoft LiveMeeting as an alternative to server-based collaboration software. Since migration in 2009, this has saved an estimated $320,000, with ROI expected to grow from 270% for the first year to over 400% in future years.
• The State of Utah uses several public cloud services (Force.com, Google Earth Pro, and Wikispaces), and has completed 70% of its private cloud project to move 1,800 physical servers in over 35 locations to a virtual platform of just 400 servers. The private cloud project alone is expected to the state save $4 million annually – over 2.5% of its $150m IT budget.
• Facing a $400 million deficit, the City of Los Angeles has been transitioning to Google Apps cloud-based e-mail, with all employees to be cut over by June 30 this year. The City’s CTO estimates a direct savings of $5.5 million over 5 years, and a total ROI (including increased productivity) of $20-30m.
  “Colorado estimates annual savings of $8m, and up to $20m in expense avoidance”
• The City of Orlando rolled out a similar Google Mail project for all 3,000 city employees in January this year. The City has realized a 65% reduction in e-mail costs, not including benefits from improved productivity, increased storage allocation (from 100MB to 25GB per user), improved security/malware detection, and enhanced mobile device support.
• The State of Colorado is shifting to a hybrid cloud model, mixing private cloud (an existing data center leveraging server virtualization), a virtual private cloud (for additional pay-as-you-go scalability), and public cloud (Google Apps for e-mail and office productivity). Just by shifting 122 servers running Lotus Notes, Microsoft Exchange, and Novell GroupWise to the cloud, Colorado estimates annual savings of $8 million, and up to $20 million in expense avoidance over 3 years.

Set SMART Goals, But Be Pragmatic

Kundra does not shy away from clearly stating his ongoing cloud computing goals in this report. By 2011, all business cases for new federal IT investment must include cloud alternatives; by 2012, all enhancements to existing systems must do the same; by 2013, all IT investments, even on legacy systems, must be justified against a cloud alternative. These SMART (Specific, Measurable, Attainable, Relevant, and Timed) goals are important to overcome the all-too-frequent adoption of disruptive technologies almost as a fad, unrelated to business goals and without a clear and realistic timeline.

However, these case studies show an essential pragmatism about the public sector approach to cloud computing. Kundra and the CIO Council recognize (as I have previously published) that the cloud will not completely replace on-premise IT, stipulating:

“Federal agencies are to deploy cloud computing solutions to improve the delivery of IT services, where the cloud computing solution has demonstrable benefits versus the status quo.”

So while cloud must be increasingly evaluated, actual cloud adoption must be justified by “demonstrable benefits” that improve IT service delivery, not just reduce costs. As I have stated in EMA research and blogged about here, it is important for enterprises (public or private) to “look for opportunities, and do what makes sense” when it comes to cloud computing. This is reflected by thought-leaders like Gartner’s Thomas Bittman (@tombitt), who explains that for some organizations “a 70% private cloud is absolutely good enough.”

Cloud Lessons For Other CIOs?

These case studies have a lot of lessons to offer other business and IT leaders, both private and public sector, in everything from mid-sized businesses to the largest enterprises. They detail many clear and realistic case studies; provide insight into achieving both specific ROI and soft benefits; show how cloud can be applied to both business- and IT-oriented goals; and give ideas for how CIOs might address real problems with cloud alternatives.

Moreover, more than any set of self-published corporate case studies, this is incredibly significant, because, as the report points out:

“The United States Government is the world’s largest consumer of information technology, spending over $76 billion annually on more than 10,000 different systems.”

This level of influence from the world’s largest consumer of IT will drive a solid and relentless march to cloud computing, a juggernaut that will likely carry the rest of us along, whether we like it or not.

However, it reads almost like promotional material from a cloud provider – which, in a way, it is – because it does not deal directly with any of the potential problems of cloud computing. It mentions security only very briefly, and then only how certain cloud implementations actually improve security (with no details). It does not give any details of how federal clouds have ensured compliance with regulations like the Federal Rules of Disclosure and DOD 5015, and industry requirements like PCI-DSS. It does not talk about if, or how, they overcame the endemic  problems of performance assurance and continuity in the cloud. Perhaps most ironically of all, it does not even mention how it overcame the tough political and departmental challenges that are cited by analysts as one of the top barriers to both virtualization and cloud adoption.

So for CIOs, this report really needs to be taken with a grain of salt. Be informed and educated by these case studies; use them to be set pragmatic expectations and SMART goals; but be wary that as much as it says about the upside of cloud computing, it avoids saying just as much – if not more – about the potential for deleterious, or even disastrous, downsides.

Effort vs. Payback is an Everyday Business IT Decision

I read recently a good blog post from Thomas Bittman (@tombitt) of Gartner Group, about how sometimes close enough is good enough. Talking specifically about private cloud, he talked about how an ‘imperfect’ cloud deployment – one that does not have all five essential characteristics, for example – might be enough for some organizations.

I especially appreciated how he highlighted some very specific, real-world examples to sustain his advice. As he shows, sometimes you don’t need a ’100%’ implementation, and for very good business reasons.

Not every IT organization needs a fully self-service interface, and many smaller organizations see no value in usage metering. They simply want to deliver services faster. For them, a 70% private cloud is absolutely good enough … it all comes down to business requirements, return on investment, and future strategy. How far you go is your decision.

via Driving for Imperfection With Your Private Cloud.

If you haven’t seen it yet, you should. It’s a quick read, only 4 paragraphs and less than 300 words. Go ahead. I’ll still be here when you get back.

The theme is very similar to something I wrote in a research report for EMA, ‘The Responsible Cloud‘, also on cloud computing. Regarding the NIST definition of cloud, I cautioned against dogmatic interpretations of cloud computing, and the notion that a ‘real’ cloud must necessarily have all of the essential characteristics, or fit some specific deployment model. Flexibility is key, I advised, and delivering on key business requirements is more important than definitions.

Two other things happened this week that made me think about this in different ways:

• An internal session at CA reviewing some customer-facing materials. All attendees agreed – we can’t preach unattainable dogma; we need to deal with specific requirements and partial deployments, as well as broad requirements that come from  ’100%’ implementations.
• A group discussion on LinkedIn, where an IT practitioner wanted advice on building a small private cloud. He was soon inundated with an unrealistic list of requirements, from hypervisor features to management disciplines, that he *must* have to build a ’100%’ cloud.

The similar inferences in three otherwise unrelated conversations started me thinking more broadly about ’100% adoption’. It IT, as in life, you never really need a Rolls Royce. You can aspire to the quality, appreciate its refinement, and in some cases you may be fortunate enough to actually enjoy it, but there is a point where it simply doesn’t make sense to pursue that level of luxury. Mostly you can get away with a Ford. Sometimes you can even make do with a second-hand Lada.

The same Pareto-like principle applies roughly throughout IT (much to the annoyance of just about every security pro I have ever met) – although the actual ratio may vary wildly, you can often get most of the benefit from less than a ’100%’ implementation.

The phrase that sprang to mind for me was the same conclusion that I published elsewhere in the Responsible Cloud report, and the same notion that many IT pros live by, day in and day out:

It is important to look for opportunities, and do what makes sense

This should not just apply to cloud computing, but across all of IT.

Take, as another example, adherence to the IT Infrastructure Library (ITIL). Now, ITIL is a great framework, and an increasingly definitive reference for best practices in IT management. Data I have seen suggests as many as 60% of all IT organizations are committed to ITIL, and that implementation of ITIL (whatever that actually means) results in measurable and specific benefits in IT costs, staff and server efficiency, operational maturity, and more.

However, I also hear and read somewhat justified rants about how “ITIL just doesn’t work … ITIL is more 1960s than 2010 … it’s useless.” Yet the truth is, as so often, somewhere in the middle. In this too enterprises can definitely benefit from avoiding the dogmatic application of every single prescription. The same is true for other standards such as COBIT and ISO, or prescriptions from standards groups like the DMTF or NIST. All can deliver significant benefits with less than a 100% implementation.

It also applies in internal adoption of standard operating environment (SOE) components, like making singular (and often binding) choices between, for example:

• VMware vs. Hyper-V vs. Xen
• HP vs. Cisco vs. IBM
• HDS vs. NetApp vs. EMC
• Windows vs. Linux vs. UNIX
• iPhone vs. WinMo vs. Blackberry
• Solution suites vs. point products
• Mainframe vs. Commodity
• Physical vs. virtual vs. cloud

In all these cases and more, although standardization can have specific benefits, the greatest benefit to the enterprise does not always accrue from making an exclusionary choice; from committing to a 100% implementation. Most IT practitioners know that heterogeneity is the new standard – whether intuitively or grudgingly. They know that sometimes the best – or at least necessary – outcomes arise from providing multiple choices, fit to support multiple use cases.

Of course some areas are less flexible. You cannot, for example, pick and choose which parts of PCI, HIPAA, or Sarbanes-Oxley compliance would work best for you. Perhaps ‘close’ only matters in horseshoes and hand grenades, but for sure it doesn’t matter in legal compliance.

However, where possible, IT – practitioners, consultants, vendors, and analysts – need to stay away from dogma. We must avoid making any architecture, maturity model, or industry standard a religious ‘all or none’ battle. Important though they may be, these are not religious battles. These are IT decisions. Moreover, these are business decisions. So we need to keep the business goals in mind, and realize that sometimes a ’100%’ implementation simply does not make sense.

Surfing  a couple of blogs today, jumping from another analyst commenting that virtualization is not cloud (a fair, if unexplored, post), I came across William Vambenepe’s post from September on the confusion between virtualization and Cloud Computing. As he did on my blog recently, I started to post a reply to his site, and then as it expanded, decided to post it as a full reply on my own blog.

I like the thinking, and agree with a lot of the principles involved. Without doubt, virtualization is not cloud. But I can’t agree with it all. Apart from technical quibbles (like the part about mainframe LPARs not running on a hypervisor), I simply find it unreasonable, if not impossible, to think of implementing cloud computing without virtualization.

My key sticking point in most of these discussions [edit: not necessarily William's post - see comments below] is that they continually assume that ‘virtualization’ is synonymous with ‘hypervisor’, or at best with ‘server virtualization’. Neither is true. When EMA first defined virtualization (a definition that has taken hold more or less throughout the industry), we defined it as:

“a technique for abstracting or hiding the physical characteristics of computing resources from the way in which other systems, applications, or end users interact with those resources.”

Even now, Wikipedia defines virtualization as “the abstraction of computer resources” and “hid[ing] the physical characteristics of a computing platform from users.”

No mention of a hypervisor there, and with good reason. Virtualization is much more than a hypervisor, and applies to much more than servers. In fact, EMA’s original definition made this clear by including the following clarifying note:

“This includes making a single physi­cal resource (such as a server, an operating system, an application, or storage device) appear to function as multiple logical resources; or it can include making multiple physical resources (such as storage devices or servers) appear as a single logical resource.”

Indeed, many forms of virtualization (and cloud) are possible without a hypervisor – like OS virtualization, storage virtualization, grid and cluster computing, terminal services, and more. So while it is widely known that Amazon runs its cloud on a classic server virtualization platform (Xen), even a Google-like cloud, which is based (as I understand it) entirely on a fully hardware-based deployment, without any hypervisors, is still using another virtualization technology – grid computing.

So cloud is definitely possible without a hypervisor, but is it possible without virtualization?

Perhaps, but it is far less than ideal.

William cited SoftLayer Technologies  as doing cloud on bare metal; and  Loudcloud as being cloud before it was in vogue. Although I am not sure the latter is true, and Softlayer provide few details about their bare-metal cloud, it seems to be possible to provide cloud computing without virtualization.

Yet with very few exceptions, it is ill-advised at best. In implementation, if not in theory, the many essential characteristics noted in the NIST cloud definition (EMA’s preferred definition) are only barely possible in a purely physical environment.

Sure, you could get rapid elasticity, rapid provisioning, minimal human interaction, dynamic resource assignment, location independence, resource abstraction, etc. with a physical deployment. While they were both substantially unsuccessful with customers, IBM’s On-Demand and HP’s Adaptive Infrastructure both accommodated these elements primarily through automation, and without virtualization (or at least with virtualization as only an optional component). Even without automation, you could imaginably provision and manage physical servers manually to achieve this on-demand, adaptive, cloud infrastructure. In theory, all things are possible.

In practice though, cloud computing without virtualization is barely realistic. It is an edge case at best. Given what virtualization can do – for resource pooling, rapid provisioning, reducing intervention, resource abstraction, workload elasticity, and more – why would you try to implement cloud without it?

And that is just on the server! Given the different types of virtualization – especially network virtualization and storage virtualization – it seems that cloud without virtualization is not just ill-advised, but positively crazy.

For example, would anyone really copy all the data from one DAS drive to another in order to ‘dynamically’ scale a workload onto a bigger machine? Would you uninstall a drive from one server, and put it into another? Would you physically switch or reprovision a network in order to abstract a new server located in a different data center? Even to the biggest skeptic, cloud without any virtualization must seem a ridiculous notion, if not an impossible one.

So yes, William is technically correct (“the best kind of correct!”) – virtualization is not cloud, and it is possible to provide cloud services without virtualization.

But (with apologies to Samuel Johnson) it is like a dog walking on his hind legs – it is not done well; but you are surprised to find it done at all.

I am getting so sick of the continual bickering over definitions of cloud computing. Even more frustrating is the hype from all the vested interests – vendors and analysts, mostly – trying to define cloud computing in ways that they imagine will best contribute to their own commercial success. And I know that I am not alone.

What is wrong with the definition that the US National Institute of Standards and Technology (NIST) – a division of the US Department of Commerce – uses?

You can read the entire definition online [link updated 8/12/11]. It is only 2 pages. Here, for the unaware, is the meat of it:

“Cloud computing is a model for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications, and services) that can be rapidly provisioned and released with minimal management effort or service provider interaction.”

Does this suck so badly that every [insert your preferred expletive epithet here] needs a new definition?

It goes on to include:

• Five essential characteristics: On-demand self-service; Broad network access; Resource pooling; Rapid elasticity; and Measured Service.

• Three service models: Software as a Service (SaaS); Platform as a Service (PaaS); and Infrastructure as a Service (IaaS).

• Four deployment models: Private cloud; Community cloud; Public cloud; and Hybrid cloud.

So what exactly is wrong with that?! Why does every man and his dog feel the need to throw their own definition of could computing into the ring?

Don’t get me wrong. Definitions are important. Definitions enable a common understanding of terminology, essential when talking about complex technologies. And I have pushed my own definitions before (like my definition for virtualization, widely adopted after Wikipedia picked it up in 2006).

But why fight city hall (in this case, almost literally)? NIST has a very elegant definition that is:

• Intelligent – it has been through (to date) 15 iterations, and has accepted input from many of the brightest minds in cloud computing (while presumably ignoring some dimmer bulbs)
• Independent – it is from a mature, well-established, and exceptionally talented US government agency, which is both apolitical, and science-based
• Commercially agnostic – it does not specify that anyone needs to be making money, nor does it preclude it, allowing cloud to be B2B, B2C, B2G, G2C, or any other model
• Accommodating – all established cloud vendors (like Amazon, Google, Rackspace, Salesforce, and others) fit into this definition, as well as private and government models.
• Clear – it is not full of jargon or ‘cloudwash’, but rather has easily understood, plain English concepts that are not only unambiguous but also usefully prescriptive
• Comprehensive – it includes all the important core concepts such as self-service, resource pooling, rapid elasticity, accessibility, usage costing, multiple use cases, and more
• SMART – it does not try to create anything exceptional or outrageous, but does define a set of Specific, Measurable, Achievable, Relevant, and Timely objectives

We trust NIST to define the official time for all of the United States. We trust it to calibrate instruments for NASA. We trust it to supply “industry, academia, government, and other users with over 1100 reference materials”.

Moreover, this is what the US government is using to define cloud computing, as noted by Vivek Kundra (the US Federal CIO). Indeed, Kundra has strongly indicated that the US government will be one of the strongest, largest, and most important proponents, providers, and consumers of cloud computing (cf. sites like apps.gov and data. gov). Other levels of government – and even other nations – will almost certainly follow their lead, and the NIST definition of cloud computing.

So why can’t people trust NIST with the definition of cloud computing, and just get on with the job of solving real problems for their customers? Bickering and chest-beating over self-enriching definitions is not needed, it is not useful, and it is not helpful.