IT is the Magpie of the Business World

I have a request. I hope it is not too onerous, because something is really starting to grind my gears.

Can we in IT please all stop claiming that any technology is going to kill another?

The latest I am reading, for example, is that NoSQL (for want of a better term) will kill off SQL.

No, it won’t.

My hyperbole aside, I know this with complete and utter certainty,  even though I am barely conversant in database technologies. Seriously, SQL hasn’t even killed off VSAM – first released in 1974 – which is still the foundation for a huge volume, perhaps even the majority, of our daily financial, logistics, retail, and government business. In fact, not only are we still storing data in VSAM, we are still programming in COBOL, and even doing it on 20 year old mainframes. So realistically, an upstart like NoSQL has no chance of killing anything.

Similarly, virtualization will not kill the physical computing infrastructures that came before it. Even most early adopters are struggling to get over 50% of their servers virtualized, while the average penetration is, by some reports, as low as 16%. Meanwhile, the percentage of desktops that have been virtualized is still in single digits. In some cases, so-called ‘legacy’ systems are actually becoming their own hypervisors (e.g. Windows, z/VM, Solaris, and Linux).

The same is true of cloud computing. Even if, as Gartner predicts, by 2012, 20 percent of businesses will own no IT assets – which I find highly dubious; and even if the cloud computing market will be worth $160bn by 2011 – also somewhat dubious; then still a vast majority of organizations will continue to own their IT assets. Even allowing for some substantial private cloud deployment (much less dubious), there is no chance cloud computing will kill the on-premise, installed and owned, IT environment.

Historically, this has always been true. Distributed computing never fully replaced mainframe computing. Indeed, the mainframe is actually experiencing record levels of growth (.pdf) in particular among heavy mainframe users (over 500 MIPS). Personal computing never replaced distributed computing either. The Internet did not kill local computing; thin clients did not kill desktops; Firefox did not kill IE (although IE did eventually kill Netscape); Java did not kill COBOL, let alone C; disk did not kill tape; Salesforce.com did not kill Siebel; Google did not kill Yahoo; Gmail did not kill Exchange.

In fact, it is really quite rare that any new technology completely kills off any other. We in IT are the magpies of the business world, collecting and hoarding all the shiny technologies we can. These are not just collector items or museum pieces though; these are real, mission-critical systems and applications. So we end up with a hybrid of critical technologies spanning not just years, but decades.

(Which is why I am such a strong proponent of heterogeneous IT management … but that is another article)

Perhaps it is just semantics, or a philosophical distaste for absolutes. Perhaps the rampant pace of IT development just makes it seem like we don’t replace technology (when of course we do).

But I would still be really happy if we could all refrain from declaring the death of any technology.

Because chances are it is simply never going to happen.

As of Wednesday this week (2/24), I am now at one of the very best IT management software vendors, CA Inc., where I am leading product marketing for their — our — virtualization management solutions.

In many ways, this was an incredibly difficult decision. EMA is a truly excellent place to work, and the role of an industry analyst was fascinating and fulfilling. The people I worked with and for are some of the best minds in IT – always intellectually stimulating, and straight-out fun to be with. It was truly my privilege to get to know them all, and especially to help my clients and my team to be successful.

Yet this was also one of the easiest decisions I have made. I believe both virtualization and management deliver incredible IT and business benefits, and as virtualization becomes increasingly ubiquitous, management of virtual systems becomes increasingly critical. I have long considered CA a leader in physical and virtual systems management, and believe CA has a great opportunity to extend its leadership in virtualization management, by helping even more IT and business people to be even more successful. As a part of  CA now, I can not only be a part of that opportunity, but can be a significant author of that success.

Moreover, it allows me to indulge my passion for technology and my expertise in marketing in an in-depth, direct, and focused way, rather than the broad, ancillary, and essentially academic role of an industry analyst. I will be able to work directly with some the biggest and most successful companies and technologies, not just in the US, but around the globe. Plus, like EMA, CA also has some incredible minds who are some of the most fun people to hang out with too.

While some will see this a move (back) to ‘the dark side’, I have always considered analysts and vendors to be two sides of the same coin – helping IT to deliver business services in more effective and efficient ways. While some may say that I have ’sold out’ my integrity as an analyst, I have always considered my integrity to be a core and consistent value — and a non-negotiable one — regardless of my employer. While some may think I can no longer champion the best interests of enterprise IT like I did while I was an analyst, I believe the best software companies, and their best people, succeed and thrive specifically because they do exactly that.

As for this blog (and my Twitter feed), all my reasons for blogging and tweeting, and what I hope to achieve (both personally and professionally) with social media, are still the same as they were when I started. I therefore intend to continue writing and posting my personal opinions and insights about technology and other areas that interest me. After all, the areas I work with haven’t really changed, so I am still going to post about virtualization, systems management, data center operations, and cloud computing.

So although I cannot help but be informed by my current position and experience, my goal is to keep posting interesting and informed ideas, regardless of my employer. No doubt some people will stop reading — which is fine — but I still hope you will keep inspiring, contributing to, reading, commenting on, and arguing about these part-time musings of a full-time technologist.

Many claims are made to support this so called Bring Your Own PC (BYOPC) approach – although they seem mostly, if not only, to originate from vendors (notably desktop virtualization and application virtualization vendors) that have a vested commercial interest in its success. I disagree with many of these claims (especially the questionable claims of cost reduction), but I do agree that BYOPC can have some benefits.

However, one of the many claims in support of BYOPC is that it will help organizations to attract and retain an important demographic of young, technologically sophisticated employees – the so-called ‘millennials’,  ‘echo boomers’, ‘generation next’, or ‘generation Y’, all loose terms generally used to describe people aged between 18 and 35 years old.

Personally, I find this claim to be absurd.

My main issue with this claim is the implicit assumption that millennials have such a strong choice in their employment options that issues like what type of operating system they use, or what device they work on, can be significant decision factors in whether or not to accept a job offer.

This flies in the face of unemployment statistics that suggest, today more than ever, millennials simply do not have this level of choice. Put plainly, in the near term millennials should be happy to simply get a job offer, let alone one that comes with over $2000 to buy themselves a shiny new MacBook Air.

Specifically, data from the US Bureau of Labor Statistics (see chart below) shows that in fourth quarter of 2009, unemployment for 18-35 year olds was on average 17%. For males specifically, this was on average 25%, and as high as 30%  (for males, 18-19 y.o). Compare this to the national average for ages 35 and above – 8% for the general population, and 9% for males – and you can see that this demographic does not exactly have abundant bargaining power on the job market.

Unemployment by age, sex, marital status 2006-2009 (Source: BLS)

Perhaps this will change over time, but as the chart above shows, unemployment for millennials has been trending up, not just since the onset of the recession around the end of 2008, but at least since 2006. It may (and hopefully will) come down dramatically, improving millenials’ bargaining power for employment,  but there is no sign that this is happening, or that it will happen anytime soon. And remember, when we are looking at BYOPC it is not only for technology workers,  but also (perhaps primarily) for knowledge workers across many fields – sales, finance, management, R&D, etc. – so any specific skill shortages in IT that may skew millennials’ bargaining power do not really come into play.

Of course, there are probably nuances here that a behavioural statistician would find and explain much better than I can – granular variations by  age, education, location, industry, and more. For example, unemployment among slightly older millennials aged 25 to 34,  many of whom presumably have college degrees, is much lower than millennials aged 18 to 24. Yet at over 10% unemployment, even this group still has an unemployment rate several percentage points higher than ages 35 and up (just under 8%).

However, I do not see any promoters of BYOPC incorporating such detailed demographic analysis to substantiate their claims for BYOPC; rather, most seem to just be making unsubstantiated claims using baseless assumptions about millennials’ employment ‘needs’ without even considering widely available independent data that substantially undermines their position.

Sure, there are studies that suggest, for example, that millennials consider “state-of-the-art technology is an important consideration in selecting an employer.” However, the most credible of these studies was conducted prior to the global economic downturn, when unemployment among 18-35 year olds was just 9-10% – almost half what it is today. In any case, an organization does not need a BYOPC program simply to provide state-of-the-art technology. If an employer gives a millennial employee a top-of-the-line company-owned Dell, HP, Lenovo, Sony, or Apple laptop, would they turn the job down just because they cannot buy it themselves?

So based on a cursory analysis of recent, credible, and available data, the idea that a young person would turn down a job in this economic climate, simply because they can’t get their preferred laptop or mobile device seems to me quite ridiculous. At best, it may help companies attract the very top tier of millennial graduates who do have multiple job offers to choose from, but I expect this would still be the least of the considerations of 18-35 year olds. Even for the best of them, this likely pales compared to significant concerns about compensation, vacation and holidays, health care, education support, flexible hours, corporate ethics, retirement funding, work-life balance, telecommuting, career opportunities, and more.

This then has significant implications for CIOs and others looking at BYOPC.  Despite the validity (or otherwise) of any other claims in favour of BYOPC, no organization should be looking to BYOPC to attract and retain staff from the echo boom generation. They would be better off looking at a dozen or more other important factors than spending the significant time, effort, and money on implementing a BYOPC program.

The most recent example of such failures is the power outage at IaaS provider Rackspace’s London facility, but of course, we have seen this before from many public cloud providers – including Rackspace in particular, and not just once. SaaS provider Salesforce.com (and its PaaS arm, Force.com) has also had one outage already this year, an event that is far from unusual, and nothing new. Amazon, Yahoo, Microsoft, GoGrid, RIM, Twitter, Paypal and many others have also had substantial (and often repeated) outages.

There are some who dismiss these failures as one-offs, write off partial or short-term failures as too low-impact to matter, or just give poor DR a pass because it is the cloud, and we should not expect any better. Others reach to find semantic differences, calling it a service outage, an application failure, a facilities outage, a power outage, or a resource shortage. Some just redefine cloud to include only those services that did not go down this week (bonus points for adding a vainglorious reference to the ‘real cloud’ or ‘true cloud’).

YMMV, but I don’t see it that way at all. With so many repeated failures in so many cloud providers, these are not just one-off failures. They don’t just happen to isolated providers, they happen across the board. Regardless of the cause – the application, the facilities, the power supply, the lightning rod – an outage of a cloud service provider is still a cloud outage. And the definition of cloud I use is not dogmatic enough to exclude any of the providers that I have cited (and others), let alone define a ‘true cloud’.

So I see every reason to believe that downtime in the public cloud is not the exception, it is the rule; that outages in the public cloud are endemic, and they are systemic.

However, this judgement is absolute, not relative. Failure in one cloud provider may (and I believe does) implicate all cloud providers, but it does not imply downtime is more of a problem in the public cloud than in traditional enterprise IT. Indeed, there is a strong argument that enterprise IT has as many if not more outages, so uptime and availability is no worse in the public cloud than with traditional IT.

In fact, EMA research has shown average enterprise IT uptime is just ‘two nines’, at 99.5%. For a 24×7 system, that is over 50 minutes of downtime, each and every week. Contrast this with public cloud providers. Even with their problems, Amazon EC2 offers a “reasonable effort” to deliver an annual uptime of at least 99.95% – or about 5 minutes downtime per week – and offers a 10% credit for “eligible” breaches. Google guarantees ‘three nines’ (99.9%) uptime for its Premier Edition, or around 10 minutes downtime per week (although it promotes a study that claims an average downtime of 15 minutes a week). The Rackspace SLA promises network, HVAC, and power will be up 100%, though it does not guarantee server availability (beyond promising a 60 minute maximum repair window), and all promises exclude ‘scheduled maintenance’.

So for the average enterprise, ‘normal’ cloud computing outages, while endemic, can still be 5 to 10 times less frequent than in their own data centers.

However, it is not a black and white issue, not least because a focus on broad uptime percentages or on single instance failures ignores the huge nuance behind a single uptime number.

For example, many environments report ‘five nines’ (99.999%) or even 100% uptime – less than one second of unplanned downtime each day – for their critical systems by using processes and tools for high availability, fault tolerance, asset maintenance, live migration, etc. EMA has also found that best performers in Virtual Systems Management – 15% of enterprises – report an average of five nines uptime.

If they need to, enterprise CIOs can invest in technology to provide two, three, four or five nines uptime within their own data center. They can implement redundant hardware, HA and FT, multi-site replication, and more – if they want to pay for it. They can monitor for outages, know exactly when they happen, and react automatically to fix them immediately (or even use predictive analytics and automation tools to avoid them entirely). They can provide this as required, as a value-add to their business unit customers, or as an additional charge (or at least an exposed cost)  to the business to let them choose how critical their applications really are.

However, with the public cloud, neither the business nor the CIO has any real choice. With few or no management or automation tools, public cloud providers simply do not currently offer the same flexibility and accountability as internal IT. Without good management tools, no public cloud provider currently matches enterprise IT at the higher mission-critical reaches of availability.

So, this fight does not end in a knock-out for either side. As is common in the real world, nothing is black and white, but rather many shades of grey.

In the end, the solid achievements of public cloud providers, despite the bad press, does not absolve them of any blame or negate generalizations of downtime being endemic in the public cloud. However, the relatively poor performance of enterprise IT on average still does not ensure public cloud will be any better in any specific cases.

What this does show, however, is that CIOs who are planning to build their own private cloud have a surprisingly high bar to reach. They should not dismiss public cloud options out of hand, but rather should strongly consider whether they can realistically and cost-effectively meet the three, four, and even five nines that public cloud providers guarantee.

The only significant and unique benefit of KVM for server virtualization (as noted by Sander van Vugt in our (virtual) debate on Xen vs.KVM Linux Virtualization Hypervisors) is that KVM is part of the Linux kernel. This ensures broad standardization, patch compatibility, simpler upgrades, and a low-impact on-ramp for existing Linux IT shops.

Yet this is a solution for a problem that does not really exist.

Large enterprises already run thousands of components, from services/daemons to drivers to applications, all as additions to various kernels. Maintaining one more (or even several more) non-kernel components like Hyper-V, XenServer, ESX, etc., is not a net negative. On the contrary, EMA data shows that virtualization actually improves the productivity of server administrators, and by an average of around 10% – up to 20% or more for best performers. For competent administrators with good lifecycle management tools, the time they spend to learn, test, and maintain hypervisors is a significant effort, but it is time paid back with interest.

On the other hand, many downsides to KVM are all too apparent.

It is easy to point to the lack of technology features and maturity in KVM – areas like live migration, paravirtualization, networking, isolation, performance, security, or a host of other  features which KVM (in some cases arguably) lacks. I have only some doubt that KVM will meet these low-level functional requirements eventually, but it will not be anytime soon. Yet they are essentially table stakes in server virtualization today.

The inherent dependency on Linux would also require a major shift in  platforms for the average datacenter (where Windows outnumbers Linux by  150:1), and a major investment in resourcing, training, and software. This is hardly an attractive proposition for a data center manager. Still, existing Linux staff will be able to pick it up, and could even have some success on their (relatively few) existing Linux platforms.

However, even if these weaknesses are overcome, KVM has a much more strategic problem – the gaping void in the KVM management ecosystem. There is almost no third-party support for KVM from management vendors. Even stated support from key partner vendors like IBM, HP, and of course Red Hat is basic at best. What’s more, EMA data suggests KVM will not foster a significant management ecosystem in the future, either.

EMA’s research on Virtual System Management showed convincingly how important management is to virtualization. Across 18 different management disciplines, almost all correlated with measurably better outcomes in metrics like MTTR, provisioning time, availability, VM density, migration speed, and more.

EMA’s new cloud research shows a similar importance. Applying mature automation and management disciplines to virtual systems is directly correlated with positive cloud outcomes like reduced CapEx, reduced OpEx, improved operational maturity and more.

Not surprising then, that over 80% of enterprises consider manageability an important or very important factor in their virtualization and cloud technology decisions.

Unfortunately, KVM ranks anywhere from 4^th to 10^th in enterprise preferences for virtualization and cloud technology providers. It comes behind first ESX, then Hyper-V or Xen (multiple implementations), often various UNIX hypervisors (PowerVM, Integrity VMs or vPars, Solaris Containers), and even z/VM. No enterprise demand means that management vendors have little incentive to support KVM.

In fact, in my conversations with management software vendors, most generally put KVM around 5th in line for support – which, realistically, means it is not even on the current roadmap. What’s more, for better or worse several of them have a vested interest in not supporting KVM (no points for guessing who).

This means KVM has little or no prospect of gaining third-party support for virtualization management tools like VM-aware backup and restore, VM provisioning, virtual resource management, VM configuration auditing, virtual performance monitoring, VM lab management, VM image control, storage management,network automation and more. The same holds true for integration with higher-level virtual systems management tools for virtual and physical data center automation and service management disciplines.

For any IT group, sophisticated management tools deliver many proven benefits. For larger enterprises especially, they are simply not optional.  Without even the prospect of a robust management ecosystem, KVM is simply a non-starter in most large-scale deployments. For my enterprise clients at least, it is certainly not a credible choice for x86 server virtualization.

Microsoft and Opalis

Today Microsoft Corporation (NASD:MSFT) announced a definitive agreement to acquire Opalis Inc., the leading independent vendor of IT Process Automation (ITPA) software.

IT Process Automation (ITPA) is a Data Center Automation (DCA) discipline that EMA defines as “the ability to automate and integrate the workflow of complex, multi-discipline IT management processes.” This automation can replace many manual, resource-intensive, and error-prone activities that typically cross multiple IT components, disciplines, and/or departments. ITPA delivers exceptional results including freeing up 77% more staff for strategic projects, providing more than 60 additional hours of system availability per year, and saving an average $500,000 more per year on staff costs than other Data Center Automation (DCA) disciplines.

This space has been gaining interest, both expanding and consolidating, for some time, as evidenced by significant development and acquisition activity from Novell (ZENworks, PlateSpin), HP (Opsware, iConclude), BMC Software (RealOps, Atrium), NetIQ (Aegis), Symantec (T-Logic, Altiris), and CA (Optinuity, Spectrum).

I think this is an excellent move by Microsoft. It will certainly make customers of both companies very happy. Microsoft and its customers gain an exceptional solution, in a discipline area that Microsoft was clearly lacking, and one which delivers many proven and exceptional benefits. For Opalis customers, it is probably a mixed bag. It will be a major change, but with Microsoft’s strength and stability, it is likely to be a positive outcome overall for Opalis customers.

This is, however, a huge blow for competitors, especially for the few large management vendors that have not yet acquired or built an ITPA solution or components, or whose own ITPA capabilities are less than stellar. For other large mgmt vendors with credible or better ITPA capabilities, this is both an opportunity and a threat. For mid-sized vendors that compete with Opalis or Microsoft Systems Center, and especially smaller vendors, this is a horrible result. Overall, most vendors will have to hustle to respond, although many will be unable to do so.

Meanwhile, Microsoft, Opalis, and their customers should be ecstatic with this deal. Few acquisitions are so clearly positive for the stakeholders as this.

You should be able to check out what the executives from both companies have to say in their blog posts:

• Blog post from Brad Anderson, Microsoft Corporate Vice President: http://blogs.technet.com/systemcenter/default.aspx
• Blog post from Todd DeLaughter, President & CEO of Opalis Software: http://www.opalis.com/blog.asp?id=1

Meanwhile, I will be expanding on the impact of this acquisition very soon with a full EMA Impact Brief. Keep your eyes out for that one – lots of significant implications for customer and competitors, without doubt!

Enter Intelligent Workload Management (IWM), which, according to the Novell press release is:

… Novell’s differentiated approach to Intelligent Workload Management [that] integrates identity and systems management capabilities into an application workload, thereby increasing the workload’s security and portability across physical, virtual and cloud environments

All I can say is … bravo Novell!

No, really. It is about time. Novell has exceptional capabilities in virtualization, automation, and service management; and it also adds critical capabilities for security management and compliance, especially around identity management.  These are all core values in what EMA calls ‘the responsible cloud’.

The EMA thesis, essentially, is that cloud computing has too many cowboys, and not enough sheriffs. Enter Novell, the “Doc” Holliday of the cloud landscape, with responsible capabilities for virtualization, automation, service management, and security and compliance.

IBM, Microsoft, Sun, and even Oracle might argue with Novell in some of its claims of uniqueness – after all, all of them have substantial capabilities in all these areas too.

However, regardless of some overreaching in their marketing, competitive threats, a nascent market, and gaps in actual product capability, Novell has an excellent opportunity to re-brand itself and deliver some exceptional capabilities to deliver on private cloud computing goals, and is as well positioned as any vendor to stake a claim to what they label ‘Intelligent Workload Management’.

Keep an eye out for EMA’s more detailed Impact Brief on this announcement. Very interesting stuff, without doubt.

Andi.

I tweeted that I think such predictions are useless when most CIOs must prove return on investment (ROI) for major IT projects in less than 6 months. Tajeshwar got me thinking more deeply about this idea with his reply:

“cio demanding roi<6 mnths r taking tactical view;3 year tech horizon must for taking strategic view & decisions”

Indeed, this is emblematic of a really interesting challenge for CIOs.

The demand for a rapid ROI, typically less than 6 months, and in some cases shorter, is a fact for today’s CIO – even more so today than before the global economic downturn. I firmly believe that CIOs demanding ROI in less than 6 months are simply realizing and reacting to the modern reality that IT can no longer be a pure cost center. Ask almost any CIO, and you will know that the ‘blue sky’ IT projects that delivered results in 2-3 year timeframes are a thing of the past.

However, as Tajeshwar implied, this demand works directly counter to the mandate for great CIOs to think and act strategically, executing on a long-term corporate vision. The same CIOs that are trying to contain or reduce costs – essentially a ‘cost center’ approach – must also be acting to make IT a strategic asset.

This does not mean that strategic CIOs are dead, or even a dying breed. On the contrary, the ability to accurately envision future trends and get a head start on competitors is perhaps more important than ever, because the rate of change in IT is so much faster, and the barriers to entry for new technology innovations seem to be always decreasing.

So to be a great CIO you need to act tactically, with projects that contain costs and deliver ROI in less than 6 months; yet also provide the business with a strategic launchpad for innovation, competitive advantage, and shareholder value.

What sort of projects can do this?

How about:

Virtualization

This is perhaps low-hanging fruit. My research for EMA clearly shows the key outcomes of virtualization are well divided between short-term ROI and long-term strategic benefits.

For example, in the short-term, virtualization reduces hardware, power, cooling, administration, rent and even software costs. Around 90% of enterprises report that it delivers real, measurable cost savings. Loading up 15 VMs or more on each physical server, allowing admins to manage on average a 10% greater workload, saving an average of $200 per system on administration costs, adding as little as $37 for each new VM in administrator staff costs (up to 28 times less than a physical system), and reducing power costs by an average of 17% are rapid and significant ROI values.

In the long-term, faster system provisioning helps bring products and services to market faster, better DR capabilities provide a strategic defense against disasters and epidemics, and better workload and resource balancing provides faster response times and better customer service – a range of strong strategic opportunities.

IT Process Automation (ITPA)

In the short-term, EMA research has shown that sites with ITPA improve their MTTR, provide almost 65 hours extra availability per year for 24×7 operations, and sites with ITPA (typically larger data centers) save on average around $500,000 more per year on staff costs alone than sites without it (easily offsetting marginally higher staff salaries). These outcomes all provide substantial short-term ROI.

Meanwhile, 95% of enterprises report that ITPA achieves one or more strategic goals, such as improving the ability to adapt to rapid change (like rapidly integrating M&A), freeing up high-level staff, providing better security and compliance, reducing business and IT complexity, reducing human errors, and integrating with best practices. Moreover, 76% report that ITPA helps achieve 2 or more of these goals, and 55% report it helps achieve 3 or more. ITPA also correlates with an overall increase in IT maturity.

Lifecycle Management

In published EMA case studies, automated lifecycle management reduced regular maintenance windows for 50 systems from 2-3 days to just 10 minutes each, and cut the cost of  distribution of a new version of Microsoft Office from $90,000 to just $30,000. It also can help to reduce overall software license costs, allocate and reuse hardware more effectively, improve end user uptime, and reduce or eliminate the (often substantial) travel, staff, and downtime costs of desk-side visits to install new software or fix problems.

EMA research also shows that automated lifecycle management helps to achieve strategic objectives. It provides faster and better service to end users (and ultimately therefore to customers), enables IT and business staff to be more productive, lets business users take advantage of new software and systems much faster, provides essential compliance reporting, and maintains strategic security values.

Part of the reason that these technologies are both tactical and strategic is that they can all be implemented in short, sharp, phases that deliver fast and specific results, while establishing a technology basis that can be leveraged – reused, over and over, in multiple new ways – to deliver strategic benefits with little or no additional cost.

For a great CIO, such technologies are invaluable. They show fast results, justifying budgets and building confidence; yet they deliver technologies they can continue to leverage for better and better strategic outcomes.

All of which meets the needs of today’s CIOs much better than blue-sky, multi-year, technology dreamings.

Good for them!

What I liked most was that they are trying to break down the barriers between systems and security management. Certainly this is something that I discuss regularly with enterprises – the need to stop focusing on silo-based management, and perhaps even more importantly, to stop pandering to silo-focused low-level managers. Almost all of the CIOs, VPs, and IT Directors who I talk with are critically aware of the problems these silos cause – including human errors, resource inefficiencies, security problems, and higher costs.

This is also a constant discussion I have within EMA, especially with the lead of our security practice, Scott Crawford – a brilliant mind on security (amongst many other subjects) who constantly thinks about security in ways I never could or would. We work and publish together on this topic frequently. Indeed, it has come up again in our latest research, which shows that security and risk management are a fundamental requirement for cloud computing – or what EMA calls the ‘Responsible Cloud’.

The upshot of all these conversations is simple – security management and systems management are not, cannot, and should not be completely separate. Not in human terms, not in processes, and not in technologies. Without doubt, anyone in a large enterprise who has ever tried to implement a patch, a configuration change, a firewall update, a software release, or a hundred other data center changes will attest to this in a heartbeat.

Of course (as Scott rightly pointed out when I last spoke with him about this), we will always need security experts, and systems experts – the two disciplines are not the same, and we will always need deep domain expertise in each. So I am not advocating complete convergence. But we need more software tools that provide integration and interoperability that allow these professional to work more effectively together.

While multi-function vendors like CA, Symantec, IBM, and others have the product portfolio to approach these cross-silo problems holistically, there are few ‘best of breed’ vendors thinking this way. Of course, Tripwire and the ever-inspiring Gene Kim (who I have sadly never met) spring to mind for me; so would Configuresoft (although now as part of EMC Ionix, hardly a niche vendor), and the indefatigable Dennis Moreau. Both inspire their teams, technologies, and customers by championing a fundamental understanding that systems and security cannot, at their heart, be completely separated.

(As an aside, these two seem like they would have been incredibly compelling arch-enemies in some ubergeek superhero genre – although I would never want to choose which should be the hero and which the villain!)

I must say that, so far at least, I don’t know the product design team from Reflex personally – guys like Hezi Moore, Aaron Bawcom, and Mike Wronski – as well as I do Dennis or Gene. However, I do know that they all have very credible security chops. Plus, one thing is clear to me.

They get it. They really get it.

And that in itself is a thing of rare beauty.

Deliberately designing functionality that addresses both security and systems management – like functional isolation, integrated access control, change segregation, granular audit trails, policy based management, and role-based access – into a systems management toolset is a rare feat, especially in startup and niche products. It is something I look for all the time, because my enterprise clients often demand it. Sadly, all too often I fail to find it – and I am not even a real security wonk! When I do, I am pleasantly surprised. When I see deep thought going into the security value of a systems management product, I am almost ecstatic.

Unfortunately, the challenge for vendors like Reflex and Tripwire (as it was for Configuresoft, and perhaps is still for EMC, Symantec, etc.) is to find customers that value this synergy. While most high-level IT execs understand this imperative, their holistic view frequently does not translate to many of their lower-level managers, or to many functional IT practitioners.

Of course, there are plenty of departmental ops managers and security managers who do get it. They strive to connect their teams with other groups, driving greater business efficiency and effectiveness as a result. However, unfortunately, many do not, instead focusing on protecting their small empires, walling themselves off from integrated management and cross-functional resourcing.

Similarly, many positive-minded individual technicians will actively seek out cross-skilling opportunities, recognizing that it makes them not just more useful but also more valuable, and more indispensable. However, many practitioners (both security and ops) can be just as bad as the most myopic managers (who they often work for), dogmatically eschewing integrated management tools and processes, seeing them as threats to their own personal domains of control.

Sad but true, best practices like breaking down IT management silos are not always adopted.

Fortunately, vendors like Reflex and Tripwire that have expertise and passion in both ops and security (and – shameless plug – trusted advisors like EMA, which is big enough to have experts in both disciplines, yet small enough that we still work together), are trying to break down these barriers.

And more power to them. They serve their clients much better by promoting the undeniable facts that security values are critical to systems management, and systems management is critical to security.

The survey revealed some very interesting data, with a very well thought out instrument and a quality sample – 300 respondents (100 each from the US, EMEA, and Asia Pacific) with at least 500 employees in the US (250 in the UK, France, Germany, Australia, Singapore and India), and all with a current or planned investment in server virtualization.

A number of data points stand out for me:

• The balance of Test/Dev implementations vs. Production continues to reflect EMA data. While production is still lagging behind test and dev as a use case, virtualization for mission-critical production is only slightly behind test and dev overall – and within the margin of error in most cases. This is good news, as enterprises clearly continue to grow real, production use cases.  It was interesting to see the differences between US and EMEA/APAC on this data point too, something EMA has not broken out in our published reports.
• Microsoft and VMware are neck and neck in enterprises’ plans for server virtualization deployments over the next 18 months. I was called crazy when my 2008 EMA research pointed to a 32% growth rate for Microsoft Hyper-V into 2009, trailing only VMware; yet here we are in 2009, and according to this new study, through 2010/11 that is going up to 49%. So who is crazy now?
• The strong growth for endpoint (desktop, application) virtualization reflects EMA data very well. It also highlights where enterprises and vendors should be heading with management technologies. It is still early days, but there are  a lot of gaps in integrated management for physical and virtual endpoints. So it is clear that this (probably even more than cloud service management) is going to be the next big problem for IT management.
• The percentage of IT services planned to be virtualized over 18 months is growing well. However, just as EMA has predicted, virtualization will remain at only around 50% of service deployments even through 2011, so there will continue to be substantial physical deployments. This reinforces my consistent (and insistent) position that effective management of virtualization must integrate both physical and virtual systems management
• Effective management continues to be elusive. EMA’s research showed this in 2006, 2008, and 2009, and this new data (with some reservations) shows the same. However, while tThe majority (64%) of enterprises rank themselves as extremely or very effective at managing virtualization, and believe they are getting better, I am skeptical. I contend many of those are overestimating their abilities (see my next points).
• Virtualization clearly increases complexity, and is clearly more difficult to manage. I felt like I was tilting at windmills when I published this opinion in 2006 and in 2008, contrary to common perceptions that virtualization made everything easier. I was certainly a lone voice, but as it turned out, a prescient one. It is great to see it being recognized more broadly, finally.
• Human issues continue to be major problems – especially skills and resourcing. EMA has found the same to be true, consistently, for many years. As recently as last week, I spoke with David Marshall of VMBlog and InfoWorld about how this continues to be a problem. This makes management tools even more important – to embed knowledge, define and execute policy, and automate routine work to free up resources.
• These data points all increase my doubt that enterprises are really being better at managing their virtual environments. It seems contradictory to me that this survey shows virtualization is more complex, management is the top inhibitor to ROI, and skills are still lacking, yet most enterprises think they are being very or extremely effective at it. Even though tool usage is more integrated and automated than it has been, this does not make sense. I am instead convinced that enterprises are really overestimating their abilities.

This is just a very small sample of the interesting data in this survey – there are more than 50 pages in the slide deck I reviewed ahead of the webcast. I encourage you to check out the webcast.  If you get in touch with HP, I am sure they will point you in the right direction; or check back here, and I will post the link when it is up.

Andi.