How should vendors and others fight FUD?

I, along with many others, witnessed this week (or was it last week?) a public squabble between two well-known vendors in the virtualization market. Of course, this is nothing new. The whole world has been watching as Adobe attacked Apple (and Apple responded) over Flash support on the iPad. Before that, of course, America was regaled by the amusing Verizon campaign attacking AT&T (‘There’s a map for that’). Last year the gloves were well and truly off between VMware and Microsoft over Hyper-V bluescreens. Apple and Microsoft were at it last year when both ran their ‘I’m a PC’ ads (each side taking very different interpretations), and they were at it again just this week.

Now, a lot of these ‘fights’ seem to be what is referred to in rugby circles as ‘handbags at 10 paces’ – a long-distance squabble with a lot of pushing and posturing and preening, but little actual contact, and no actual damage. But occasionally these fights – like in rugby – get very serious, with real hits on both sides, and a lot of very real damage (to reputations, customers, influence, sales, and more). Unfortunately, in both cases, the damage seems to be to both sides of the stoush.

This impact is especially apparent when it is influencers (bloggers, analysts, media, etc.) who are (re)publishing the attacks. After all, many influencers are driven by a marginally slower version of the ’24-hour news cycle’ – the need to get content to print quickly so they get more eyeballs as the first to ‘break’ a story. This means that they can end up accepting any reasonable story at face value. Some of the more provocative authors seem unfortunately to do this more consciously, and have led to the sad resurgence of ‘yellow journalism’ (especially driven by the pay-per-click model for Web site advertising). However, even the most honest and scrupulous of authors can occasionally publish excitedly about unconfirmed future releases, saucy though unfounded rumours, or interesting secrets and leaks. Or they may simply end up for one of many reasons publishing content that is not untrue, per se, but is simply not evenly balanced.

All of this has spurred me to think more deeply about how to react to vendor FUD (Fear, Uncertainty, Doubt). The way I see it, there are three core stakeholders in these mix-ups – the vendor(s), the enterprise customers/buyers, and various commentators/influencers.

So how should these three constituents react to unbridled FUD from software vendors?

Vendor Responses

The first decision for a vendor is whether to respond at all. We should never underestimate this option – depending on the case, it is frequently better to simply take the high road. After all, as the old saying goes, ‘never wrestle with a pig – you both end up covered in s**t, but the pig enjoys it’.

If a vendor does choose to respond, the next question is, how strong should their response be? They must decide between a subtle approach, and attacking the issue head-on. It is one thing to whisper snide remarks in dark alleys, another to come straight out and state the facts as you see them, and something entirely different again to really go in ‘boots and all’. If they decide to go big, they can choose anything from an attack blog to a full-blow PR campaign – but each has significant issues.

Social media is clearly an option, but is a vendor blog or tweet really going to be effective, or is it just singing to the choir? A full-blown PR campaign attracts big attention, but gives oxygen to the fire, and risks both putting the competition on a pedestal and making the responder look petty. Using personal relationships to bring up the issues one-on-one with customers is more subtle and ‘high-road’, but will never reach everyone that has already heard the FUD.

Getting influencers to balance the table for you is an excellent outcome if it can be achieved. However, it is not always an easy option. Some influencers have already made up their mind by the time they publish; some may simply have moved on from that story and have no desire to go back to it; still others may just not be interested in talking to a vendor. How a vendor works with influencers to help provide balance after they have already published a damaging story is a perennial issue. If anyone figures this out definitively, let me know!

Buyer Responses

For the CIO and other enterprise buyers, the real problem is not so much the fight itself, as much as knowing what to believe. So the first response should always be to try to find alternative viewpoints from a wide variety of opinions.

Certainly gathering multiple analyst opinions is a great option. If you are a subscriber with one or more analysts, and the issues raised are important enough to burn through your subscription time & dollars, set up a quick call to discuss directly. If it is not a burning issue, wait and see if they publish a research note on the topic (handy hint: you can often get these for notes free if a vendor decides to license them – but be aware that when vendors do license analyst reports, it is mostly because they are positive about that vendor, so take it all with a grain of salt). If you are not a subscriber, then try just reaching out on Twitter. Some analysts will give up 140 characters (or even more) for free, although of course many will not.

Beyond the independent analyst community, look toward different media sources so you are not just reading one side of the story, or even just what one source is publishing. Some stories build a life of their own, some journals and blogs are part of a single network (and may be influenced collectively). Some individual authors or outlets will have a decided bias – whether permanently or temporarily, intentionally or accidentally – so look for another source with a different bias. And remember that almost no human communication can be without any bias at all.

Another excellent source for alternate viewpoints is your peers both in IT and business. Reach out to your user groups, contacts from conferences, internal colleagues and former colleagues/peers at other companies. To find new peers and new opinions, try using social media like LinkedIn groups, Facebook, Google groups, Twitter etc. Post invitations to discuss the issues on your own blog if you have one; if you don’t, then comment on other people’s blogs with your own opinions and questions.

You can also reach out to the vendor themselves, as well as their competitors, and get them all to respond directly. Try connecting on LinkedIn, Twitter, or their blog; or call or e-mail a sales rep or their marketing people. Believe me, they will love to hear from you, and be more than happy to give an alternate opinion – perhaps over lunch or a beer! You will at least then get both sides of the story. Even though both sides may be horribly biased, you can normally figure out some middle ground for yourself.

Influencer Responses

It is simply a fact of life that influencers – including analysts, media, bloggers, tweeters, and more – can be just as vulnerable to a well-crafted message as any other humans, and can buy into well-crafted vendor FUD just like the rest of us. However, most influencers also realize that it is critically important to their credibility and livelihood to present a balanced view.

The most important requirement then is probably the most basic, and a matter or course for most professional writers – check your facts with reliable sources. Go beyond an initial or single source, even if they are unimpeachable – there is always the chance that a single source has been honestly misled themselves, or even that they are simply not well-informed of all sides of an issue.

When writing about one vendor – especially based on their own releases or references – you should also actively  try to find out what their competitors are saying. Again, their sales and marketing people will be more than happy to talk with you.

Influencers especially need to make sure their content is defendable from all angles, as this independence is fundamental to their credibility and reputation. When I wrote as an analyst, I always made sure I was able to defend my content against the most rigorous accusations from all sides – the enterprise users I wrote for, the vendors I worked with, the vendors they competed with, my peers within the firm, and the broader influencer community. Many writers even have a formal process for this peer review, and it is certainly a best practice among the larger analyst firms.

If, despite your honest efforts and intentions, you find that you have inadvertently published some content that lacks balance or independence, then please make the effort to redress the balance. It is only fair to show the opposite opinion. That does not always mean re-writing or publishing a correction; on the contrary, it may be a very positive opportunity to publish a new article, blog, or research note as a follow-up, which can even magnify the number of readers, hits, and content sales being driven by the one issue.

Influencers should especially treat vendor leaks, rumours, and exclusives very cautiously. When given references, you should try to find references that are not recommended by the vendor, and make sure to verify their information with other independent third parties.

There Must Be Other Options

These are just my top-of-mind thoughts on the topic. To be honest, I am really not sure what options are best for any of these stakeholders (or even if there are other stakeholders that I am not focused on). Moreover, I am sure I have missed some other good options. I have discussed some of this online, but I am far from convinced of the efficacy of any one approach.

Moreover, there is one unfortunate caveat to all the above: when you are inside the IT echo chamber, it can be difficult to even find alternate opinions. As we all talk amongst ourselves, we repeat to each other what we have all heard from each other, so we all risk just going with the flow. No matter where you stand in this regard, it is always important to keep looking. Other voices are almost always out there.

Of course, the best responses to FUD will vary from one situation to another, but are there some that are always appropriate? Are there some that will never work? Are some responses just fundamentally wrong? Even if some options are odious and unattractive, are they nevertheless fair game if they are still effective (as some believe about political attack ads)?

Feel free to comment here, or just hit me up on Twitter (@AndiMann), and add your own ideas. Maybe I will run a survey to see if I can get more input that way.

Because there has to be a better way to deal with FUD than handbags at 10 paces.

Most devops discussions are missing the target

I am hearing a lot about the rise of a concept called ‘devops’ – a mashup of ‘development’ and ‘operations’. I am not at all an expert in this area, but from what I can tell, devops is aimed at streamlining rapidly iterative application delivery to allow for greater development and business agility. Devops aims to achieve this by breaking down the barriers – human, process, and technology – between application development and system operations.

Interestingly, the concept is new enough that, as I write this, there is not even an entry for it in Wikipedia yet. I did find a blog by Damon Edwards (on Twitter – @damonedwards) very useful though, as he explains the age-old disconnects between application developers ‘throwing software over the wall’, and ops who are painfully resistant to change. James Urquhart (@jamesurquhart ) blogged very recently on the concept too , and again provided some very helpful content. Conversing online with them and others also helped me to formulate some more concrete ideas about devops – or at least some more concrete questions.

My interest was especially piqued when I understood how closely devops is connected to virtualization, cloud, and automation – my core interests:

• Cloud – Devops has antecedents in ‘rogue’ developers (or developers from smaller shops) using cloud resources (IaaS, PaaS) for new projects, and will benefit greatly from cloud-based development and deployment, as cloud providers do not impose the restrictions of internal change-averse ops teams, and developers can essentially manage their own ops requirements instead.
• Virtualization – In-house devops (which needs more heavy lifting) is greatly assisted by virtualization, as virtual machines become the new base unit for application packaging, avoiding application rollout failures caused by incompatibility between the test and production environments (hardware, OS, middleware, etc.).
• Automation – In-house devops is also greatly facilitated by automation, which can use standard workflows to automatically provision and configure these complete application VMs, as well as backup and restore VMs, allowing complex composite application deployment and rollback at the click of a mouse.

Clearly devops has many very attractive outcomes – drive agile business, reduce delays, smooth application releases, deliver value faster. It is a very seductive idea. Who wouldn’t want it?

However, most of the writings I see about devops are really about dev, not ops. As a result, they don’t really capture the whole story of the application lifecycle. They justify devops as an antidote to the problems that ops are causing – slowing down release cycles, imposing arbitrary rules, screwing up deployments, killing developer productivity, hacking manual scripts and configs, stopping the business from being agile – but fail to recognize both the failings of developers that contribute to the problems, and the role of operations in delivering critical business outcomes during the application delivery lifecycle.

On the contrary, discussions mainly focus on how developers can sideline or change operations, positioning devops as the lone hero in the battle against inefficiency, as application developers fix all the problems (!) by controlling or automating key release management operations like provisioning, deployment, integration, patching, and software update. Meanwhile, ops are marginalised, along with their timesinks and roadblocks, satisfying the needs of an agile and rapidly changing business.

See – seductive, isn’t it?

Yet this seems to me (as a former op) fundamentally flawed, a development-centric neologism based on an incomplete understanding of the real purpose and role of IT operations, or of operations’ history in the development of ‘agile’ IT.

The way I see it, devops misses that target on how IT ops serve business needs too, and seems to gloss over ‘coal face’ realities like:

• Who handles ongoing support, especially software update for the unrestrained sprawl of non-standard systems and components.
• Who ensures each new application doesn’t interfere with existing and especially legacy systems (and networks, storage, etc.)?
• Who handles integration with common production systems that cannot be encapsulated in a VM, like storage arrays (NAS, SAN), networking fabrics, facilities, etc.
• Who handles impact analysis, change control and rollback planning to ensure deployment risk is understood and mitigated?
• Who is responsible for cost containment and asset rationalization, when devops keeps rolling out new systems and applications?
• Who ensures reporting, compliance, data updates, log maintenance, Db administration, etc. are built into the applications, and integrated with standard management tools?
• Who will assure functional isolation, role-based access controls, change auditing, event management, and configuration control to secure applications, data, and compliance?

Because, if you have ever worked with both ops and apps, you know it is not going to be apps.

Now, in defence of devops, I am sure it is being implemented and conferring major benefits, especially in small organizations with little IT management discipline. I am sure the supporters of devops have some positive goals in mind too. What’s more, it is addressing a very real problem – ops really should spend more time on better processes and controls than in ‘daily deployment muck’.

However, devops should be a two-way street. As a former op, I know that the apps team have to pull their weight too, by addressing gaps like:

• Including ops during the design process, so applications are built to work with standard ops tools
• Taking ops input on deployment, so applications will go in cleanly without disrupting other users
• Working with ops on capacity and scalability requirements, so they can keep supporting it when it grows
• Implementing ops’ critical needs for logging, isolation, identity management, configuration needs, and secure interfaces so the app can be secure and compliant
• Giving ops some advance insight into applications, especially during test and QA, so they can start to prepare for them before they come over the wall
• Allowing ops to contribute to better application design, deployment, and management; that ops can do more for the release cycle and ongoing management than just ‘manipulating APIs’

See, ops do enable business – and agile business at that – by ensuring that new applications coming into an existing complex environment are safe, secure, reliable, integrated, and responsive, regardless of how complex IT is, or how many moving parts there are. Devops seems to miss this important detail.

So I am sceptical of how devops will work in large, well-run IT environments with important and necessary operational controls, especially the > 60% of organizations that are committed to ITIL best practices (like formal and integrated management of change, configuration, release, assets, etc.).

After all, ‘agile’ does not magically obviate the need to identify and prevent bad changes, to reject apps that breach operational compliance, to ensure each new application adheres to standards, or to prevent uncontrolled sprawl of heterogeneous software.

I still have a lot to think about on this topic, and am trying to keep an open mind. But my best guess right now is that, for enterprises at least, devops either will not take hold or will not last. It seems most likely to be instead, at best, a transitory state on the path to a ‘new normal’. As with all ‘revolutions’, it has started outside IT ops, yet I expect will eventually co-opt and migrate wholly to operations in some form. Once the revolutionaries in development understand how many business needs besides agility actually require  routine, process, management, and controls, they will back away from devops the same way they backed away from ownership in other IT revolutions – like the deployment of mini computers, desktops, and web applications.

If it does turn out this way – don’t worry. Operations will again dutifully take the reins, and clean up the mess that devops will leave behind. Because that is what ops do – they manage what they are given, and keep the business running, regardless of the mess that gets thrown over the wall at them.

In any case, whether devops takes root or not, hopefully we will all learn something about cooperation, automation, agility, and control. Because all stakeholders in the devops discussion – development, operations, and business owners – could benefit from that.

As of Wednesday this week (2/24), I am now at one of the very best IT management software vendors, CA Inc., where I am leading product marketing for their — our — virtualization management solutions.

In many ways, this was an incredibly difficult decision. EMA is a truly excellent place to work, and the role of an industry analyst was fascinating and fulfilling. The people I worked with and for are some of the best minds in IT – always intellectually stimulating, and straight-out fun to be with. It was truly my privilege to get to know them all, and especially to help my clients and my team to be successful.

Yet this was also one of the easiest decisions I have made. I believe both virtualization and management deliver incredible IT and business benefits, and as virtualization becomes increasingly ubiquitous, management of virtual systems becomes increasingly critical. I have long considered CA a leader in physical and virtual systems management, and believe CA has a great opportunity to extend its leadership in virtualization management, by helping even more IT and business people to be even more successful. As a part of  CA now, I can not only be a part of that opportunity, but can be a significant author of that success.

Moreover, it allows me to indulge my passion for technology and my expertise in marketing in an in-depth, direct, and focused way, rather than the broad, ancillary, and essentially academic role of an industry analyst. I will be able to work directly with some the biggest and most successful companies and technologies, not just in the US, but around the globe. Plus, like EMA, CA also has some incredible minds who are some of the most fun people to hang out with too.

While some will see this a move (back) to ‘the dark side’, I have always considered analysts and vendors to be two sides of the same coin – helping IT to deliver business services in more effective and efficient ways. While some may say that I have ’sold out’ my integrity as an analyst, I have always considered my integrity to be a core and consistent value — and a non-negotiable one — regardless of my employer. While some may think I can no longer champion the best interests of enterprise IT like I did while I was an analyst, I believe the best software companies, and their best people, succeed and thrive specifically because they do exactly that.

As for this blog (and my Twitter feed), all my reasons for blogging and tweeting, and what I hope to achieve (both personally and professionally) with social media, are still the same as they were when I started. I therefore intend to continue writing and posting my personal opinions and insights about technology and other areas that interest me. After all, the areas I work with haven’t really changed, so I am still going to post about virtualization, systems management, data center operations, and cloud computing.

So although I cannot help but be informed by my current position and experience, my goal is to keep posting interesting and informed ideas, regardless of my employer. No doubt some people will stop reading — which is fine — but I still hope you will keep inspiring, contributing to, reading, commenting on, and arguing about these part-time musings of a full-time technologist.

The most recent example of such failures is the power outage at IaaS provider Rackspace’s London facility, but of course, we have seen this before from many public cloud providers – including Rackspace in particular, and not just once. SaaS provider Salesforce.com (and its PaaS arm, Force.com) has also had one outage already this year, an event that is far from unusual, and nothing new. Amazon, Yahoo, Microsoft, GoGrid, RIM, Twitter, Paypal and many others have also had substantial (and often repeated) outages.

There are some who dismiss these failures as one-offs, write off partial or short-term failures as too low-impact to matter, or just give poor DR a pass because it is the cloud, and we should not expect any better. Others reach to find semantic differences, calling it a service outage, an application failure, a facilities outage, a power outage, or a resource shortage. Some just redefine cloud to include only those services that did not go down this week (bonus points for adding a vainglorious reference to the ‘real cloud’ or ‘true cloud’).

YMMV, but I don’t see it that way at all. With so many repeated failures in so many cloud providers, these are not just one-off failures. They don’t just happen to isolated providers, they happen across the board. Regardless of the cause – the application, the facilities, the power supply, the lightning rod – an outage of a cloud service provider is still a cloud outage. And the definition of cloud I use is not dogmatic enough to exclude any of the providers that I have cited (and others), let alone define a ‘true cloud’.

So I see every reason to believe that downtime in the public cloud is not the exception, it is the rule; that outages in the public cloud are endemic, and they are systemic.

However, this judgement is absolute, not relative. Failure in one cloud provider may (and I believe does) implicate all cloud providers, but it does not imply downtime is more of a problem in the public cloud than in traditional enterprise IT. Indeed, there is a strong argument that enterprise IT has as many if not more outages, so uptime and availability is no worse in the public cloud than with traditional IT.

In fact, EMA research has shown average enterprise IT uptime is just ‘two nines’, at 99.5%. For a 24×7 system, that is over 50 minutes of downtime, each and every week. Contrast this with public cloud providers. Even with their problems, Amazon EC2 offers a “reasonable effort” to deliver an annual uptime of at least 99.95% – or about 5 minutes downtime per week – and offers a 10% credit for “eligible” breaches. Google guarantees ‘three nines’ (99.9%) uptime for its Premier Edition, or around 10 minutes downtime per week (although it promotes a study that claims an average downtime of 15 minutes a week). The Rackspace SLA promises network, HVAC, and power will be up 100%, though it does not guarantee server availability (beyond promising a 60 minute maximum repair window), and all promises exclude ‘scheduled maintenance’.

So for the average enterprise, ‘normal’ cloud computing outages, while endemic, can still be 5 to 10 times less frequent than in their own data centers.

However, it is not a black and white issue, not least because a focus on broad uptime percentages or on single instance failures ignores the huge nuance behind a single uptime number.

For example, many environments report ‘five nines’ (99.999%) or even 100% uptime – less than one second of unplanned downtime each day – for their critical systems by using processes and tools for high availability, fault tolerance, asset maintenance, live migration, etc. EMA has also found that best performers in Virtual Systems Management – 15% of enterprises – report an average of five nines uptime.

If they need to, enterprise CIOs can invest in technology to provide two, three, four or five nines uptime within their own data center. They can implement redundant hardware, HA and FT, multi-site replication, and more – if they want to pay for it. They can monitor for outages, know exactly when they happen, and react automatically to fix them immediately (or even use predictive analytics and automation tools to avoid them entirely). They can provide this as required, as a value-add to their business unit customers, or as an additional charge (or at least an exposed cost)  to the business to let them choose how critical their applications really are.

However, with the public cloud, neither the business nor the CIO has any real choice. With few or no management or automation tools, public cloud providers simply do not currently offer the same flexibility and accountability as internal IT. Without good management tools, no public cloud provider currently matches enterprise IT at the higher mission-critical reaches of availability.

So, this fight does not end in a knock-out for either side. As is common in the real world, nothing is black and white, but rather many shades of grey.

In the end, the solid achievements of public cloud providers, despite the bad press, does not absolve them of any blame or negate generalizations of downtime being endemic in the public cloud. However, the relatively poor performance of enterprise IT on average still does not ensure public cloud will be any better in any specific cases.

What this does show, however, is that CIOs who are planning to build their own private cloud have a surprisingly high bar to reach. They should not dismiss public cloud options out of hand, but rather should strongly consider whether they can realistically and cost-effectively meet the three, four, and even five nines that public cloud providers guarantee.

Anyone who knows me, knows that I had for a long time steered away from Social Media – blogs, Twitter, etc. – for a number of reasons. But I am of the increasing belief that many of my reasons were, are at least are becoming, invalid. For example:

• As a research analyst, my clients pay for my knowledge, expertise, and advice. I had thought that blogging would put some of that valuable content out for public consumption for free, devaluing my client’s investments. However, it is clear that what I put out as advice for my clients is quite separate from what I Tweet or blog. The former is formal, well studied, action-oriented, backed by empirical research, peer-reviewed, and subject of deep analysis. The latter is likely to be casual, off-the-cuff, opinion-oriented, with little data or analysis, and no peer review.  As a result, it is now clear to me that my formal research and analysis retains its value, even though I may post on similar or related topics in my blog.

• I saw Twitter and Blogs as very self-centered, uninteresting, without credibility, and generally not much use. So many words with such little meaning. This now-classic Penny Arcade comic (NSFW language) pretty much summed it up. But after reading Twitter for a few months, then using it, and discovering some excellent blogs from other Twitter users (and other analysts), I can see that some of these social media outlets are contemplative, interesting, well-crafted, and more than credible for their purpose. Besides, even though so many blogs are complete rubbish, mine didn’t have to be. Hopefully it isn’t.

• I could not see any business need for social media. On that, I was just plain wrong. Of course, social media doesn’t need to have a business purpose. I fully intend to blog on occasion about sports, cooking, TV, movies, music and a wide range of things in my interests, just as I occasionally Tweet about them. But social media also can have a business outcome. I know on Twitter I have expanded my circle, and am pleased my Twitter feed now reaches some excellent business contacts, and provides a wider audience for my analysis and research. It has also generated some very interesting discussions that have informed my research, and on more than one occasion, actually brought clients to my attention (or perhaps the other way around). I hope my blog will have a similar outreach, promote similar discussions, and even perhaps help generate business for me and my employer.

• I didn’t think I needed to. My clients had not asked me to, I had journal columns I wrote regularly as an outlet, and really didn’t think a blog would help. Well, since then I have noticed my clients and prospects increasingly paying attention to my competitors’ blogs (even as lame as some of them are! ), and have actually had some of my clients specifically ask me if I had a blog, and when I said that I didn’t, they have even told me that I should start one. Given different outlets have different content, it seemed that a blog could help me express my views outside my formal channels, help my clients understand more of my views, and help my readers understand the technologies I cover.

• I thought I had nothing to say on a blog. I am still not entirely sure that I have anything worth reading – time will tell – but I certainly feel I have something to say. My work allows me a certain type of outlet for my research, my analysis, and even some of my thoughts, such as short briefs, advisories, journal articles, conference presentations, and more. However, it does not allow me to publish the sort of freeform musings that I can on a blog. My blog will not replace my work output; it will provide an additional, personal, outlet.

• I thought I didn’t have time to blog. Well, to be honest, I still don’t. But that shouldn’t be a reason not to do something important.

So, why now? Pretty simple, really. I started using Twitter at the beginning of the year. Lurking at first, and then sceptical, I soon embraced it for all the good connections and discussions it has given me, personally and professionally.  For a while now, I have been an avid Twitterer, but I keep getting into the situation where I want to say more than 140 characters will allow. I see others expanding on their Twitter ideas in their blogs, so thought it would be a good thing to do.

So now that I have finally gotten around to blogging, I hope you enjoy it.

Andi.