If you read the Windows Live blog yesterday, you’d be forgiven for thinking Microsoft’s Hotmail team had been surprised by a wonderful piece of research into webmail spam performance.

“Hotmail has come a long way in spam protection and is now among the best in the industry in keeping spam out of your inbox,” the company’s group program manager, Dick Craddock, writes. “Our own internal metrics, customer feedback, and even a recent third-party report confirms that no mail service offers better protection than Hotmail.”

That’s quite a big claim, given Hotmail has something of a reputation as a spam magnet. But Craddock goes on to highlight the results of this seemingly independent report:

“As much as we invest in our own telemetry and instrumentation to understand the spam problem, sometimes it’s nice to get an outside perspective. Cascade Insights gave us just that recently with a comprehensive study of the major email services to see how each performed in the face of incoming spam.”

Here’s what Microsoft’s Craddock and the 14-page report – which is linked to from the blog – don’t tell you. The research was commissioned and paid for by Microsoft. Microsoft chose the webmail services to be tested, Microsoft had right of veto over publication of the results. If Microsoft had come second or third to Gmail and Yahoo in the spam test, you can bet your mortgage we wouldn’t have heard about it.

There’s nothing wrong with companies commissioning their own research – for internal use or otherwise – provided they are up-front and transparent about what they’ve done

How can I be so certain? Because in 2009, Microsoft commissioned the same research firm – Cascade Insights – to do a similar piece of research. Hotmail came second to Gmail. Did Microsoft publish the results? “No, I don’t believe it did,” Cascade’s Scott Swigart told me on the telephone this afternoon. “Instead of publishing it, it used it [the research] internally.”

Now, there’s nothing wrong with companies commissioning their own research – for internal use or otherwise – provided they are up-front and transparent about what they’ve done. Nowhere in Microsoft’s blog post nor the full Cascade report does it mention Microsoft paid for the report. Indeed, I’d argue the carefully worded blog posts attempts to give the impression it was entirely independent.

“We were excited to see an analyst go deep on spam and compare the different webmail providers, so we’ve paid Cascade for rights to access and distribute their private report and methodology,” Craddock writes, as if the findings suddenly appeared out of nowhere. Cascade Insights insists it had full control over the test methodology.

Industry leading?

Microsoft’s claim that “nobody offers better protection than Hotmail” is also questionable.  The latest report shows that Hotmail is statistically no better than Gmail at spam protection – both let 49% of spam through to the inbox in Cascade’s tests, when rounded up to the nearest per cent. Yes, Hotmail very slightly edged it  – 48.57% of spam let through, compared to Gmail’s 48.88% – but considering it came second to Gmail in 2009, the Google spam protection arguably has a better long-term record.

And look at those percentages. Hotmail’s test inbox was almost half-full of spam. Yes, the tests were effectively creating a honeypot scenario, where a fake email address was set up and then plastered across the web to attract the spambots. But a detection rate of just over 50% is hardly anything to boast about. And it’s much worse than the “3% for a typical Hotmail inbox” that Craddock boasts of at the very top of his post.