The post was picked up on Slashdot and generated a lot of comment mostly from anti-Flash zealots and those who thought I was questioning the maths rather than the methodology (a survey commissioned by Adobe based on a small panel of opt-in users who were asked whether they could see various items of plug-in content complete with player download dialogs!).

However there was one particularly useful response…

…pointing interested parties in the direction of the “Rich Internet Statistics” site riastats.com. This is a single page site devoted to tracking the player penetration of rich internet application platforms and it outscores Adobe’s own tracking for a number of reasons:

• Figures are based on statistics derived without the need for a survey and from a much wider sample – at the time of writing around 1.5 million unique daily users over the previous 30 day period.
• You can embed some code to find results for your own site – which as James Ward the technical evangelist from Adobe pointed out is the statistic that really matters.
• You don’t have to dig down to find the crucial figures for reach based on the player version.
• You can see how Flash penetration compares to Java and Silverlight (mysteriously missing from Adobe’s analysis) and you can track changes over time with the Line chart.
• You can easily and interactively break down figures based on browser, operating system and language (and filter out stats from sites specifically about ria technologies)

Perhaps most important of all, the site itself is an excellent demonstration of why the Flash deniers and blockers are wrong (not that they’ll ever find out). Flash is capable of much more than irritating pop-up ads and riastats.com shows how it can be used to deliver content more attractively and, crucially, more effectively than HTML.

Riastats.com certainly isn’t perfect – the fact that it is Flash-based does suggest a certain in-built bias and it really ought to publish the sites where statistics are tracked and correct the typo in its title – and of course there’s no single statistical “truth”.

Having said that, what’s the answer? What are the riastats.com figures for Flash penetration?

At the time of writing (drum roll) riastats.com puts the figure for Flash player penetration at roughly 97% – and around 51% for Flash 10 content compared to Adobe’s 56%.

This 2% gap on the headline claim might seem insignificant but it’s not.

One of the problems of the 99% figure – a major plus if you’re Adobe – is that it is seen as effectively ubiquitous. 97% is getting there – and a real achievement and a massive advantage over rivals – but it’s definitely not ubiquitous.

This has important practical implications. In short don’t be tempted to provide Flash-based navigation or other content that is better handled via the truly ubiquitous HTML. Moreover don’t use Flash 10 content unless you have to and, if you’re a Linux-oriented site, be wary of using Flash at all – according to the current stats, 1-in-7 Linux users won’t see anything and less than 1-in-5 will see version 10 content.

These practical issues are important but, ultimately, the main reason that the riastats figure of around 97% is significantly different from Adobe’s figure of 99.0% is simple: it’s believable.

But should the claims be taken at face value?

Well let’s look at it in a bit more detail. At the time of writing there is a major headline:

“Flash content reaches 99.0% of Internet viewers”

followed by:  

“Adobe Flash Player is the world’s most pervasive software platform, used by over 2 million professionals and reaching 99.0% of Internet-enabled desktops in mature markets as well as a wide range of devices.”

“Over 2 million professionals!” Eat your heart out Dr Evil. Presumably they meant to say the Flash authoring tool is used by that number. Amusing, but hardly reassuring.

Sure enough look at the PC Penetration page and you’ll see that the latest estimate (December 2008) for Flash player penetration is 947 million users out of a total 956 million internet-connected devices. No doubt with the next quarterly update the 1 billion figure will have been smashed.

However things aren’t quite as straightforward as they look – as Adobe shows with its notes. 

The first of these reveals that the total number of PCs is based on a forecast made two years ago – an age in internet time. Already then the margin of error on numbers at least is enormous.

The second reveals that the figure is based on devices capable of reading Flash player 7 content. To be fair to Adobe they do give the penetration stats for different player releases and thanks to auto-updating the figure for the latest Flash player 10 is already around 55%. That’s pretty stunning in the timeframe but it’s not 99% (ie remember to target your SWFs at the lowest player capable of rendering your project).

The third note is the most significant:

“Total Player penetration is a calculation of the total number of PCs connected to the internet, multiplied by the weighted percentage of worldwide penetration from the Millward Brown study. This is an assumption made by Adobe.”

So what is the Millward Brown study? Well to begin with it’s not the fully independent survey you might assume as it was commissioned by Adobe.

Having said this, Adobe looks admirably open and helpful giving full details on the methodology page. You can even take the survey yourself – though I trust that the data from surveys originated from Adobe’s Flash Home page aren’t included!

I’m sure they’re not, but of course any internet-based research is automatically self-selecting – users who find and sign up to become members of the Lightspeed panel for rewards including music downloads will naturally tend to be home-based heavy internet users who have picked up more plug-ins than average as they go.

Moreover the survey works by showing animations and asking if you can see them. You are instructed not to download the plug-in when the dialog appears (why weren’t these disabled?), but I think it’s human nature to want to see what you’re missing and to hit OK rather than Cancel and especially on the first page where this is likely to happen which just so happens to be the Flash 10 page.

On top of which the underlying numbers on which such a major claim are built seem tiny with an apparent total survey sample size of 4,600 ie around 0.0005% of the suggested 956,000,000 total (and then weighted according to the CIA World Factbook!).  Adobe then seems to “assume” that these figures scale up neatly, but presumably Millward Brown doesn’t endorse that assumption. Moreover it’s only when you dig down that you discover that the figures give a signficant margin of error of around +/- 5% with 95% confidence.

So where does this all leave us?

As regular readers will know I’m a big fan of the power and potential of the Flash platform (though not of the majority of current Flash-based sites) and would like to see it spread as widely as possible. 

Moreover I’m sure that Flash is the most pervasive cross-platform web platform and if you throw in Flash Lite-enabled mobile devices, arguably the most pervasive software-lite platform. I’m certainly not disputing that it’s more popular than all other rival web platforms such as Java and so the best choice for most scenarios.

However in that Adobe headline both the seeming precision – “99.0%”- and the poorly-defined wooliness – “Internet viewers” – are clearly deceptive. 

More to the point is it really possible that 99% penetration could have been reached? Including Linux users? Including users at work? Including brand-new systems? Including my granny?

I guess the big test is whether the next set of figures from Adobe will finally break through the even more psychologically-significant 100% barrier.