Follow-up: Benchmarking Windows 7

Well, my last blog post certainly kicked up a storm. I'm glad so many people found it stimulating: I'm always interested to hear your responses.

But a few of you have raised good questions about the tests I used to compare performance between XP, Vista and Windows 7. So let me explain them in a bit more detail.

For the feature "Memory Laid Bare" in issue 169, I ran a series of tests, using the standard PC Pro benchmark suite, to find out how much RAM you need for peak Windows performance. One of my findings was that, on a given system, our benchmarks complete significantly more quickly under XP than Vista, regardless of how much memory is installed.

Unsurprisingly, it was GDI-heavy tasks that suffered the most. On the 2GB test system I used for that feature, our Microsoft Office test took 6:51 to complete in Windows XP; under Vista it slowed down to 8:07.

But calculation-intensive operations, such as video encoding, were also affected: our Canopus ProCoder exercise took 5:01 under XP but 5:11 under Vista. While that's a much smaller gap, it demonstrates that the OS affects efficiency even for seemingly CPU-bound tasks.

In light of these results, I was intrigued to discover that both of these applications perform identically in Vista and Windows 7 – especially with commentators consistently reporting that the new OS feels faster. To me, there's one obvious inference: that Microsoft has responded to criticism of Vista's performance by speeding up the interface while leaving the back-end fundamentally unchanged.

But of course, Windows 7 is still a work in progress, and while Microsoft is starting to nail down the feature set (see Barry Collins' reports on the new tools for IT admins, Device Stage, networking, touch controls, and user interface), its developers still have a year in hand to tidy up the code. They could yet surprise us by streamlining the internals to bring back XP-like levels of performance.

Then again, of course, they could spend that time inventing a truck-load of glitzy new services to slow the whole caboodle down even more.

Here's hoping they really have learnt from Vista.