Labs

Security suites

Testing security software is a unique challenge. Most types of software are self-contained: from spreadsheets to games, everything you need to get a clear picture of each package's abilities and limitations is right in front of you.

With security software, though, only half of what you need is in the box. Sure, the software may install and sit happily in your System Tray, but to get a real idea of its capabilities you need to expose it to genuine threats. So we've put each security suite in this month's Labs through a range of different exercises using genuine malware, carefully selected to represent as accurately as possible the real threats facing your PC.

We've also taken a series of measurements and timings to discover exactly what impact each package has on available system resources. You'll find key results over the next three pages, after which each suite receives a detailed written review, revealing its particular strengths and weaknesses. At the bottom of each review, you'll find a star rating out of six for Performance, Ease of Use, Value for Money and a final Overall rating calculated by combining these.

The tests

Each of the security packages is deployed on a clean installation of Windows Vista Home Basic. We use physical hardware rather than virtualisation to ensure real-world results, and a router-based ADSL internet connection exactly like a home PC. Default installation options are used and, once the packages are installed, we take no special steps to configure them beyond ensuring the most recent updates are applied to both Vista and the security package.

Performance

Our Performance rating measures how effectively each suite protects your PC from attacks. We arrive at this rating by carrying out a number of practical tests.

The first test determines each package's ability to identify files containing malware. We test this using a collection of executable viruses, trojans, keyloggers and other file-based threats. We download each file in the collection from our own private server via two routes: once via HTTP and once by email, using the Windows Mail client. In the wild, many malware items are distributed via only one of these avenues, but if a security package overlooks a known threat coming in via an unexpected route then that indicates a potential future point of vulnerability in your machine.

Alongside downloadable files and email attachments, a growing threat are browser exploits - security breaches that allow a website to download and run files on your computer without your knowledge. We therefore visit a range of sites known to host browser exploits and observe how - or indeed whether - each security package intercepts the download. We simulate real-world conditions by visiting sites recently reported to the "badware clearing house" at www.stopbadware.org, and which our own tests confirm to be hosting exploits.

Another threat is phishing, whereby thieves use forged emails and websites to trick users into handing over their personal details - often including bank details and other sensitive information - to crooks. Phishing scams are normally carried out by means of spam emails, so to test how effectively each package protects you against this type of attack we forward a range of real-life phishing emails - received within the previous 24 hours - to our Windows Mail account and note how and when, if ever, the security software prevents us from following the enclosed directions.

Next, we test each package's integrated firewall: this is done with the GFI Security Scanner tool, launched at each package from a remote computer on the same internal network. This enables us to see how the software protects your computer, without the benefit of any extra protection that may be provided by a router or other external circumstances. GFI reports which ports are accessible, and whether the system is exposing any known vulnerabilities, be they serious (such as would leave a computer open to compromise) or minor (such as cached information being available to outsiders). The results of this test don't translate into straightforward numeric terms, so they're not included in our results charts, but we detail each package's performance in its written review and factor it into the final score.

The final test contributing to the Performance score is of each package's spam filter. We take a collection of around 1,300 emails, of which more than 1,200 are spam, deliver them to Windows Mail and note what proportion of the spam is correctly marked as such (none of the packages wrongly marked any normal mail as spam). We perform this test twice, once with the standard Windows Mail junk filter enabled and once without, with the aim of simulating other clients such as Outlook Express on Windows XP. This is a harsh test, with no opportunity for training, and gives only a measure of comparative out-of-the-box performance: we'd expect the results from all packages to improve over time.

In deriving a final Performance rating from these results, we place the greatest emphasis on malware detection, web-exploit detection and firewall performance, since these are the critical areas in which failure could really spoil your day. Each package's record with spam and phishing emails is also factored in, but is weighted less heavily.

Several packages also offer peripheral features, such as backup and system tune-up functions: these are outside the scope of a comparative test and we don't factor them into the Performance score. We give details in the feature table below, though, so you can see at a glance just what you're getting in each package.

Ease of Use

All of this month's security packages are mature products, and none is discouragingly difficult to use. Nevertheless, some are friendlier, clearer and more responsive than others, and this contributes to the Ease of Use score. We're also concerned with unnecessary intrusions, so in addition to our own observations we carry out a "false positives" test in which we launch 35 legitimate applications that a security suite could mistakenly identify as malicious. These include network-administration programs that might look like worms, typing tutors that act like key-logging programs, screen-capture utilities that resemble surveillance packages and so on.

The good news is that, in this month's test, not one package returned a single genuine false positive - in other words, none of these legitimate packages was incorrectly flagged as dangerous. However, some packages pestered us with requesters before allowing our applications to run, while others silently did the right thing. This difference in user experience also contributes to the Ease of Use score.

The final factor in the Ease of Use score is the impact each package has on your system resources. To quantify this, we take two measurements. First, we time how long Windows takes to boot on a system with a default installation of each package, and compare this to the time it takes for a clean system to do the same. Since Vista is a complex operating system, we disable a number of services, including SuperFetch and Windows Update, which can otherwise affect boot times regardless of third-party software. This permits a straightforward comparison between the demands of each package. Since some modular packages continue to load and tie up the system after the desktop appears, we stop timing only once the CPU usage drops to less than 10% and remains there for ten seconds.

Second, we make a note of the total available memory when the system becomes available and, once more, compare it to the memory available for a clean system.

Value for Money

When you buy any of the security products in this month's Labs, you'll be entitled to updates for a limited time, ordinarily annually. Since malware is constantly evolving, regular updates are necessary if you're to remain protected, so security software is effectively licensed rather than sold. We weigh up the various licensing schemes offered by each supplier to reach an overall indication of comparative value. The Value for Money score reflects this, but also takes into account each package's scores for Performance and Ease of Use: a cheap security suite is, after all, still poor value if it doesn't deliver effective protection. It's always worth remembering, though, that many suppliers will have occasional special offers, and boxed editions may be available at a lower price in some of the high-street stores.

Overall

Finally, we award an Overall score to each package. This score is an average of the other three scores, although due to rounding it may appear slightly higher or lower than expected.