Features

What do the following activities have in common: listening to music on your iPod, downloading an application installer, flicking through someone's holiday photos online, sending an archive over email, watching a DVD, or saving your CV as a PDF?

The answer is that these and countless other tasks involve compressed files - files that in one way or another are smaller than the originals from which they derive. Modern computers are so fast we barely register that every time we open a Jpeg, MP3, PDF or any other compressed format, we're actually carrying out a complex mathematical operation, and the ease with which we bandy about such format names suggests that we take this technology for granted.

This is understandable, as we're used to the benefits these compressed files bring. So how does compression work? Are all compression algorithms the same? Is a Jpeg a lossy or a lossless format, and what's the difference? We've talked to industry experts, brushed up on our high-school maths and brought our own experience to bear to answer these and other questions. This is intended as a primer. We're not going to explore some of the more exotic maths involved in some of the more impressive compression sleights of hand, but we'll make sure you're familiar with the basic concepts.

Along the way, we'll be looking at the limits of human perception, stock-market predictions and the tale of a student who is smarter than his teacher. The world of

ADVERTISEMENT

compression is a lot more interesting than you might think.

Why compress?

At the birth of the Internet, networks were painfully slow, and compression and other tricks were necessary for transferring anything meaningful at all. Now, however, more than 99% of the UK population is connected to an exchange capable of delivering broadband, so why are we still obsessed about making files smaller?

There are two main reasons. The

first is that current broadband speeds might sound impressive, but they're still comparatively sluggish. Researchers at the Fraunhofer Institute for Telecommunications, Heinrich-Hertz-Institut in Berlin recently set a new record for network data transfer, pushing 2.56 terabits of information a second over a 160km link by using phase modulation to pack four binary states into a light pulse on a fibre-optic cable. That's the equivalent of more than 60 DVDs of data a second. However, even the very fastest ADSL connection available to consumers today, 24Mbit/sec, is at best less than 0.001% of that speed. We're

only just at the beginning of an age where codecs are good enough and connections fast enough to start streaming movies in

a limited way over the Internet; we can't afford to waste bandwidth.

The second reason is to do with storage. Costs may have plummeted and capacities rocketed, but we're generating information at a rate unparalleled in history, and if we were to keep all that data in an uncompressed format, we wouldn't have enough space to store it. This is especially true these days, as we generate so much space-hogging video in the form of news, television shows and movies.

Lossy and lossless

At a basic level, there are two different ways to compress data: lossy and lossless. The critical difference is that lossy data compression achieves smaller file sizes by throwing away the data it thinks we can live without. Lossless data compression, on the other hand, keeps every single bit of data, but uses clever tricks to save storage space.