Gallery

Build your own distributed file system

Posted on 10 Nov 2009 at 15:13

Simon Brock marvels at Google's resilient distributed file system, and builds his own from simple, cheap hardware and open-source software

Whenever our application talks to the resilient file system this duplication could be managed in one of two ways: either it could talk to one node that then passes on any changes to other nodes, or it could talk to all nodes at the same time.

The advantage of talking to a single node is simplicity, but there are a number of disadvantages. For example, if our application is talking to a node that suddenly goes down, how do we know whether the changes we made to that node were ever propagated to the others?

The multinode method avoids such problems, guaranteeing that if a file is on one node then it must be on some others too, and any node that doesn’t have a copy of the file knows where to find it.

While that might tempt you to dismiss the single-node route as flaky, many applications – particularly web servers – contain files that rarely change, which are written only once but read many times.

That means that read resilience is more important than write: even when a file does change, various levels of caching between the user’s web browser and the server may cause it to be some time before that change is reflected to all users.

Some database administrators will already recognise the scenarios just described: MySQL, for example, supports two ways of replicating changes across a collection of database servers. The simpler is to use binary log file replication, where changes to a table on one server are sent to the other databases in the cluster.

The important point is that the application making a change will think it’s succeeded once it has finished on one node, which works well enough but can lead to small replication delays and problems when a server fails.

The more resilient mode is to use MySQL Cluster, but that requires more resources and requires that any change must succeed on all nodes before the application will be told it has completed.

MySQL FS and SeznamFS

If the replication methods employed by a resilient file system are so similar to those employed by a database, why not use a database as the back-end to our file system?

This is the basic idea behind MySQL FS, which uses a MySQL database to hold all the file system information. If we ignore the replication aspect, then at first glance using a database to underpin a file system sounds downright stupid, as after all file systems need to be fast while databases are slow. However, on reflection it may not be as stupid as you think, for several reasons.

First, various operating systems over the years have attempted such a fusion between file system and database. In the dinosaur world of IBM mini-computers System/38 led the way, while more recently Microsoft hinted about combining SQL Server with the file system for Windows Server, although in the end it was never released.

MySQL FS isn’t as sophisticated as either of those systems, but it does work. A modern file system has to implement a very similar storage architecture to that employed by a database – it supports directories organised in a tree-like hierarchy, while a database has tables linked together via relations.

There’s also a similarity between the way an application must walk over this directory structure or access data held in linked tables. In both cases, remembering the results of expensive operations can lead to substantially improved performance – a database caches results from query table indices, while an OS caches results from querying directory structures.