Google bug reveals Web stats

By Steve Malone

Posted on 21 Nov 2005 at 10:43

Search engine Google has acted quickly to plug an embarrassing hole in its Sitemaps facility. Visitors who 'verified' that they owned a site with Sitemaps could see private information about the site.

Sitemaps is a facility provided by Google that provides webmasters with information about their sites. Some of it - such as 'links' and 'inurl' - is meant to be public but other data is supposedly only available to the site owners. A bug crept in that allowed anyone to claim they had a right to see the information.

The problem arose in the way that Google checks that the visitor 'owns' the site. The search engine does this by generating a unique page URL that has to be placed under the domain name. When asked to verify, Google will check whether the page exists. However, it turns out that this is not quite what it does. What it checked was whether it received a '404 Page Not Found' message. Some sites do not generate a 404 but instead say, refer the request to a similar page. In these cases Google would accept that the page had been 'verified' and provide the Sitemaps account with the information.

To be fair, the private information was not that detailed and mostly consists of referred pages although Google has promised to increase the information provided in the future. However, it is embarrassing that a company which boasts wall to wall Computer Science PhDs should get caught out by such a simple oversight.

In a blog posting, Vanessa Fox from Google Engineering said that the bug is one which has crept in recently. When Sitemaps was introduced a couple of months ago the system checked to make sure that the web server is configured to return a 404 correctly when a request for a non-existent page is made. According to Fox 'with our latest release, a bug prevented this process from working correctly'.

Google is also attempting reassure webmasters who may be alarmed at Google revealing potentially sensitive information. Fox says that the hole has been plugged, and to ensure the security of all sites using the Google Sitemaps tool, the company will re-verify all sites added in the previous 48 hours.