Yahoo! crawls the Web from new address

Posted on 7 Jun 2007 at 15:03

Yahoo! has finished moving its Web crawler to a new domain. Slurp, which indexes pages for Yahoo!'s search engine, will now operate from crawl.yahoo.net and the change could have implications for website administrators.

There is no need to change the robots.txt file, since the crawler user-agent is still Yahoo! Slurp, and since the IP address from which it operates has not changed, IP-based filtering can be left as-is.

However Yahoo! does recommending ensuring that any network or firewall set-up does not keep crawl.yahoo.net out and encourages webmasters to set up reverse DNS-based authentication of the crawler to ensure that no rogue bots masquerading as 'Slurp' visit your site. Instructions are provided.

In addition to verifying whether the crawler is genuine or not, this method, rather then IP address-based access control, will ensure that any website will be compatible with any future changes.

The change of domain is little more than a rebranding exercise. Previously Slurp operated from inktomisearch.com, a legacy of Yahoo!s acquisition of Inktomi in 2002. There is no indication that this will change how sites are indexed.