I’ve recently had to develop some custom functionality for a web page that would only be visible if a GoogleBot was crawling the site.
The development part of it is pretty easy – you check for the HTTP User Agent with either an “if” or a “switch/case” statement and do your thing. Simple concept. Same for most languages, with just minor syntax differences.
So how do you test what the site will render when the GoogleBot pays it a visit without writing your own browser? I came across Chris Pederick’s User Agent Switcher for Firefox.
It lets you select which browser/agent you want to simulate right from Firefox’s drop down menu (screenshot below), which makes testing a breeze.
Check it out for yourself, as well as some of his other cool web dev tools here: http://chrispederick.com