Adsense crawler also needs to crawl your site in order to KNOW what the heck to put into the ads on your pages… don’t make the mistake of using robots.txt to accidentally BLOCK google. I have seen this happen a number of times. AND I have seen it the other way where people are developing a top secret new site and FORGOT to block the crawlers from coming right in and indexing the site for the web. Big mistake there. If you have a development part of your website or have sample sections, please sure to properly implement a robots.txt file AND perhaps some meta nofollow tags to prevent the spiders from a’crawlin’ yur site. New info from the Google Adsense team to help you make sure this doesn’t happen to you!