Adsense crawler also needs to crawl your site in order to KNOW what the heck to put into the ads on your pages… don’t make the mistake of using robots.txt to accidentally BLOCK google. I have seen this happen a number of times. AND I have seen it the other way where people are developing a top secret new site and FORGOT to block the crawlers from coming right in and indexing the site for the web. Big mistake there. If you have a development part of your website or have sample sections, please sure to properly implement a robots.txt file AND perhaps some meta nofollow tags to prevent the spiders from a’crawlin’ yur site. New info from the Google Adsense team to help you make sure this doesn’t happen to you!

Related posts:

1. Google AdSense, Analytics Integration is complete.
2. Google Adsense Referrals program is gone…
3. Adsense Analyzing earnings by page views vs. ad requests
4. Utilize GOOGLE TOYS To Do Better With GOOGLE Search?!?”
5. Search Engine Market Share July 2009

Tags: adsense, robots.txt

This entry was posted on Tuesday, June 10th, 2008 at 11:45 am and is filed under GOOGLE, seo. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.