Pure Digital Thought

I noticed GoogleBot crawling the site and thought the order was funny. Shouldn’t GoogleBot be looking at the robots.txt file before crawling the site? That’s one way to get your bot in trouble.

66.249.66.227 - - [02/Sep/2005:15:46:04 -0500] “GET /2005/08/28/rss-problems/ HTTP/1.1″ 200 3079 “-” “Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)”
66.249.66.227 - - [02/Sep/2005:15:46:22 -0500] “GET /2005/08/25/the-good-ole-days/ HTTP/1.1″ 200 3806 “-” “Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)”
66.249.66.227 - - [02/Sep/2005:15:46:46 -0500] “GET /robots.txt HTTP/1.1″ 404 208 “-” “Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)”