Comments on: Killer Robots From Outer SEO Space: How to Dominate the Robots.txt File

By: Ashley Nixon

Ashley Nixon — Mon, 07 Feb 2011 18:51:04 +0000

This is all interesting but if i mess with this file will this upset the search engine gods. Im fresh out of stuff to sacrafice to them.

By: Ivan

Ivan — Mon, 26 Oct 2009 14:42:06 +0000

Thanks for the informative article.

Blocking Baiduspider using the Disallow method does not seem to work. I have seen numerous posts where people have basically given up since baiduspider seems to ignore the robots.txt file.

Is there an effective method to achieve this that you are aware of?

By: Dave Bascom

Dave Bascom — Tue, 03 Mar 2009 05:15:18 +0000

Looks pretty good to me. It keeps all the WP garbage URLs out of the search engines.

By: Russ

Russ — Tue, 03 Mar 2009 00:13:50 +0000

I’ve read a few places that this is a robots.txt file that people like to use for wordpress. Anybody have any thoughts on this?

User-agent: *
Disallow: /cgi-bin
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /wp-content/plugins
Disallow: /wp-content/cache
Disallow: /wp-content/themes
Disallow: /category
Disallow: /tag
Disallow: /author
Disallow: /trackback
Disallow: /*trackback
Disallow: /*trackback*
Disallow: /*/trackback
Disallow: /*?*
Disallow: /*.html/$
Disallow: /page
Disallow: /*feed*

# Google Image
User-agent: Googlebot-Image
Disallow:
Allow: /*

# Google AdSense
User-agent: Mediapartners-Google*
Disallow:
Allow: /*

Sitemap: http://www.DomainName.com/sitemap.xml