On Site Search SEO Tip #1

A major item that a lot of sites overlook and a lot of SEO’s don’t really understand is the “site search” function on a site. I’m going to give you a scenario that three of our clients have faced, with similar results.

Let’s say you own a real estate site with information being pulled from the MLS database via IDX or some other listing server. None of the content that your on-site search result displays is in any way unique from all the other real estate sites online. Which means that even if you have 20,000 pages indexed according to Yahoo, not even 1% of these pages have a great chance of showing up unless someone searches on the exact title of your listings page.

This idea also holds true for paginating issues in shopping carts and blog pages, but we won’t be delving into those two items in this post. So you have two choices basically (I’m assuming your site isn’t homes.com, realtor.com, or one of the large nationwide realty sites).

1. You can write unique titles, meta descriptions and listing descriptions for each listing, which would take you the rest of your life.


2. You can exclude all your search results from the search engine spiders by adding an exclusion for your on site search results subfolder to your robots.txt file.

What if your search results get displayed using something like this: www.yoursite.com/index.php?search=action&name=something

Can you use the robots.txt file to exclude that search result, or any search result on the index.php page, without also excluding your index page from the bots? The answer in this example is no. But, if you’re going to be de indexing your search pages anyway, you can move your search results to display under a separate subfolder called /search/ or something and then using your robots.txt file, you can make all of your search listings no indexed, while still retaining one main search page, perhaps at search.html. The exclusion would look exactly like this in your robots.txt file:
User-agent: *
Disallow: /search/

Get Internet Marketing Insight For Your Company - SEO.com


  1. says

    Google has come out and said that they don’t want to index search results pages that they feel don’t add value to their SERPs

    Here’s what they say:

    Use robots.txt to prevent crawling of search results pages or other auto-generated pages that don’t add much value for users coming from search engines.


    It’s kind of one of those touchy areas though, because most site owners feel that their SERPs pages are feakin’ awesome and should be included in the Google search. Sly webmasters try to get around this Google guideline by changing the file or directory name to something other than “search.” Although, if there’s nothing unique about the content on those pages, they won’t get indexed very well anyway, like you said.

  2. albert says

    Another item that many people over look with the search function is seeing what people coming to your site are looking for. There is a way in Google Analytics to see what it is people are searching for on your site. If a lot of people are searching for something on your site you might want to take a clue and make it easier for them to find- ie a link pointing to the product or page maybe even consider using that term to optimize other pages on your site.

  3. Matt says

    “Can you use the robots.txt file to exclude that search result, or any search result on the index.php page, without also excluding your index page from the bots? The answer in this example is no.”

    Another option is to use a META NOINDEX tag in the header of the search results pages, instead of robots.txt. This would allow the index page to still be indexed. The main cost in this case is that the spiders would still crawl your search listings, possibly causing load and contention problems for your servers.

  4. Marcel Fuursted says

    Actually there are websites that have recieved penalty for having indexed search results from the site search. So it’s not just a good idea… Its very important to exclude the search results in robots.txt 😉

  5. Amanda says

    Interesting post, though I have never heard of anything like this. I do not have a search field on my blog. Should I? It sounds complicated to do all of that. I feel like I need to have a degree in order SEO properly.

  6. Daus says

    Hmm, 2008 post. But now, I think there are so many website that include their search result in robot.txt. Did google cancel their rule? Who knows? 😀

Leave a Reply

Your email address will not be published. Required fields are marked *