XML Sitemaps And Your Website

XML Sitemaps How To Add One To Your Website

XML Sitemaps: Making It Easier For Search Engines To Crawl Your Site

Have you recently updated your URL structure from underscores to hyphens? Are you constantly updating products and pages on your website? If so, it might be time to dig into XML Sitemaps, and get yours updated. Let’s take a look at this important and sometimes underutilized resource we have at our disposal as Webmasters and SEOs.

So what exactly is a XML Sitemap? A Sitemap (capital S) is a XML file that lists the URLs on a website that you want to be crawled and indexed along with a priority and a recommended frequency for crawling each specified page. Google, Yahoo and Bing are all search engines that support the XML Sitemaps protocol.

To give you a little background, just like robots.txt and markup (schema.org) all the major search engines came together to form a protocol for us to follow which creates a standard for the way you can let the search engines know about pages that might not get indexed in the regular crawling process.

There are many resources available on the web for creating a XML sitemap and depending on the size of your website, these files can typically be created in a short amount of time. Once you have created a XML sitemap and have uploaded it to the root directory of your website (http://www.yourwebsite.com/sitemap.xml), you will want to let Google and Bing know about its existence. You should have  Webmaster Tools accounts set up and verified for both search engines. Having these accounts set up and verified not only allows you to submit the location of your XML Sitemap but also gives you valuable insight into how these search engines crawl and index your website, more specifically URLs submitted and URLs indexed.  If you don’t currently have these accounts set up, starting the process is very easy. For both Bing and Google, one option for verifying your account is to add the meta tag they provide you into the <head> section of your site’s home page. If your website site is not verified in Google Webmaster Tools and you are using the asynchronous snippet (which you should be!) and it resides in your <head> tag (Google won’t verify the account if your tracking code resides in the <body>), you can save yourself a step and click on alternate methods, Google Analytics options, then verify.

Now that we have a XML Sitemap created and uploaded, we need to let the Google and Bing now that it exists and is ready to be crawled. This can be done in Google Webmaster Tools, by clicking on Sitemaps, clicking on the ADD/TEST Sitemap button in the upper right hand corner of the page and adding the location of your XML Sitemap, www.yourwebsite.com/sitemap.xml.  Google is kind enough to let you know if there are issues with your Sitemap. Other data you will be provided with is the date your Sitemap was processed and how many URLs were submitted and indexed.

The process for adding and verifying your XML Sitemap in Bing Webmaster Tools is very similar to Google’s. Log into your account, click on your profile which will direct you to the dashboard. Select crawl in the top right navigation where you can select sitemaps in the left navigation from which point you can add, remove or re-submit your XML Sitemap.

In addition to making sure the search engines know and have verified the location and format of your XML Sitemap, it also needs to be listed at the bottom of your robots.txt file which is the first file a search engine bot will hit on your website to get instructions on certain directories and files on your website to ignore during its crawl.

After completing these steps you will have sent Google and Bing a roadmap of the pages on your site you would like crawled. I hope this quick overview has been helpful to anyone not familiar with XML Sitemaps and look forward to any additional insights or comments.

Get Internet Marketing Insight For Your Company - SEO.com

9 Comments

  1. Jeff Black says

    I clicked on your link to xml-sitemaps for sitemap generation. That seems to be the best one out there for non-enterprise sites. Are there any others that you have found that are good? I know you can use xenu to create one, but it’s a bit more of a manual process than I would like.

  2. Kent says

    Thanks for the information. For so many years keep creating sitemap for my website, I now only know that “is a XML file that lists the URLs on a website that you want to be crawled and indexed along with a priority and a recommended frequency for crawling each specified page.”

    At least I know what I am doing now.

    By the way, with Sitemap, we can only influence bot to crawl but not control them, am I right? And it doesn’t guarantee it helps in SEO.

  3. Peter Rota says

    Hi Brian this is a great post when people talk about SEO they rarely talk about XML site maps but, without these it will certainly effect your SEO efforts, it was good to hear also about updating them as well this is something i have not thought about. But, it is certainly something that i will think about as I continue forward in my SEO efforts.

  4. Ian Locke says

    Just wondered if there was any reason you mention changing from underscores to hyphens – was it simply as a warning to people in case they had, or to recommend they do so (as I seem to recall seeing a Matt Cutts video saying Google preferred hyphens)?

    • Brian Jensen says

      Ian, thanks for the comment and great question. While hyphens and underscores both serve as separators in your URLs, Bing and Google treat them a little differently. According to Matt Cutts who leads up the Google spam team, Google joins at the underscore and separates at the hyphen. He goes on to explain that the rankings difference between the two is minimal. My thought is that having keyword rich URLs separated by hyphens could help Google better understand what a page is about. Bing however has came out and publicly said that it makes no difference which one you use. My main reason from mentioning this in the post is that making the change will also create a new URL that will need to be updated in your XML Sitemap.

  5. Foro Vietnam says

    How can you know which urls are not indexed by google?
    I mean, your screenshot you have submitted 72 urls but only 68 were indexed… which are those urls and why they weren’t indexed?
    Thanks

  6. Gordon M. says

    Brian,
    Thinking back now that I have set up my xml sites, I was not aware that I would have to update my xml site if I decided to change keyword using hyphens. I thank you for this great information I will be updating my xml site soon!

  7. Rahul Gupta says

    Both the onsite maps and offsite xml are importatnt one for user or visitor and one for search engines.

Leave a Reply