XML Sitemaps: Making It Easier For Search Engines To Crawl Your Site
Have you recently updated your URL structure from underscores to hyphens? Are you constantly updating products and pages on your website? If so, it might be time to dig into XML Sitemaps, and get yours updated. Let's take a look at this important and sometimes underutilized resource we have at our disposal as Webmasters and SEOs.
So what exactly is a XML Sitemap? A Sitemap (capital S) is a XML file that lists the URLs on a website that you want to be crawled and indexed along with a priority and a recommended frequency for crawling each specified page. Google, Yahoo and Bing are all search engines that support the XML Sitemaps protocol.
To give you a little background, just like robots.txt and markup (schema.org) all the major search engines came together to form a protocol for us to follow which creates a standard for the way you can let the search engines know about pages that might not get indexed in the regular crawling process.
There are many resources available on the web for creating a XML sitemap and depending on the size of your website, these files can typically be created in a short amount of time. Once you have created a XML sitemap and have uploaded it to the root directory of your website (http://www.yourwebsite.com/sitemap.xml), you will want to let Google and Bing know about its existence. You should have Webmaster Tools accounts set up and verified for both search engines. Having these accounts set up and verified not only allows you to submit the location of your XML Sitemap but also gives you valuable insight into how these search engines crawl and index your website, more specifically URLs submitted and URLs indexed. If you don’t currently have these accounts set up, starting the process is very easy. For both Bing and Google, one option for verifying your account is to add the meta tag they provide you into the <head> section of your site’s home page. If your website site is not verified in Google Webmaster Tools and you are using the asynchronous snippet (which you should be!) and it resides in your <head> tag (Google won’t verify the account if your tracking code resides in the <body>), you can save yourself a step and click on alternate methods, Google Analytics options, then verify.
Now that we have a XML Sitemap created and uploaded, we need to let the Google and Bing now that it exists and is ready to be crawled. This can be done in Google Webmaster Tools, by clicking on Sitemaps, clicking on the ADD/TEST Sitemap button in the upper right hand corner of the page and adding the location of your XML Sitemap, www.yourwebsite.com/sitemap.xml. Google is kind enough to let you know if there are issues with your Sitemap. Other data you will be provided with is the date your Sitemap was processed and how many URLs were submitted and indexed.
The process for adding and verifying your XML Sitemap in Bing Webmaster Tools is very similar to Google’s. Log into your account, click on your profile which will direct you to the dashboard. Select crawl in the top right navigation where you can select sitemaps in the left navigation from which point you can add, remove or re-submit your XML Sitemap.
In addition to making sure the search engines know and have verified the location and format of your XML Sitemap, it also needs to be listed at the bottom of your robots.txt file which is the first file a search engine bot will hit on your website to get instructions on certain directories and files on your website to ignore during its crawl.
After completing these steps you will have sent Google and Bing a roadmap of the pages on your site you would like crawled. I hope this quick overview has been helpful to anyone not familiar with XML Sitemaps and look forward to any additional insights or comments.