Often times when I visit the mall, I have the same experience. I park my car next to a store with an exterior exit. That’s important for me because I need to find my car when I am finished shopping. Once I enter the mall, I wander around looking for a map. Usually the malls have kiosks in the middle of the walkways with maps nicely coded with some crazy numbering system derived from a handful of advanced non-linear equations.
This is when my first-grade reading abilities kick in- the skill where pictures speak a thousand words. After walking around for a bit, bumping into people and finding some random store I never knew existed, I exit and realize I parked on the other side of the building. All the way back to my car I’m cursing under my breath, vowing that I will never return to this labyrinth.
My experience with malls is not much different from the way search engine robots might feel when visiting websites. We lay-out what we believe is a perfectly logical website. A search engine robot visits the site, and some weird anomaly in the navigation confuses it, causing the site to be abandoned after a partial index. After a period of having the site live, we realize not all of the pages are indexed in Google.
So, what do we do? Add a sitemap.
Unfortunately, a site map written in Klingon doesn’t help the search engines. Just as I need a readable map in a mall, search engine robots will usually index a site quicker if you have a sitemap they understand.
When optimizing a site, it’s important to include a sitemap in a format that Google and other search engines can navigate. One of the most common is the XML-sitemap. Another option is the URL list, which includes a text file with one url per line, and is saved as urllist.txt. If you have videos on your site, you can submit a sitemap with all the videos. The same thing applies to sites that provide geo data, samples of code, and news. Google can also use RSS 2.0, Atom 1.0 and mRSS feeds as sitemaps. There are even sitemaps for mobile pages. As each XML sitemap should contain no more than 50,000 URLs, multiple sitemaps are sometimes required to index an entire site.
As of last December, Google made submitting a sitemap easier. You no longer have to specify the file type. Google will detect the file type and handle it appropriately. There are three different ways to submit a sitemap- which I’ll discuss in my next post.