With 40 million websites in existence, and more than 3 billion
web pages indexed by Google at the time of this writing (July 2003), its no
wonder that more and more people are relying on search engines to find their way
through the unruly world that the web has become.
Nowadays, it is crucial to get your pages indexed by the most
important search engines. To maximize traffic to your site, you must make
sure that all your internal pages are indexed, not just your main page
Fortunately, you dont need to submit each of your pages manually. The most
efficient way is to create a Site Map (a list of links to all the pages
in your site) and link to it directly from your homepage.
How Will A Site Map Help Me?
Search engines find pages by crawling the web. They go
through the code of all the pages in their database (also called index), following links
to other pages and adding them to the database (in fact, more pages are added this way
than by manual submission).
However, search engines have trouble following links from pages buried
too deep within the directory structure of a site. A Site Map solves this problem
by giving the engines access to the links to all your pages once they follow the Site
Map link in your homepage. For more effectiveness, place your Site Map in your
root directory (where your index page is).
Site Maps: Not Just for Search Engines
While some web users will find their way through your site by following navigation
links or by using the search box, others will turn to your Site
Map. If you design your Site Map carefully, it will not only be useful to the search
engines, but to your human visitors as well.
Here are some pointers:
The Site Map should act pretty much like the table of contents of a
The Site Map must clearly show all the sections of your site, and the
information contained in each of those sections.
Every item in your Site Map must be hyperlinked to its URL.
If its not too long or cumbersome, use each pages TITLE as
the link text, since this tends to increase the relevance of your site.
Otherwise, use the word or the short phrase that best describes the content of the page.
Make sure that you place the link to your Site Map at a visible location
in your homepage (users shouldnt need a map to find your Site Map!).
Dont get creative: simply call the link Site Map.
search engines will ignore it.
How can I check if my pages have been indexed?
Once you have created and uploaded your Site Map and placed a link to it in your
homepage, submit both your homepage and your Site Map page to the search
engines. You will then have to wait until the search engines do a web crawl.
In the case of Google, the largest search engine, this happens approximately once a month.
To check if a page on your site has been picked-up and indexed by Googles, go to www.google.com and use the allinurl
command in the search box:
Where yourdomain.com/yourpage.html is the URL of the page you want to
To get a list of all the pages in your domain that have been indexed
by Google, youll have to use the site command, followed
by your domain name plus a word (or group of words) that you know appear
in all your pages (for example, a copyright statement or some footer text):
If after typing this command you get a list of all your pages (or at
least a significant number of pages that werent in the index
before), this will be a strong indication that your Site Map has been successful.
You can freely reprint this article. Just include the following resource box at
Mario Sanchez publishes The Internet Digest ( http://www.theinternetdigest.net
) a website and newsletter that gives you free advice on web design and Internet
The SEO Book: (by
Aaron Wall) Author Aaron Wall takes you from A to Z on a journey to understand
the basics of Search Engine Optimization. No hype, just honest and effective advice.
This eBook comes with free updates for life. Highly recommended.
The Nitty Gritty of Search
Engine Optimization(by Jill Whalen) This special report in ebook form is
probably the most thorough guide on how to write for the search engines. Making sure
that your main keywords are well represented in your page copy without sacrificing
readibility is not always easy, but this special report shows you how to do it.