The leading search engines, Google, Yahoo! and
MSN Live, use crawlers to find the pages for
their algorithmic search results. Pages that
are already linked from another search engine’s
indexed pages do not need to be re-submitted
because they will be found automatically. Some
search engines, notably Yahoo!, operate a paid
directory submission service that guarantee
crawling for either a set fee or cost per
click. Such programs usually guarantee quicker
inclusion in the database, but do not guarantee
any specific ranking within the search results,
just that your site will be indexed.<br>
<br>
Yahoo’s paid inclusion program has drawn much
criticism from both advertisers and
competitors. Two major directories, the Yahoo
Directory and the Open Directory Project both
require manual submission and human editorial
reviews before being accepted. Google offers
Google Webmaster Tools, to which an XML Sitemap
file can be created and submitted for free
online to ensure that all pages are found,
especially pages that aren’t easily
discoverable by automatically following links
between pages, such as dynamic content.<br>
<br>
Search engine crawlers (or robots) take
into account a variety of different factors
when crawling a site. Not every page is added
or indexed by the search engines. Distance of
pages (depth) from the root directory of a site
may also be a factor in whether or not pages
get crawled and indexed.
Recent Comments