Different types of Search engines: Directories and Crawlers

There are 2 types of search engines (although the results from most of the big ones come into both categories in some shape or form):

Crawlers:

These search engines send automated search programs - called spiders or robots - out to scour the internet regularly to update their indexes.

Your website is likely to get listed on the crawler based search engines by just being on the internet provided the spiders can find your website (if they find a link to your website or are pointed at your website by submitting your website URL for inclusion).

The spiders will attempt to navigate through your website pages (if it can) and list them all. Although in some cases the spider will not attempt to crawl deep links - web pages buried deeply in the hierarchy or layers of a website. You can also instruct the spider not to index some of your web pages. There are plenty of ways that the design of your website could be inhibiting or shutting out the spiders and preventing them from indexing your pages. See the common design problems here.

The internet is a large place and it may take some time for the spiders to find your site so don’t panic for at least 2 months.

Free inclusion is the norm for crawler based search engines but automated submissions are often blocked so a manual submission is a better bet. Some charge a fee to speed up your inclusion. Some charge a fee for a submission.

When you change your website the change is likely to be reflected on the crawler search engine the next time the spider visits your website.

Directories or Indexes:

These are search engines where your listings depend on a human reviewer or editor. Your website is included or not on their say so and in the manner they think appropriate.

You must submit your website if you are to be included and it may take a while before you come to the top of the list for consideration.

You will normally need to select an appropriate category for your website to be potentially listed under.

You submit a short description to the directory for your website, or editors write one for websites they review.

A search looks for matches only in the descriptions submitted.

If you change your website the change will not be reflected in the search engine unless it is re-submitted and reviewed.

You may have to pay to be considered for inclusion (whether you get included or not) and there may be an annual charge.

Mixed results:

There is a complicated relationship between search engines – some have arrangements for using each others listings and from time to time these arrangements change. So you can appear in a given search engines search results even though you are not indexed by them.

Each search engine is likely to favour certain indexes but may include results from others.

Hundreds of Search Engines:

There are hundreds of search engines. Most of which are niche market orientated. They are less important than the big ones but it may be worth seeking out the relevant ones if you have the extra time available.

The major Directories and Crawlers are described here http://searchenginewatch.com/links/major.html

Next : >> Content always comes first