Don't spam the spider
Before submitting your Web site to a spider-based search service, learn more about the intricacies of these services. There might be more involved than you think.

The way of the spider

Unlike the simple process of submitting your URL to a directory, which involves sending it to a human editor, submitting it to a spider-based search service involves no editorial review. Your submission finds its way to a queue that a spider visits; then the spider records your URL for possible inclusion in an index.

There are several points to keep in mind when submitting your site to a spider service:  

• Spider results may appear on search engine sites that use spiders as well as search sites that license spider technology.
• It's not always apparent which sites use spider results. And even if you can tell, it's not always clear which spider's results are being shown.

Spiders can bite

Here's another factor to consider when sending your URL to a spider service: Submitting it too often -- i.e., more than once every six weeks until your site is accepted -- may cause your submission to be viewed as overzealous marketing, or spamming. This can cause your site's ranking to be lowered or its listing to be dropped. It's easy to over submit your URL if you submit it to two sites that use the same spider service's results.

To keep track of which spider services are used by which search sites, and to avoid submitting your URL too often, you need to know the major spider service affiliations. Then you can assess where to concentrate your submission efforts.

A tangled web

Understanding the fairly complex matrix of spider services can help you avoid over submitting your URL. Here's some information to help you get started:

AltaVista operates a spider and serves results at its site. Inktomi, meanwhile, provides the spider technology for many search portals. You need only submit your site to Inktomi once to get in. If it's accepted, your site will be found on MSN Search, HotBot and many other search sites.

Though MSN Search licenses spider-based results from services such as Inktomi, MSN lists primary results from LookSmart.

While submitting your site to MSN Search means you're submitting to Inktomi, it's also true that Inktomi provides results for HotBot. So you shouldn't submit your site to both MSN and HotBot because Inktomi will get both submissions.

Spider genealogy

Inktomi provides results for more than 125 major portals worldwide: primary results for iWon and canada.com, and secondary results for MSN Search, AOL Search, HotBot, About.com, LookSmart, NBCi, GoTo and 4anything.com, to name a few. Being aware of these connections can help you to avoid over submitting your URL.

If you submit your URL to Inktomi through its paid inclusion program, you should not submit it to MSN, HotBot, canada.com or NBCi. To submit your site to Inktomi for free, canada.com is the best entry point.

Submit carefully

Google serves secondary results at Yahoo!, so you should consider making it a high-priority submission target. To submit your site to Google, you must do so directly at Google.com. Since the submission is direct, you're in no danger of mistakenly over submitting.

AltaVista uses its own spider and, as with Google, requires you to submit your site directly, so there's no danger of mistakenly submitting too often.

Beware of submission creep

Know that if you submit your URL to Excite, it will also wind up at WebCrawler -- so don't submit to both services. You'll notice that the submission forms for the two services are identical.

Excite is an affiliate of LookSmart's paid submission program. Unless you intend to submit to LookSmart through Excite, don't bother with the Express or Basic Submission options on Excite; they link to LookSmart.

See FAST results

FAST Search at alltheweb.com provides secondary results for Lycos as well as several international search sites. U.S. site owners are attracted to this service because it allows their sites to appear in Lycos results.

To submit to FAST Search, do so directly.

While Lycos gets its secondary results from FAST Search, its primary results come from Open Directory Project (ODP). Lycos still maintains its own spider, which passes results along to alltheweb.com.

To prevent over submitting your URL to FAST Search, you may want to submit it first to Lycos, and then wait about six weeks to see if it shows up in FAST Search. If it doesn't appear, submit it directly

Like AltaVista, NorthernLight powers its index with its own spider technology. To avoid excessive submission, submit your site to NorthernLight directly.

Stay on top of things

The search engine world changes frequently, so it's difficult to capture the facts in a way that remains accurate. You can keep track of changes by reading Search Engine Watch, I-Search and information made available at the search engines themselves.