1. What is a Search Engine?

This may seem obvious, but it's not. All sites that point toward the content of other sites are not the same. To begin with, the Internet has a profusion of sites that contain links or indexes of other sites. In addition, these sites don't all operate for the same purpose or in the same way.

In the broadest sense, search sites fall into two families:
Directories and Text Indexes (also Directories)

A directory site is based upon a hierarchy of categories created by editors or producers at the site. The site then creates a process for receiving information about web sites and placing them into this hierarchy. The web site information generally comes either from the editors or from the listed sites themselves. A directory site doesn't actually search the Internet or an index of text from web sites. Rather it searches its hierarchy structure and the information that has been entered into that structure by editors or web sites.

If you use the tips we offer further down in this primer you can obtain excellent ranking results on Yahoo and other Directory Sites.

Text Index Sites
The second kind of search site and the kind that dominates the world of search engines is the Text Index Site. These sites use specialized and powerful software programs called, spiders, robots or daemons to scour the Internet scanning for new web sites or changes at web sites already visited. These programs suck up key bits of text from the web sites (or in some cases every single word on the sites) and place them in a special kind of index optimized for fast searching on the Web. In some cases, the spiders will also trace the links for your site's Web pages and index those, as well. Excite, Alta Vista, Infoseek, HotBot and most other well- known search sites are Text Index Sites.

These are the true Search Engines of the Web and the sites that form the backbone of our free service, SmartAge SiteRank.

Because Search Engines send their programs out to your site to gather information, understanding how they work and what they look for can hugely effect how they view and rank your site. SmartAge SiteRank helps you achieve the best results with these search engines and monitors the success of your rankings on the leading search engine sites.

2. How do Search Engines Search?

In general, all search engines work alike, but in detail every one is different.

All search engines begin with a word of phrase typed in by a visitor to the site. This "keyword" is then compared to the search engine's spider-generated index of text from thousands and thousands of web sites. The goal at all search engines is to use this text sifting to create a list of web sites that most closely matches the intent of the keyword entered. This is called "relevancy," and is the basis upon which search engines customarily sort web sites; the most "relevant" site tops the list; with the least relevant consigned to the nether regions.

Alas, every site uses a different definition of relevance, and all of them base relevance on a complex combination of factors. Managing these factors stands at the heart of effective search engine ranking. Among the factors used by search engines to establish relevancy are: the frequency of the specified keywords in the web page or document the number of times the keywords appear in the web page's title {the appearance of the keywords in "meta tags" appended to the web page {the positioning of keywords within the web page - i.e.: words in the first paragraph may be rated higher for relevancy than the same words in the last paragraph. {\tab the appearance on the page of certain word patterns; these may be either a negative factor (x-rated for instance could drop a site's relevancy in a search on a non- adult oriented keyword) or positive (the appearance of Baltimore next to Orioles in a search on the keywords "baseball teams").

Each Search Engine applies these, and other factors, in a unique way. Now let's examine how you can develop your web site for maximum search engine effectiveness.
3. What are the key concepts in understanding Search Engine site ranking?

Keywords
These are the word or words that people type into a Search Engine. Your site to be ranked high or low based on its relevance to certain keywords. So, it is imperative that you think about what words and phrases more accurately reflect your site's benefits and then place them in the most potent places on the site. We will describe how to do this in detail below.

Frequency
The more often a particular keyword appears in your web page, the higher your ranking is likely to be. So, it is good to have key words and phrases repeated on your web pages. However, this effect is not infinite. Many search engines, burned by sites that have rampantly repeated words merely to be highly ranked, have placed limits on the number of times a word is counted for relevance, and in some case assessed penalties for profligate use of a word or phase on a single web page.

Position
Appearance of keywords in some locations on a web page, such as in its title or its first sentence are often viewed as being more valuable than appearance elsewhere.

Metatags
Metatags are a set of words included in HTML pages as additional descriptors of the page, but that don't actually appear on the page itself. Some Search Engines include metatag words in their relevancy calculations, others do not. Including keywords in metatags can be helpful, particularly with pages whose purpose may not be clear in text (such as a page with a big photograph), but should be viewed as largely supplemental to other techniques. We will describe how to manage metatags below.

Netiquette
It is perfectly appropriate to use every legitimate trick and technique to get your web site the attention it deserves, but honesty and restraint are critical. In early web days, so many sites abused key words and tried to manipulate the Search Engine ranking process in ways that had little to do with the actual benefit of their page, that Search Engines has been forced to tighten their rules and limit how sites can describe themselves. Remember that a visitor who comes to your site with ill-founded expectations won't be a good customer for your business.

4. What is the process for being listed and ranked on a Search Engine?

Step One: Get the Search Engine to notice your site.
In many cases, this can be as straightforward as going to the site and filling out a registration form. Usually, this form doesn't result directly in a listing. Rather, it initiates the process that will eventually bring you onto the site. Generally, when you register with a Search Engine, it will check immediately that the listed URL is at least active; so don't register sites or pages until they are actually up on the Net.

Step Two: Pre-indexing.
Once a Search Engine knows you exist, it slates your URL for a visit by its spider program. Most often, this takes place in two stages. The first stage is pre- indexing. Here, the scouring program visits your site and creates the table of keywords that will be associated with it. Pre-indexing can take place from 1 day (Alta Vista) to 2 weeks (Webcrawler) after registration.

Step Three: Primary Indexing.
For search engines that pre-index, the spider program returns and performs a more thorough scouring of the web site. It will not only check the referenced page, but also links from that page. A site's relevancy ranking can shift dramatically between pre and primary indexing. Primary indexing follows pre- indexing by up to a month. Some Search Engines, such as Excite, Lycos and Northern Lights, combine pre and primary indexing into a single operation. This is bad, because it delays initial indexing for at least a week after registration, but good because it completes the process usually in less than three weeks.

5. What happens when I make a change to my site?

Search Engines send their spiders to revisit known pages on a regular basis; it varies from a few days to several months. However it is wise not to rely on the Search Engines to pick up changes in your site. Instead, whenever you add or substantially change a page, re-register it with the search engines.

6. What web page elements cause Search Engine Spiders problems?

Some web page elements can interfere with the spider program's scouring process:

Frames
Spiders view frames as links from the main page. So, if you have valuable information in frames, the Spider won't pick it up on pre-indexing and may still get it wrong on primary indexing. The way to handle this is to have a no-frames version of your page available, and to make sure your HTML code makes this clear to a spider. You do this by adding keyword text between the <noframes> and </noframes> tags of the HTML code. The noframes tags are usually placed below the HTML tags that set up the frames: <frameset> and </frameset>. By placing keywords between the <noframes> tags you are giving the spider something to work with it might miss otherwise. Your site will look unchanged on browsers that support frames, but will be at least readable on those that do not, such as Netscape 1.0.

Tables
Spiders generally view the text on a web site linearly. The spider will start at the beginning of column one and read down the page. This means that if you have key topics at the top of three columns, the spider will downplay the value of the latter two areas for search relevancy. What can you do? Try placing an intro paragraph with keywords for all areas at the top of the page or the top of the left- \tab hand column. Or, if you have left hand column navigation, make sure that column contains a full set of keywords. Metatags can be particularly important for pages with key ideas spread all over the page.

Images
Spiders can't decipher images. This not only includes photos but and GIF art you might use for logos, column headings or other display type. Use headers with images in your text, i.e. a picture of a baseball bat with the code: - <H1>Baseball Bats</H1> accompanying it. And, incorporate keywords into the ALT text with images. This not only can help spiders pick up the content value of an image,it also provides information to those who turn off image loading to make web sites download faster into their browsers.

Place ALT tags after the image file, like this: <img src="/images/quilt.gif" alt="Double Wedding Ring Quilt! A Quilt and Knit Shop Exclusive Craft Offer">

Once again, remember, there is no universal panacea for search engine ranking. AltaVista, Infoseek, Lycos and WebCrawler read Alt text, but Excite and HotBot do not.

7. Tips and tricks for web friendly design

Title Your Site Carefully.
The HTML Tag <title> is often ignored by site designers, but it is one of the first places spiders look for keywords. The <title> </title> tag sets the text that appears on the browser title bar when the site is loaded and that appears on any bookmarks for the site. Here's how you set up a <title> tag:

<head>
<title>The title information on your site goes here!</title>
</head>

Put keywords in this area. "My Home Page" won't tell a spider anything. "The Quilt and Knit Shop! Sweaters and Bed Covers." Will create keyword associations for Quilt, Knit, Shop, Sweater, Bed, Covers and Bed Covers.

Use metatags. As we have noted, Search Engine spiders often reference metatags embedded in the HTML code for a web page. You should use metatags generously to describe the attributes or your page, but with an eye toward accuracy, rather than toward "fooling" the spider to list your page unnaturally high.

And remember that metatags have different influence at different Search Engines: AltaVista, Infoseek and HotBot read and give weight to metatags Lycos and Northern Light weigh them no more heavily that ordinary body text. Excite doesn't even look at them.

Here's how to set up metatags for your page. There are two types: Description and Keywords. Enter the description inside quotation marks and the keywords inside quotation marks, with each word or phrase separated by commas and a space. Keep the description below 25 words, which is all most spiders will read, and keep the keywords to 7; some engines will actually penalize you, or ignore the tags, if you use more.

<HEAD>
<title>The Quilt and Knit Shop! Sweaters and Bed Covers</title>
<meta name="description" content="Bargains for Quilt and Knit Product Buyers.">
<meta name="keywords" content="quilt, knit, sweater, bed, bed cover, shop, bargain">
</HEAD>

Organize your information wisely.
Remember that many Search Engines weight information in the first sentence or paragraph or at the top of the first column, more heavily than other text. So, organize your page accordingly.

If your page begins:
"When I was a boy, I always had a dream; I knew that somebody I would do something terrific and start my own successful business... "

The spider isn't going to pick up anything useful. Your page will be poorly ranked. If, on the other hand, your page begins: The Quilt and Knit Shop Story Quilting and knitting, creating the best designs and bargains in sweaters and bed covers, form the heart of my store and this e-commerce shop. The spider will pick up Quilt, Knit, Shop, Design, Bargain, Sweater, Bed Cover, Store, E-commerce and Shop as keywords of high importance on your site.

8. How do I get the best ranking on a Directory Site?

Unlike spider-driven Search Engines, directory sites place web site URLs within a hierarchy of subjects they create. The site uses this hierarchy as its first cut in response to any keyword search. So, the key consideration with a directory site is where in their hierarchy does my site naturally belong?

This might sound simple, but in fact it's rather complex and takes both time and consideration. First, different Directory Sites have difference hierarchy; a single word might have different connotations on different sites. So, you have to actually examine each Directory Site you are interested in, to make sure you understand what category makes sense in each environment.

Unfortunately, this can only be accomplished by visiting each directory site and examining it. That is why SmartAge SiteRank cannot place your site on directory sites automatically.

When you examine a Directory Site, look for sites similar to yours. Don't try to outguess the market and go where potential competitors aren't. You need to be in the categories and sub-categories that contain others like you. That is where interested individuals will go and will be directed by their searches.

Also, don't twist yourself into knots to get into "favorable" situation. For instance, if a Directory Site lists numbered site names first within a category, don't rename your site 123Quilting, just to beat the odds; your name has so many other more important considerations behind it. And don't try to shoehorn your site into Categories where it doesn't belong, because you think they are more likely to be visited. These associations won't bring appropriate visitors to your site.

If you don't see a Category that seems appropriate for your site, tell the directory site about it. Sites are constantly expanding and retooling their hierarchies. Remember, they didn't create all those myriad subcategories on their own!

9. How do I get listed on Yahoo?

Yahoo is not only the most widely visited Directory Site on the Internet; it's unique in the way it selects sites for inclusion in its hierarchy. There are two ways to get a site on Yahoo's radar screen. We'll discuss the least common option first.

In addition to it's on-site registration program (which is where the vast majority of its listings come from) Yahoo has launched a unique kind of spider program. This program doesn't scour the entire Web, instead it visits a select set of sites that announce the existence of new Web sites or that give awards for site innovation and excellent. Yahoo uses information gleaned from these sites to direct the attentions of its editors.

So, you can be listed on Yahoo even if you have never registered there! So one way to get onto Yahoo is: Get a GREAT PR person! There are literally hundreds of so- called Announcement Sites on the web today. Some are general new- site-of-the-day services; others focus on particular topic areas, etc.

To get the attention of these sites you'll need to present them with a crisp package of information about your site and its benefits. It is truly a PR, rather than a technical, process.

And, remember that, in this process, the decision about where in the Yahoo hierarchy you go is made by the editors, not you.

The more common way to get listed on Yahoo is to register at the site. Here is an overview of that process:

1. Locate the best Category for your site. Do this by either drilling down into the Yahoo hierarchy, using the ideas we mentioned above. Or, enter a Yahoo search using keywords from your web site and see where most of the responses are located. If you have a purely commercial site, Yahoo will require that it be somewhere in the Business and Economy Subcategory, so look there. If you have a Personal Homepage, Yahoo will insist that it be placed in the Entertainment/People Subcategory. And, if your site has primarily a regional impact, look into the regional categories for the right location to place yourself in.
2. Go to the Yahoo page listing the sub-Category you want to be listed under. Click the Add URL icon at the top of this page. You will get a form to fill in, with the Category you are located within already included. If you want to be listed in more than one Category, go to your second choice, then click on right mouse button with the cursor over the Yahoo logo; move down to copy link location and click. Then go to the location of your first choice, get the Add URL form, and paste the link to the other subcategory into the form. This reduces the chances that you will enter it incorrectly.
3. Fill in the ADD URL form. Yahoo derives the keywords it uses in searches about your site from the Title and Comments fields on this form. Fill them in with particular care. Include in them the keywords from your web site. It is best to use complete sentences in filling out the form with keywords in them, rather than simply entering words. The same principles for good Titling hold here as in your own Web site design, as we described above. Comments should follow the same rules as metatag descriptions on your own site. Remember that this form is an application to an editor, so be accurate. If you're not, they'll just ignore you.
4. Submit the form. Click the "submit" button and you'll see a Thank You screen, unless you made a mistake in completing the form. If you made a mistake, you can just hit the browser back key to retrieve the form and fix the error.
5. Be patient. Yahoo is swamped with submissions, receiving tens of thousands a week. They say a site will be up within two weeks of registration, but often miss that target. Check the site after two weeks. If your site isn't there, go to the Where's My Site area on Yahoo for information about what to do.
6. Make sure your listing is accurate. Once you appear on Yahoo, make sure your listing is where it should be and contains the right information. Do a few keyword searches to see how your site shows up in the results. If you feel something is seriously wrong, let the folks at Yahoo know.

10. Where can I find out more about Search Engine Ranking?

Here are some good sites for more information about how Search Engines work, and how they establish rankings:

Alta Vista, About Alta Vista Search
http://www.altavista.digital.com/av/content/about_our_story_3.htm

Alta Vista Search: Adding an URL
http://www.altavista.digital.com/av/content/addurl_meta.htm

Excite, Excite Help: Getting Listed
http://www.excite.com/Info/listing.html

InfoSeek, InfoSeek: Results
http://www.infoseek.com/AddUrl?pg=DCaddurl.html