If you use search engines for searching, you've probably wondered how they choose which of the million hits to display first. If you've ever tried to get your site listed in a search engine, you've probably wondered how to get your site listed first so people will be able to find your web site.
Based on current experience in getting the Saint Louis Cultural Flamenco Society website listed, I've found some interesting information on search engines and the tradeoff between economics and ethics. Please note I have no financial interest or ax to grind in this work. It would have saved me some time if I had understood the following before trying to get the website on the search engines.
Our goal is to be listed in the first few entries when someone types "Saint Louis Flamenco" into their search engine. Since there is only one flamenco group in Saint Louis, this would seem a reasonable goal. The society is in no way controversial. With some knowledge of the search engines, the website has been designed to be easy for search engines to look over.
The Saint Louis Cultural Flamenco Society website, I hope you will agree, is a website suitable for viewing by people of any age. There is no sex, violence, foul language, hate group writing. The website is a high quality website that is easy to use to get the information you want about our flamenco group in Saint Louis. The society is a nonprofit organization dedicated to the arts. Like most such organizations, making ends meet financially is a challenge - there is no big money behind the society.
With all that, you would think that it would be easy to be listed in all the search engines. In fact, it is not easy to be listed in all the search engines. We'll talk about the difficulties in getting listed and the search engine ethics implications, and then commend a few search engines.
Let's start by looking at cataloging websites from the search engine point of view. Their goal, ultimately, is to make a profit. Almost all see the route to profit must start with satisfied users. They understand that they need to provide their users with the best information possible while making it easy for users. What do they need to do to provide this service? They must:
| 1. | Make money. A business doesn't survive without making money. |
| 2. | Have a very large base of pages for their users to search. Some engines now have well over a billion web pages cataloged. |
| 3. | Make it easy for the users. If a user can find what they want, they will come back again. This means a good cataloging system, fast response, and hits on what the user wants. |
| 4. | Never offend the users. With the exception of a few search engines, they don't want to have pornography, hate groups, violence listed in their engines. |
| 5. | Don't support illegal activities. No legitimate search engine really wants to help someone find out how to make poison gas, sell drugs, or provide a pedophile connection. |
The search engines' ethical difficulty is reconciling number 1, profit, with numbers 2 and 3, user service. Meeting numbers 4 and 5 means not listing immoral and illegal websites, which requires people to scan potential listings and reject the bad ones. In addition, people can try to list a website with the right keywords, and can filter out websites which will not interest anyone. Having people review a website prior to listing is a significant cost and adds delay in listing, but it can't be avoided. We have incredibly fast machines available, but they don't have judgment.
They do it in several ways.
| 1. | Search engine employees and volunteers search the web for interesting materials. |
| 2. | Webmasters and users submit their websites for consideration for listing. |
| 3. | The search engine gets website listings from other search engines or online directories. |
| 4. | Some search engines send out requests for information to several other search engines for each request you make of them. These are called "metasearch" engines |
Let's look at the cost of each of these techniques.
The first technique requires putting employees to work searching. This is fairly expensive, but essential to getting comprehensive listings. It takes some research on any subject to find the best websites covering it. This research is why you will find some search engines are extremely good on certain topics. The "Ask Jeeves" search engine is extremely good at providing well-organized information on a number of subjects.
The second technique involves getting forms submitted by the general public. This is very expensive for the search engine company, because they must look over a huge volume of material. There will be duplicate requests to look over many sites. Because many websites of questionable virtue are submitted, most search engines will not list a website until its employees check it out. This can take considerable time. Giving users priority for money is a practice that some search engines make. This is a practice of questionable ethics.
The third technique, getting the information from other search engines, probably involves a fee, but allows the search engine to cover most of the websites that have ever been cataloged. It is possible, then, for a small search engine company to provide broad coverage. The net result of this is that a set of 20 search engines probably only has about 7 independent data sources. If you don't find your result with one search engine, switching to another may still result in the same data. You have to know the search engines well to get the results you want.
The fourth technique is to ask other search engines in real time. It is even possible for a metasearch engine to not do any direct collection of websites, relying only on what other search engines have done. The difficulty in doing this is showing potential users that your search engine is actually adding something of value.
Looking at financial reports, you will see that most search engines lose money. Even though the search engines have made our modern use of the internet possible, they can't operate without money coming in. They are generally very secretive about their income sources, so the techniques listed here will not apply to all search engines. Not all search engines use all of these revenue sources. Like many ethical issues, the income sources are at the root of most questionable ethics. Here are some of the sources:
| 1. | Charge advertisers for presenting online "banner" ads to users |
| 2. | Collect marketing data on consumer habits, then selling the data or using it for targeted advertising. |
| 3. | Charging websites to become listed |
| 4. | Charging websites for better placement in lists |
| 5. | Charging websites to purchase keywords for themselves |
| 6. | Charging users for searches |
| 7. | Charging other search engines to use their catalog |
If you know of other sources, please send an email.
Direct advertising to users
The first moneymaking technique is direct advertising. Most search engines have advertisements running on your browser. We've come to expect advertising as the price of a free search engine. The ads are often called "banner" ads.
Advertisers are willing to pay more for advertising if it is targeted at people who are more likely to buy. For example, readers of Fortune magazine have much higher than average incomes, and are interested in investing. An advertiser would be willing to pay more (per subscriber) for an ad in Fortune if they wanted to sell investments, because an ad in Fortune will bring more sales.
Collection of marketing data
Search engines have an opportunity to target ads at select groups. They collect marketing information in a few ways.
First, you often register with a search engine to customize it. Your registration information gives your name, your address, and several other pieces of information. While few search engines will directly sell this information without your consent, I believe many use it to target advertising.
Second, the search engines get information on what you are searching for directly every time you submit a search. They can also find out information about where you were before getting to the search engine. By collecting this information over a period of time, the search engines can compile a profile of your interests. If you made a search for websites that would sell new cars, then you are a good prospect for auto advertisements.
In fact, most online shops collect such data and try to compile information on you as a consumer.
Targeted ads can sell for five to ten times as much as untargeted ads. If a search engine could gather detailed information on all its users, it could increase its revenues significantly. A whole industry has grown up to turn the click data into marketing information.
The collection of this data raises several ethical questions we will address in the next section.
Charging for listing
Some search engines charge for listings. One charges $200 for a listing that will be entered within 48 hours, $50 for a listing that will be entered within 8 weeks, and nothing for a listing that might be entered when they get around to looking at it.
Search engines that do this feel that they are providing a valuable service to the webmaster by listing the website. By charging significant fees for listing, they expect to avoid insignificant websites. What they don't understand is that they really become a paid advertising directory rather than a legitimate search engine.
Charging for list placement
You may wonder how website ranking is done on the website hits that you get on a search. If there are a million hits, only the top ten or twenty will ever be examined by the user. It's worth big money to get listed first when a set of hits is pulled up by a user.
Generally, a search engine will look at the words in the title and website text to see how close a match they get to the search words. A numerical score from 0 to 100 tells how close the match is. The highest matching scores are shown first.
There is a description field, actually a <META> tag in which a website will give a self-description. Abuses by some websites in the wording of these descriptions has led some search engines to ignore website-generated descriptions, relying strictly on their parsing of the website.
The actual criteria used to generate the match scores is generally proprietary for each search engine.
I recently came across a search engine that was willing to raise a website's score in return for money. This practice is not one which most users would be happy with. It also raises the ante for websites. If you don't have a significant promotion budget, you won't really be visible to the public in some search engines.
Selling Keywords
Some search engines will also sell keywords
! If you buy a keyword, then your website would automatically come up first.This practice is another step towards search engines becoming mere paid advertising directories.
Charging users for searches
Charging users for searches is the pre-web technique that was generally used. Such services as Lexis/Nexis charge to look up relevant articles. To put this in perspective, the quality of the information they produce is generally far superior to what a search engine would produce.
The difficulty with such a technique is that
there will be a limited base of users. With a limited customer base, advertisers are not interested. With the ubiquitous internet, the name of the game is to get more users, get more hits on the search engine.Most search engines provide all their services for free to the users. Northernlight charges for certain special searches, and there are probably several others with special value-added searches.
Charging search engines for cataloged websites
Some search engines provide data to others. I don't know the financial relationship, but expect that some money changes hands to provide the data.
Looksmart, for example, claims to provide the base of cataloged websites that several major search engines use. The Netscape Open Directory provides information for several search engines. Altavista, generally considered the engine with the largest catalog, is often searched when there is no result on another search engine.
There is, to the best of my knowledge, no code of ethics for search engines.
Since they are generally owned by corporations and provide a free service to users, we could say "caveat emptor (Let the buyer beware)" and ignore ethics.There is not yet a great hue and cry among the public for ethics in search engines, or at least not one I can find by looking on search engines. Most people are not aware that some of the search engines they are using are, in fact, paid advertisements.
I would like to see a code of ethics for search engines. I have some suggestions for introducing ethics.
First, in keeping with the general idea of freedom of the web, there should not, except on the issue of user privacy, any government laws. Whether search engines have ethics and integrity will not directly cause any loss of life or financial harm.
Second, any code of ethics for search engines should be voluntary. If people are interested in a search engine that does not meet standards, they should be free to use it. It should be enough that the search engine can display a seal showing that they are in compliance with an ethical code.
Third, any code of ethics should not be set directly by a search engine industry trade association. Industries are often not very good at setting standards for their own behavior with the public. I would suggest that the ethics code be developed, with both user and search engine input, by ethics, computer science, and business professors in universities around the world. The broad nature of a proposed body of ethics would make it difficult to achieve consensus, but only a broadly based body could produce a code that is accepted all over the world. Professional societies, such as the Institute of Electrical and Electronic Engineers, already have ethics committees. Perhaps these could get involved.
Fourth, a search engine that meets the code of ethics would be able to display an official seal with this claim. Search engines failing to meet the ethical code would, of course, not be able to make this claim. For the meaning of this to come to the user, there would have to be some advertising on the web of the meaning of the seal. Search engines that met the code of ethics would likely help in publicizing the meaning.
To me, the key point in search engine ethics is disclosure to the end users of potential financial conflicts. If a search engine sells placement or keywords, it should disclose this clearly so users know that they are getting a biased view of priorities. If a search engine sells marketing data, this should be disclosed. If a search engine uses marketing data in targeting advertising, this also should be disclosed.
The search engine should disclose to the users the general source of its data. The user would then know which other search engines would give more independent results. A user could assess that the search engine had more direct information if the search engine collected more original data, spidering websites itself.
We've found that the most effective way of submitting a site to a search engine is to go to each of the major search engines and find where they will allow you to submit a site. This is not easy to find, and the submission procedure does not work properly in all websites. The end result is that it appears that many websites consider recommendations as a nuisance, to be added if there's some extra time available. Let's look at a few search engines that we can say nice things about.
So far the only search engine that has picked up our submission of our new website is yahoo. Not only did yahoo list our site within a week, they sent a nice email, part of which is attached below.
Thank you for taking the time to add your site. We rely on users like yourself to make Yahoo! complete and comprehensive. In order to keep Yahoo! accurate, please also let us know of any future changes that might affect your listing.
The metasearch engine whatuseek provides a search capability for individual websites (they run our search capability) for free.
The askjeeves search engine provides a good set of material for learning about search engines. There is, however, little material critical of search engines on the web
We hope to be able to add other search engines for commendations on this site.
If you have commentary on the issue of search engine ethics, and particularly if you believe there should be a code, please email me at wittenberg@ieee.org . I will try to put together a group of interested people.
Peter Wittenberg