Internet Searching Is Not Always What It Seems

By ROBERT BERKMAN

It's no secret that the World Wide Web has greatly changed how we create and retrieve information. But many users of the Web don't realize that the results of their searches may be skewed by a new industry that has emerged to advise people about how to improve their Web pages' rankings in search engines -- and even in directories, which employ human indexers and librarians to ensure that each site is listed in the right category -- thus increasing the sites' hits.

The new experts are not advising people to pay for better rankings for their sites. None of the major search engines or directories accepts payment to increase a ranking -- although some directories, like Yahoo (http://www.yahoo.com) and LookSmart (http://www.looksmart.com), offer an express service for $199, which means that sites are reviewed for a listing within a few days, rather than several weeks or months. The few larger engines that do rank sites by how much they pay -- Go To (http://www.goto.com) is the most prominent of those engines -- tell users they do so. And although some search engines offer what they call "keyword buying," that simply means that a company's advertising banner appears when a searcher types in certain words. For instance, if you type in "sneakers" on AltaVista (http://www.altavista.com), a New Balance ad will appear -- at least, until another company pays for the privilege.

Nor are the experts teaching people to trick search engines, although some of the engines can be fooled. A couple of years ago, in early April, I was doing a search on "taxes" and "I.R.S." on AltaVista, and the first site listed in my results turned out to be a porn site, though its title related to filing taxes. Apparently, the site fooled AltaVista by repeating the words "tax," "I.R.S.," and related terms in the site's title, as well as in the invisible portion of the page called metatags (metatags can be read by search engines but don't appear on the version of the site that a visitor sees).

Keyword stuffing, as that tactic is called, and other attempts to trick search engines continue, of course. Search engines have gotten much smarter at detecting such tricks, and will ban sites that they catch using them. But, like an arms race, some site owners are always trying new strategies to bypass the engines' defenses, causing the search-engine vendors to come up with new solutions, and so on.

Most of the advice about how to increase a site's rankings is quite benign, even obvious. Much of it boils down to creating a clear and accurate title for a page, using metatags effectively, and learning the correct submission guidelines for each search engine. Nothing wrong with that -- in fact, you would be more likely to find the sites you want if everyone followed such advice.

Other advice is not as innocuous, however. For example, some experts recommend that site owners create what are called bridge or doorway pages, which are written for the sole purpose of getting high rankings on search engines. A site may have dozens of those pages, each focusing on different keywords, and each aimed at a particular search engine's ranking formula. Once you reach one of those bridge pages, you are immediately forwarded to the site's real home page. Naturally, that could mean a detour from the topic you're searching, and a waste of your time. As Danny Sullivan, editor of Search Engine Update, explains, "Webmasters will flood search engines with bridge pages," pushing more-relevant pages out of the top rankings.

Perhaps more troublesome is the increasing conflation of advertising and editorial content on the Web. If marketers want to get their sites listed early in search results, and thus to attract more users to the sites, they need to offer more than a short sales spiel, price information, and product features: They need good content. The trouble for searchers is that there is a great deal of difference between the information you find about alternative energy on Exxon's site, say, and a scholarly article on the subject at a university's Web site.

Another reason highly ranked pages are becoming more commercial is because sites that are selling something have the greatest incentive to ensure that their pages are ranked highly. Thus, they are the most likely to invest their resources in trying to manipulate search engines. Searchers are increasingly likely, therefore, to find pages from commercial sites ranked higher than purely informational ones. (Search engines that rank pages based on how many other pages are linked to them -- a notable example is Google, at http://www.google.com -- seem to do better at weeding out the commercial sites.)

More than ever, searchers will find ostensibly hard facts on all types of sites, and they will need to determine who is behind the site, and what the mission and qualifications of those people are.

Unfortunately, it's not as easy to evaluate sites as it used to be. Until recently, ".com" indicated a commercial site and ".org" a nonprofit one -- so the suffix used to be a good way to tell sites apart. But now site owners may select either suffix when registering their sites with Network Solutions, the entity that assigns domain names -- although ".gov" and ".edu" addresses are still limited to purely governmental and educational sites.

What can you do to avoid being lured to less substantive sites? You can improve your search strategies by adding keywords to your search, which should give you fewer, but more-relevant, results; and by adding "not" or the minus sign to exclude terms like "order" and "buy," which appear on many commercial sites, or the names of companies that turn up often on less-relevant pages in a search. And, although it is still true that the sites at the top of a list of search results are likely to be the most relevant, don't neglect the sites listed lower in the list, especially if you find that the ones at the top are commer cial.

You can also get better results by being careful about how, and where, you search. Use directories like Yahoo or LookSmart instead of standard search engines, which don't have human experts to check what categories sites appear in.

However you reach them, use sites that you trust. For example, LibraryHQ (http://www.libraryhq.com) is run by a trained librarian who lists sites of interest to other librarians. Such vertical portals -- or "vertals" -- are becoming increasingly popular as a way to cut through the vast amount of information on the Web.

Online databases are also good sources of information. You can search them on the Web or through most libraries. Some free databases are available through the BigHub (http://www.thebighub.com). But don't limit your searches to the Web -- most of the world's knowledge still resides in print.

Finally, when you are on the Web, look out for commercial sites that masquerade as informational ones. Always find out who is behind the site, and what their mission is. Look for a link on the page to a sponsoring organization; if you find one, note the organization's purpose. If any directors or board members are listed, check out their other affiliations. Look for any political, ideological, or commercial agenda that could detract from the site's credibility. If you can't find any information on the site itself, try a search for the site's owner on a search engine.

Whether the results are helpful, manipulative, or downright deceptive, the owners' attempts to obtain a high ranking for their sites are a new force worth reckoning with.

Robert Berkman is a member of the faculty of the graduate media-studies program at the New School University, and conducts workshops on searching the Internet. He is the author of Find It Fast: How to Uncover Expert Information on Any Subject, the fifth edition of which will be published later this year by HarperCollins.

Easy-to-print version

E-mail this article