Search engines like Google and Yahoo are testaments to the power of programs called Web crawlers—robots that scan the Internet, collect documents, and file them away in searchable indexes. The giants of the search world have typically kept their crawlers under wraps, but Alexa Internet, a subsidiary of Amazon that measures traffic on the Web, has chosen to break ranks. The company has released its crawler—which sifts through as many as five billion web pages every month—online to computer programmers.
Users will have to pay to play with the industrial-strength crawler. But technology experts say the tool could make it much easier for computer scientists and amateur enthusiasts to put new spin on the world of Web searching. (Wired News)
