Smaaz’s Blog

Wikia Release Open Source Distributed Web Crawler

Wikia, Inc. today unveiled major next steps in its work to build a new search platform founded on open-source search protocols and human collaboration at the O’Reilly Open Source Convention (OSCON). In a morning keynote address, Wikia co-founder Jimmy Wales discussed business models and his vision for building the LAMP stack for search, which can be done by assembling open-source technologies.Wales announced that Wikia has acquired Grub, the original visionary distributed search project, from LookSmart and released it under an open source license for the first time in four years. Grub operates under a model of users donating their personal computing resources towards a common goal, and is available today for download and testing at: http://www.grub.org/ .

“We’ve had a tremendous response from very interesting commercial players in the search space,” said Jimmy Wales, co-founder and chairman, Wikia, Inc. “The desire to collaborate and support a transparent and open platform for search is clearly deeply exciting to both open source and businesses. Look for other exciting announcements in the coming months as we collectively work to free the judgment of information from invisible rules inside an algorithmic black box.”

Grub, now open source, is designed with modularity so that developers can quickly and easily extend and add functionality, improving the quality and performance of the entire system. By combining Grub, which is building a massive, distributed user-contributed processing network, with the power of a wiki to form social consensus, the open source Search Wikia project has taken the next major step towards a future where search is open and transparent.

“In looking at the overarching industry, it has become clear that open is the business model of the future,” said Michael Grubb, Senior Vice President, Technology, and Chief Technology Officer, LookSmart. “We are pleased to collaborate with Wikia and believe that Grub will thrive under an open source license. We are happy to be able to assist in the movement to make search a more open proposition and look forward to seeing things progress from here.”

To keep up with the latest developments around open source search or to volunteer, please visit the community wiki at: http://search.wikia.com .

Add comment July 28th, 2007

Three new Features in Live Search Images

The Live Search Blog today announced three new features for Live’s Image Search, that help you find faces, portraits and black and white images. Try these links to see them in action:

1. without the new features – jimi hendrix
2. with new face filtering on – jimi hendrix filter:face
3. with new portrait filtering on – jimi hendrix filter:portrait
4. with new black and white filtering on – jimi hendrix filter:bw

For now, the way you access these new features is by typing. You’ll need to add filter:face, filter:portrait, or filter:bw to your query.

Add comment July 28th, 2007

Google’s Unavailable_After META Tag has Arrived

Google’s Dan Crow announced today that the Unavailable_After META tag is now live:

“We have introduced a new META tag that allows you to tell us when a page should be removed from the main Google web search results: the aptly named unavailable_after tag. This one follows a similar syntax to other REP META tags. For example, to specify that an HTML page should be removed from the search results after 3pm Eastern Standard Time on 25th August 2007, simply add the following tag to the first section of the page:

<META NAME=”GOOGLEBOT” CONTENT=”unavailable_after: 25-Aug-2007 15:00:00 EST”>

The date and time is specified in the RFC 850 format.

This information is treated as a removal request: it will take about a day after the removal date passes for the page to disappear from the search results. We currently only support unavailable_after for Google web search results. After the removal, the page stops showing in Google search results but it is not removed from our system.”

Add comment July 28th, 2007