Internet Anthropologist Think Tank

Friday, August 01, 2008

The biggest search engine ?

Menlo Park based Cuil will launch later this evening with an index of 120 billion web pages, making them arguably the most comprehensive search engine on the web (Google doesn’t disclose the size of their index, although they claim to know about a trillion unique web pages) (Update: see our very early testing here). They’ve also dropped one of the “l’s” from their name - previously the company was “Cuill.” Either way, it’s pronounced “cool.”

The super-stealth search project was founded by highly respected search experts. Husband and wife team Tom Costello (CEO) and Anna Patterson (VP Engineering) were joined by Russell Power. Patterson and Power are also ex-Google employees, and the company has been the subject of intense speculation over the last couple of years.

Much of the secret sauce of Cuil is in the way they index the web and handle actual queries by users. Both are costly to scale, and Cuil claims to have found a way to massively reduce those costs. That allows them to run the search engine a lot cheaper, even at Google-scale should it ever reach that point. By some estimates, Google spends a billion dollars a year to run the back end infrastructure of it’s search business.

Source:

BACKGROUNDER: Billions of pages a day.

G

.

Labels: Anna Patterson, Cuil, google, Search Engines, Searching, Tom Costello, Web search engine, World Wide Web

Share |

Monday, July 21, 2008

Terrorist Math, 60 million pages and growing.

Our "Terrorist Sites" search engine has over 1,200 web sites.
The search engine has Indexed over 11,700,000 web pages.
Search Terrorist over 1,200 sites, forums
Our estimate is that there are 5,000 terrorist related web sites.
And another 1,000 secret sites.

Our G2 staff puts the number of terrorist web pages at over 60,000,000, terrorist related web pages. And growing.

60 million web pages promoting terrorism.

This suggest a Paradigm for how al Qaeda is able to regenerate, recruit suicide bombers and fund its self.

And I admit a a lazy preference to collect Terrorist web sites, and grow the list of indexed pages into the millions, because it is harder to take down terrorist web sites, and huge numbers of indexed web pages LOOKS good.

The ability to collect intelligence from these sites feels good and does contribute to our work.
But the better Paradigm would be to see the number of indexedd pages decline as the terrorist web is shut down, the loss of their C2, recruiting and funding drives, and Info WAR abilities.

I understand the drive for US Intelligence agencys to want to collect intelligence.
But it would be more effective to shut them down.

The preference for collecting intel vs closing them down is counter productive.

Allowing them to grow from a few hundred sites in 2002 to thousands, accounts for
al Qaedas ability regenerate to continue to grow even in the face of sever attrition.

It is my desire to see the number of indexed pages decline.
The sites should be copied and then shut down.
We don't have the funding to undertake this operation, secure storage etc..

We are watching for the US military to start to control this domain.

We are seeing the web masters switch from Iraq to the Taliban, and in languages
we can't translate, Pashtoon and Urdu.