DARPA Wants to Help You Search the Web Better

The Defense Advanced Research Projects Agency has kicked off a project to fine tune Web searches by topical domain rather than general subjects, allowing it to “maintain technological superiority in the area of content indexing and Web search on the Internet.”

DARPA plans to use the system initially to counter human trafficking, which is enabled by websites, forums and chat rooms.

DARPA said its “Memex” (a combination of the words memory and index) project will, among other things, search “deep Web” content missed by commercial search engines and “will address the inherent shortcomings of centralized search by developing technology for domain-specific indexing of Web content and domain-specific search capabilities.”

Memex will develop technology to enable discovery, organization, and presentation of domain-relevant content. DARPA envisions the new system providing fast, flexible and efficient access to domain-specific content as well as search interfaces that offer valuable insight into a domain that previously remained unexplored.

DARPA wants researchers to develop advanced and highly automated Web-crawler software to penetrate sites and resources that have erected crawler defenses to aid domain-specific indexing and a domain specific search engine.

The research agency said the Memex system can help  counter human trafficking, which, “especially for the commercial sex trade, is a line of business with significant Web presence to attract customers and is relevant to many types of military, law enforcement, and intelligence investigations.”

Human trafficking forums, chats, advertisements, job postings and hidden services “continue to enable a growing industry of modern slavery,” DARPA said. “An index curated for the counter trafficking domain, including labor and sex trafficking, along with configurable interfaces for search and analysis will enable a new opportunity for military, law enforcement, legal, and intelligence actions to be taken against trafficking enterprises.”

The Memex project takes its name and inspiration from a 1945 article in The Atlantic titled As We May Think, in which Dr. Vannevar Bush, head of the White House Office of Scientific Research and Development, envisioned an analog computer to supplement human memory, the Memex, which would store and automatically cross-reference all of the user’s books, records and other information.

DARPA expects the Memex research project will run three years with proposals due April 8.