Searching in a Small World

In the 1960’s Stanley Milgram performed an experiment where subject individuals were given a letter and asked to send this letter to an individual unknown to the subject through a chain of personal acquaintances. The result, that these letters flowed a chain of weak personal acquaintances, was used by Mark Granovetter in “The Strength of Weak Ties” to support for his observation that weak ties (acquaintances) and not strong ties (close friends) are the most important trafficker of information in a network.

The Free Network Project (Freenet) is the architect of a fully-decentralized, anonymous network, where each user (node) stores many, encrypted small pieces of many different files that represent a small subset of the files published on the network as a whole. The user, him or her-self, does not know what small bits of files are stored by the network at any one time and does not explicitly know where to find all the pieces that constitute a file they want. The basic network design is described in Ian Clarke’s thesis, “Freenet: a Distributed Anonymous Information Storage and Retrieval System”.

Inherently, searching for information in this type of decentralized network cannot be done via some centralized file indexer as pages only reside temporarily in the same location and the fact that the user needs to know what they are searching for before they can view results inherently destroys the capabilities of any global indexer because it would need to analyze a sequence of queries across this network for every possible query string.

So, searching for information is relegated to a process where a node asks other nodes it is aware of to forward a request for information to nodes that are likely to be close to the actual information. But, as Milgram showed in terms of letters with his experimental observations, the number of steps before the request for information would reach an authority on that information is surprisingly small. This phenomena is known as the Small-World Phenomena. For a more intense algorithmic analysis of these networks, see professor Kleinberg’s paper: “The Small-World Phenomena”.

Suppose one wanted to search this network by keyword and entered a query for information on a certain subject. For the search, one wants to find not only the files needed across the network but also the highest valued authority on the subject. The Freenet routing system increases the number of copies of much-trafficked files as well as moves files closer to where they are being accessed. As it currently stands, you have to know the key of the file you are searching for but these keys also include namespaces that describe general topics.

Suppose your first hit was pointing to the key for a file and as you access that you get returns from many other nodes saying that as a group they had all found a different authority that you might want to look at. This could potentially be an example of a useful information cascade, where there might be multiple misleading pieces of information one receives as there is no way to trackback the goodness of each hub but as the new information has a high in-degree it could potentially be useful.

In sum, searching over anonymous, decentralized networks proves to be much harder than the standard search-engine indexer. In the shadow of these issues, we face multiple questions. How can an agent apply a social and algorithmic analysis that provides an effective method for traversing a graph? How could we ever anonymously sell these search results? Is an anonymous, decentralized network an efficient, better way to store information?

Posted in Topics: Mathematics, Science, Technology

Responses are currently closed, but you can trackback from your own site.

Comments are closed.



* You can follow any responses to this entry through the RSS 2.0 feed.