Last month, representatives from Palantir Technologies (which is indeed a reference to the Lord of the Rings object of the same name) held an information session at Cornell where they discussed two revolutionary data analysis tools.  One essentially creates networks out of the underlying structure and connections in data, using the methods by which the data is entered and information about who enters it at what time from where to complement this, in order to reveal insights about data that could otherwise be very tedious to discover, or ignored entirely.  The other is concerned with financial analysis and prediction of trends in stocks and auctions.  While their product page (http://www.palantirtech.com/products.html) is particularly vague, their tools target specific deficiencies in the way that information is processed today.

The first tool, Palantir Government, attempts to solve the problem of data overload, primarily within government intelligence agencies. Currently, as data is amassed, it typically remains in several locations (papers in file cabinets, multiple email accounts, databases, etc.) and cannot therefore be easily considered as a coherent whole, with connections between different pieces of data immediately evident.  This tool, however, brings all this data together into one central, versioned store accessible by everyone in the intelligence agency so as to avoid duplicate or wasteful efforts. (By “versioned,” I am referring to the way that source control utilities such as CVS and Subversion automatically keep revisions of data as it is modified, allow “branching” [creation of separate paths that can be worked on by different people, then merged together when done], etc. See http://en.wikipedia.org/wiki/Revision_control for more information.)  Then, any user can simply search for a name, date, or any other information relevant to his or her case, and the tool will create a graph with a node for the search term at the center, containing all the information in the system about that person grouped by category, and links to all other items that that node has had contact with (phone calls will link people, jobs will link coworkers, and so forth).  When available, transaction histories (credit cards, bank statements, etc.) visually display the flow of money between individuals.  Although this tool does not employ heuristics to determine points of interest automatically, Palantir’s decision to concentrate on how all available information is displayed, particularly with regard to connections between various nodes, makes discovery of points of interest practically effortless by users of the system.

The second tool, Palantir Financial, provides an easy way to search for trends across different stocks, compare predicted future returns, and manage a wealth of information about any financial service (stocks, bonds, hedge funds, bank accounts, and so forth).  What makes it unique is that searches are defined in terms of relative parameters instead of absolute parameters (”stocks whose sell prices increase at least 5% over two months” versus “stocks whose sell prices are over $75″).  This allows one search to be applied to any stocks across any interval of time.  Furthermore, every aspect of this program related to displaying data works on a similar concept of filters, allowing the displayed data to be truly dynamic, and allowing users to respond as quickly and accurately as possible to changing market conditions.  As with the first tool, this tool does not employ heuristics or algorithms to make decisions about finances automatically.  Rather, it provides the most intuitive view of the data (which computers are better at), so that the users can concentrate on making decisions about it (which humans are better at).

Although these tools are only available to government organizations and high-profile corporations at the moment, they are definitely indicative of the future of data analysis and display.  The trend from algorithmic and heuristic analysis as a means of automatically making decisions about data is giving way to the use of those techniques to display data in the most intiutive way posslible, providing insights into previously unnoticed connections between seemingly unrelated things, or previously overlooked connections between the things that seem so obviously related that the possibility for them to be connected in other ways seems incredulous.