Flight Patterns

(Video requires QuickTime)

Last year, a three-minute video depicting air traffic patterns in the U.S. garnered quite a bit of attention across the internet. It was created by Aaron Koblin (who also gives credit to Scott Hessels and Gabriel Dunne), using actual FAA data, thus resulting in a visualization of flight paths into, out of, and throughout the country. As the video begins, these paths are first depicted as light streaks against a dark background, while the time of day and total number of airborne planes shows up at the bottom of the screen. As it progresses, the data is represented in different ways to show the emergence of various patterns. Immediately, as the first paths begin to form, the viewer sees a network take shape, with the busiest nodes corresponding to major airports in large cities. The video notes that patterns form even without geographical features; indeed, the viewer can make out continental borders and familiar cities without any such markings.

While the video has received more appreciation for its stylistic rather than purely scientific merits, it is suggestive for our study of networks and their applications. For example, it is clear the presence of connecting edges between cities is time-dependent; as night gradually turns to day, activity slowly picks up starting in the east, and ending in the west. It would be interesting to consider the consequences of having a network (not necessarily flight patterns) that might have some explicit cyclical time dependence, and what that means for each of the nodes, for the idea of betweenness, or for power held by each of the nodes. For instance, early on in the class we saw a graph representing relationships between high school students. However, the graph didn’t contain information about the duration of relationships, or when they happened, only that they happened. Other suggestions which come to mind are paths through which diseases might spread (through infected passengers), or how air pollution might be concentrated (in areas of dense activity).

http://www.cmu.edu/joss/content/articles/volume2/Bonacich.html

This article was taken from the Journal of Social Structure, an online journal on social structure, and provides an in depth look at almost the exact same information that we have been learning about in class. The purpose of the paper is to do look at the various types of social structures and the way in which they can change over time through the use of computer simulations. It looks analyzes social networks in a manner very similar to the way we have been in class by looking at the gains of each member of the network and finding people who have more power than others (through the use of an exchange game), even coming up with classifications for the position (of power) that a component can have. In fact, some of the structures that the author examines and names are one’s that we have looked at, such as his four-chain and tee networks.

The paper does have points where it differs from what we’ve learned in class, by adding different variants on the way the game is run and the networks are set up. The first thing to notice is that the author does not use nodes and edges to depict the networks, instead using a square grid with each square connecting to a maximum of eight other nodes. Another difference is the way his game is run. The main difference is that his variant allows for the “movement” of the people in the networks - if a person is in a weak position in a structure, it can move away from that structure and possibly form another one with other people. Also, exchanges are made between people only when they get the best offer possible and once they do so, they stop moving. With this game variation and the method in which he uses to depict the networks, he can create computer simulations in which the computer moves nodes and checks for the power of each node according to the rules of the game, depicting the evolution of a the network. A final situation he looks at is the usage of these simulations to model the evolution of bipartite networks, providing a visual depiction of the method of finding market-clearing prices.

The author concludes that the paper doesn’t actually have any analytic results in regards to network evolution and stability. However, the simulations have shown the author several things. First of all, if members in a social network that are not satisfied with their power are allowed to move, most networks will reach a network where the distribution of power is about equal for everyone except for a network in which there are two categories of unequal size(i.e. buyers and sellers). He also found that stability in the pattern of exchanges and profits in networks where there is no movement allowed is present in all networks except those that are what he calls coreless (there is no pairing such that each person is satisfied with his partner). When looking at the stability of the network structure itself, he found that Strong Power networks (networks where there are those who are very powerful at the expense of others) were the only unstable ones. In addition, he found that in non-bipartite networks, those that had an even number of people ended up with indeterminate (network has only slight effect on power - power mainly effected by relations between people) or equal power networks. Those with an odd number ended up with coreless networks. In bipartite graphs, those that have an equal number of people in both categories will end up with indeterminate networks.

Last month, representatives from Palantir Technologies (which is indeed a reference to the Lord of the Rings object of the same name) held an information session at Cornell where they discussed two revolutionary data analysis tools.  One essentially creates networks out of the underlying structure and connections in data, using the methods by which the data is entered and information about who enters it at what time from where to complement this, in order to reveal insights about data that could otherwise be very tedious to discover, or ignored entirely.  The other is concerned with financial analysis and prediction of trends in stocks and auctions.  While their product page (http://www.palantirtech.com/products.html) is particularly vague, their tools target specific deficiencies in the way that information is processed today.

The first tool, Palantir Government, attempts to solve the problem of data overload, primarily within government intelligence agencies. Currently, as data is amassed, it typically remains in several locations (papers in file cabinets, multiple email accounts, databases, etc.) and cannot therefore be easily considered as a coherent whole, with connections between different pieces of data immediately evident.  This tool, however, brings all this data together into one central, versioned store accessible by everyone in the intelligence agency so as to avoid duplicate or wasteful efforts. (By “versioned,” I am referring to the way that source control utilities such as CVS and Subversion automatically keep revisions of data as it is modified, allow “branching” [creation of separate paths that can be worked on by different people, then merged together when done], etc. See http://en.wikipedia.org/wiki/Revision_control for more information.)  Then, any user can simply search for a name, date, or any other information relevant to his or her case, and the tool will create a graph with a node for the search term at the center, containing all the information in the system about that person grouped by category, and links to all other items that that node has had contact with (phone calls will link people, jobs will link coworkers, and so forth).  When available, transaction histories (credit cards, bank statements, etc.) visually display the flow of money between individuals.  Although this tool does not employ heuristics to determine points of interest automatically, Palantir’s decision to concentrate on how all available information is displayed, particularly with regard to connections between various nodes, makes discovery of points of interest practically effortless by users of the system.

The second tool, Palantir Financial, provides an easy way to search for trends across different stocks, compare predicted future returns, and manage a wealth of information about any financial service (stocks, bonds, hedge funds, bank accounts, and so forth).  What makes it unique is that searches are defined in terms of relative parameters instead of absolute parameters (”stocks whose sell prices increase at least 5% over two months” versus “stocks whose sell prices are over $75″).  This allows one search to be applied to any stocks across any interval of time.  Furthermore, every aspect of this program related to displaying data works on a similar concept of filters, allowing the displayed data to be truly dynamic, and allowing users to respond as quickly and accurately as possible to changing market conditions.  As with the first tool, this tool does not employ heuristics or algorithms to make decisions about finances automatically.  Rather, it provides the most intuitive view of the data (which computers are better at), so that the users can concentrate on making decisions about it (which humans are better at).

Although these tools are only available to government organizations and high-profile corporations at the moment, they are definitely indicative of the future of data analysis and display.  The trend from algorithmic and heuristic analysis as a means of automatically making decisions about data is giving way to the use of those techniques to display data in the most intiutive way posslible, providing insights into previously unnoticed connections between seemingly unrelated things, or previously overlooked connections between the things that seem so obviously related that the possibility for them to be connected in other ways seems incredulous.

In lecture we began talking about power in networks and the characteristics of nodes that might lead to power. Dependence, exclusion, satiation, and betweenness were all cited as being typical traits of powerful nodes. The research that identified these traits was based on experiments in which humans bargained over money, with the most powerful nodes being the ones who emerged with the greatest earnings. Many cases discussed in Malcolm Gladwell’s “The Tipping Point,” however, involve the social network which transmits ideas and fads, where power isn’t necessarily the ability to bargain for the best deal - it’s the ability to influence other peoples’ opinions.

Gladwell characterizes powerful nodes as Mavens, Connectors, and Salesman. In the terms of our lecture, mavens exercise the power of dependence: others are dependent on their expertise for making informed decisions. Connectors’ power is drawn from their betweenness in that they have many contacts and are able to bridge people far across the network. A salesman’s ability to spread his ideas and opinions to others doesn’t really fall under any of the categories we discussed in lecture, but could be classified as the power of persuasion. Its not really power due to a certain position in the network, rather it is due to personal traits. Given an influential position in the network, a salesman could have the ability to sell an idea to the connectors and mavens who could bring it to the rest of the network.

One interesting phenomenon that Gladwell talks about is the dependence of social networks on context. Chapters 4 and 5 both talk about how a message’s effectiveness is dependent upon the context it is transmitted in. For example, in an effort to reduce crime in New York City’s subway system the Transportation Department spent millions of dollars removing graffiti from the trains. This is counterintuitive given the model of power in networks that we have discussed - a more natural idea would be to target the most prominent criminals (powerful nodes) with the idea that by stopping them you would send a message to other, less powerful, offenders. As it turned out, the subtle change in the environment caused by cleaning up the cars sent a very powerful message to subway riders and crime dropped immensely, even throughout the city. I think these environmental factors are an important part of social network analysis, but they are probably difficult to model. Still, I think its something to consider as we continue our discussion on power.

Gladwell’s book talks about power in terms of mavens, connectors, salesman and context; we talked about it in terms of dependence, exclusion, and other characteristics. I think Gladwell’s model is a useful complement to the model we discussed in class for analyzing the division of power in networks.

The auction abstraction is applicable to many different areas.  One interesting are is in the sports world where the free agency process plays a large role (if the league’s collective bargaining agreement allows for this) in where players end up and how teams perform.  The sellers are the players themselves and the buyers are the teams.  Teams payoffs are determined by their valuation minus what they pay but this is essentially how much money they saved.  The payoff for the purposes of this post will only the immediate benefit to the team and will not factor in the players performance or extra revenue generated once the player is signed.

A paper about the effectiveness of the baseball free agency system titled Matching and Efficiency in the Baseball Free Agency System can be found http://www.jstor.org/view/0734306x/di009546/00p0208o/0

Prior to free agency upon the expiration of a player’s contract, his/her only option were to hope for a trade or sign a new contract with the current team.  A baseball player took his team to court and won the right to “free agency” for all players after 6 years of playing in the major leagues.  Free agency really just gives a player ownership of his/her skills as a baseball player and therefore gives him the right to sell that service to the team of his/her choosing.  So again the teams are the buyers and the players are the sellers.  This market has the interesting characteristic that there are many sellers who will sell for significantly less than than their ideal price because 1) even the low salaries are a nice living wage and 2) for some players it is about being able to do something they love which is priceless.

The third page details a doomsday scenario for the sellers in a marketplace.  In the 1980’s the team owners (buyers) were not happy with the perpetually increasing salaries they had to pay their players.  The buyers began to come up with ways to keep the salaries lower such as agreeing to bid on only their own players (eliminating competition and forcing players to accept lower bids) and proposing lower bid in a cooperative in systematic fashion (again effectively getting rid of the competative bidding wars that drive salaries up). 

 The number of players looking for a roster spot always exceeds the number of spots available so there are no market clearing prices.  We can assume that the valuations are independent.  This system is such that buyers are not on a level playing field because some cities are more attractive to play in than other cities and so the player may be willing to accept a minimum salary of more or less money.  This means that the cost of obtaining player x for one team is not the same as the cost for another team.  This complicates the auction process in my opinion.  Team y and z bid for player x.  Player x takes that bid and adds a team specific value to reach a “true bid” for that team.  Team y and z may or may not discover each others’ bids at their discretion and the discretion of player x.  Player x may or may not share the “true bid” because sharing his preference for a team will produce lower monetary bids while sharing his dislike for a team will produce higher bids.

 The paper discusses a system where all bids proposed to a player are available.  This alternative benefits the players less.  Teams are not participating directly with each other but they can view other teams’ bids and cater their bids to the apparent demand for the player.  A bidding war is still very possible though.

The paper also discusses a system where each team simply submits a maximum value they would be willing to sign a player for and each player submits a minimum value they would be willing to play for.  A computer would then find the a combination of players and teams that maximized social welfare according to the team max - the player min.  After that the player/team pairs would attempt to work out deals based on a computer-generated range and if they could not reach an agreement the player would be placed back into free agency and eligible to be bid on by other teams.  This third system is actually quite intriguing although something about it seems to take the “human”-ness out of free agency.  There would be no signing contracts, shaking hands, negotiating, or speaking to each other unless the computer says so.  Efficient system but with interesting implications for the human dynamic of the free agency marketplace…

This article (http://www.inc.com/magazine/19950301/2182_Printer_Friendly.html) from Inc.com discusses the new industry model in Hollywood, that of small businesses connected in a network. The article discusses how more and more outsourcing is being used to put together different parts of a movie in different areas with different companies taking the helm of the mini-projects that come together. This creates an emphasis on developing ties between small businesses and large businesses in order to maintain healthy prospects of getting to work on projects. This mimics the way that social networking sites and network news sites that have been posted about in this blog distribute power and destroy the traditional model of a simple hierarchy. It thus also relates to Gladwell’s concept of taking graph theory and applying it to become a sort of grand unified theory that explains intrinsic behavior, and thus an analysis of the nascent networking model is relevant to this course.

The article also discusses power, as said in the lines, “The real creators of the film, responsible for everything from its special effects to payroll and security, are a host of small companies and freelance contributors who collaborate for only as long as the project requires — and who are drawn from the loose network of specialists that today holds the real power in Hollywood.” As digital effects and post-production companies become more attractive, they thus garner more power for themselves as they will have more corporations interested in them. It also means that within the network these small businesses have to compete with each other, which is relevant to the game nature of this course. The article discusses the “intensified global competition” and “rising customer expecations… born in part from the explosion in choices.” This really emphasizes the competitive aspect, and thus can be related to a game.

The consequences of this system are judged to be beneficial, as is common with market economy strucutres. The classic argument being given is that the competition makes better prices and better products.

Another piece of information that the article has about networks is location. Like in urban theory and planning, clusters have been known to be very important to developing industry because of the networking that it allows the companies withion them. Here the article talks about the importance of where power is being concentrated.

Another network piece of information is of ties, as the article talks about Richard Hart, who got a job as a lightning techinician at MGM through a friend.

http://www.msnbc.msn.com/id/17176086/

The article listed above discusses the decreasing role of specialists in the New York Stock Exchange (NYSE). Within the past few months, the number of traders on floor of the stock exchange has dropped from 3,000 to 2,100. The cause of this decrease in specialists is due to the increased dependence on electric transfers. In class, we discussed how the specialists work: they process ask and bid prices for certain stocks. But, computers can perform this very same function with greater speeds and efficiency. Accordingly, the average speed of a transfer has dropped drastically, from 9 seconds to three-tenths of a second.

The new dependence on electrical systems is due to competition with Non-Intermediated Stock Markets, specifically NASDAQ. In class, we defined Non-Intermediated Stock Markets to be markets that function without specialists, and relied solely on direct trades. These stock markets are “more open and transparent, where all trades are equal and without the influence of brokers that might have a bias.” The increasing popularity in these automated trading networks has forced the NYSE to adapt to the digital age. Slowly, the shouts of “buy” and “sell” are being replaced by computer operations.

However, even with the increase of reliance on electric systems, these specialists are not going to be completely replaced. The article states that humans hold one advantage over computers: human intuition. Computers cannot trade with a so-called “gut instinct” that many specialists rely on. Instead of turning towards a fully computer-run system, the NYSE has become a hybrid market, which “infuses the speed of electronic trading with the experience of seasoned professionals.” With this new hybrid stock market, the NYSE gains the efficiency of running most trades through computers. But with certain cases that demand more attention that most, they still have the ability to run that trade through a specialist.

As Louis Pastina, an executive vice president at the NYSE in charge of the hybrid rollout, puts it: “You might buy a book on eBay, but you might not want to buy a brand new Mercedes Benz that way. The same thing applies here.”

It appears that LinkSViewer has taken the next step by applying graph theory to a new business venture. This company is a “leading-edge Visual Network Analysis software and consulting firm” who actually take the connections between people and businesses and generate a network based on that information. They then sell the resultant picture as “a visual explanation of complex relationships that is easy to read and understand”. An example network on their site is the relational map between Facebook, Friendster, and LinkedIn available here.

Of course they only aggregate publicly available information (e.g. board members and investors of a company), so it’s technically feasible for someone else to do a similar operation (perhaps they only want to research a few companies). However, LinkSViewer advocates their system not as an information database, but as a facilitator for research. It’s interesting to see that the early-adopters of graph theory in business are touting it’s visual qualities as opposed to any theories or conclusions one might draw from seeing the network as a whole. Apparently, like the security and safety industries, (social) network visualization is becoming an area with great potential but no immediately quantifiable economic benefits.

http://community.livejournal.com/ljsecret

If you have a secret that you wish to reveal to everyone and no one, without anyone knowing who you are, this is the place. ljsecret is a site where anyone can write down a secret on a “postcard” of their choosing (preferably one that has to do with the secret) and submit it. The information you send is received by a group of moderators who look through thousands of secrets (submitted of course by others) and pick a number to be displayed on the site.

Once a secret is placed for all the world to see, in comes the part that really makes this web site an interactive social network. It is that anyone is allowed to comment upon the secrets given, and most often entire discussions arise, between those who have a habit of visiting this site, or between first-time strangers.

What has formed then is essentially a virtual community, where people reveal the hidden part of themselves, and let others embrace these strange (often embarrasing or even frightening) facts, comment on them, and perhaps even give some good advice.