Prof. Alex `Sandy’ Pentland, MIT Media Lab
http://media.mit.edu/~pentland
pentland@media.mit.edu
Adam Smith and Karl Marx were wrong — or at least they only had half the story. The daily ebb and flow of our societies is the sum of billions of individual exchanges — people trading information, money, goods, or just gossip. When we average over these billions of individual events we obtain the familiar groupings we call markets and classes as well as statistics such as market prices or percentages of voters. These averages and groupings were the foundations for the grand theories of the Enlightenment and remain the subject of most social theorizing and political reasoning.
But when we examine the detailed patterns in these billions individual exchanges we find that we can begin to explain many things that appear as random fluctuations when discussed in terms of groups and averages. Examples of such phenomena are market crashes, bankruptcies, fads, and political movements. The new lens that lets us examine society in such fine-grain detail is known popularly as Big Data: billions of telephone call records, credit card transactions, and GPS location fixes. These new digital information sources let us precisely measure patterns of interaction between people, between people and merchants, and then chart the patterns of experiences people have as the move about their city.
The patterns that provide the most insight have to do with the flow of information. The flow of information can be seen in telephone calls or social media messaging, of course, but also by assessing for the amount of novelty and exploration in individuals’ purchasing patterns (as seen in credit card data) and physical mobility (as seen in GPS tracks).
Here are some examples of insights that can be gained from examination of these detailed patterns of information flow.
Market bubbles: Our research has shown that the richness of information flows in Foreign Exchange (FOREX) and commodity trading accurately (measured by social media messaging) predict return on investment. This `richness’ measure can vary between groupthink (everybody saying the same thing) to isolation (nobody talking). At the groupthink extreme we get financial bubbles and at the isolation extreme there is not enough information flow for good market dynamics. In between is a `wisdom of the crowd’ region with high returns. In real-world experiments, we have shown that by applying appropriate incentives we can shape the market to remain in this healthy region.
Social outcomes: Our research has shown that patterns of information flow (measured by telephone communications) can accurately predict social outcomes such as crime rate and incidence of infectious disease along with regional GDP and creative output (as measured by patenting frequency). In real-world experiments we have again shown that we can shape these information flows through appropriate incentives and see improved outcomes as a consequence.
Growth and creativity: Perhaps most interesting findings from our research are that patterns of exploration (measured by purchasing behavior, physical mobility, or communications) are correlated with productivity growth and measures of creative output. We have seen this most clearly within companies, but the same patterns appear to hold for cities and entire regions. This suggests that incentives for greater exploration and mixing of information, experiences, and ideas will result in greater growth and creative output.
These scientific findings illuminate the promise of Big Data for governance and policy. When we use Big Data to look beyond aggregates such as markets, classes, and parties and instead examine the fine-grain patterns of information exchanges, we can have much greater visibility and control of market behavior and social outcomes, and can begin to set scientific, reliable policies for growth and innovation. Just as importantly, Big Data can give us unprecedented instrumentation of how our policies are performing, so that they can be quickly adjusted and revised as needed.