The New York Times posted an article from John Markoff :Government Aims to Build a ‘Data Eye in the Sky’, informing that a US Intelligence Unit launched a research program to analyse public data and find predictions of social and political relevance.
Now social scientists are trying to mine the vast resources of the Internet — Web searches and Twitter messages, Facebook and blog posts, the digital location trails generated by billions of cellphones — to do the same thing.[combine mathematics and psychology to predict the future, as the ‘psychohistory from Isaac Asimov]
The most optimistic researchers believe that these storehouses of “big data” will for the first time reveal sociological laws of human behavior — enabling them to predict political crises, revolutions and other forms of social and economic instability, just as physicists and chemists can predict natural phenomena.[…]
This summer a little-known intelligence agency began seeking ideas from academic social scientists and corporations for ways to automatically scan the Internet in 21 Latin American countries for “big data,” according to a research proposal being circulated by the agency. The three-year experiment, to begin in April, is being financed by the Intelligence Advanced Research Projects Activity, or Iarpa (pronounced eye-AR-puh), part of the office of the director of national intelligence.The automated data collection system is to focus on patterns of communication, consumption and movement of populations. It will use publicly accessible data, including Web search queries, blog entries, Internet traffic flow, financial market indicators, traffic webcams and changes in Wikipedia entries.
No need to mention that they also mentioned the data privacy issue in the article. There are many comments to this news, and I extracted here an important part from Ike Solem’s first comment:
The fundamental flaw in Asimov’s notion of “predicting history” involves the mathematical concept of chaos, otherwise known as “sensitive dependence on initial conditions.”
[…] certain features of physical (and biological) systems exhibit sensitive dependence on initial conditions, such that a microscopic change in a key variable leads to a radically different outcome. While this has been heavily studied in areas like meteorology and orbital physics, it surely applies to ecology, economics, and human behavioral sciences too.
Thus, it’s a false notion that by collecting all this data on human societies, one can accurately predict future events. Some general trends might be evident, but even that is very uncertain. Just look at the gross failure of econometric models to predict economic collapses, if you want an example.
So there is always the possibility of an unforseen agent that changes the predicted behaviour. Still, much more trends will be uncovered from the available big data sets than the ones discovered by human minds as it is up to now. But what about the ‘quantum effect’? If a trend is announced publicly, would that announcement make people to follow it just because they are expected to do so? Or otherwise, wouldn’t it make them change their behavior radically? I think we are still far away from human behavioral prediction.
