Nate Silver in his excellent "The Noise and the Signal" warned that we are much in awe of Big Data. But, that Big Data predictions in many fields have been disastrous (financial markets and economics just to name a few fields). With more data, you get more spurious correlations, more false positives, and erroneous answers. In stating so, he also liens on the excellent work of Ioannidis who indicated that over 2/3ds of scientific findings are wrong as they can't be replicated (based on extensive reviews of working papers). In other words, watch out for the many traps of multiple hypothesis testing, especially when you have not even phrased the hypothesis to begin with. "Correlation does not entail causation" still prevails.
Now in a new book (called Big Data) written by Viktor Mayer-Schonberger and Kenneth Cukier, Big Data looks far more promising. Given the size of the sample that often equates to the entire population, you can detect granular relationships between subsets of the data you could never before. And, within this Big Data era correlation seems far more important than causation. Figuring out what variables are predictive gets you far better and rich results than figuring out which ones are truly causal (that often turns into an elusive chase). The author mentions several new tools that are aimed at extracting and analyzing Big Data sets including neural networks, artificial intelligence, machine learning, sensitivity analysis among others. Being unfamiliar with any of those (and very familiar with traditional statistics and hypothesis testing in particular), I can't judge if the author statement is accurate (he is not a quant). Do those techniques truly avoid the traps of spurious correlations, multiple hypothesis testing, model overfit and false positive results?
Can you reconcile both views: Nate Silver vs Viktor Mayer?