Big Data vs multiple hypothesis testing?

Question

Nate Silver in his excellent "The Noise and the Signal" warned that we are much in awe of Big Data. But, that Big Data predictions in many fields have been disastrous (financial markets and economics just to name a few fields). With more data, you get more spurious correlations, more false positives, and erroneous answers. In stating so, he also liens on the excellent work of Ioannidis who indicated that over 2/3ds of scientific findings are wrong as they can't be replicated (based on extensive reviews of working papers). In other words, watch out for the many traps of multiple hypothesis testing, especially when you have not even phrased the hypothesis to begin with. "Correlation does not entail causation" still prevails.

Now in a new book (called Big Data) written by Viktor Mayer-Schonberger and Kenneth Cukier, Big Data looks far more promising. Given the size of the sample that often equates to the entire population, you can detect granular relationships between subsets of the data you could never before. And, within this Big Data era correlation seems far more important than causation. Figuring out what variables are predictive gets you far better and rich results than figuring out which ones are truly causal (that often turns into an elusive chase). The author mentions several new tools that are aimed at extracting and analyzing Big Data sets including neural networks, artificial intelligence, machine learning, sensitivity analysis among others. Being unfamiliar with any of those (and very familiar with traditional statistics and hypothesis testing in particular), I can't judge if the author statement is accurate (he is not a quant). Do those techniques truly avoid the traps of spurious correlations, multiple hypothesis testing, model overfit and false positive results?

Can you reconcile both views: Nate Silver vs Viktor Mayer?

Answer 1

This isn't the whole answer, but an important consideration is which part of your data is big.

Consider the following example. I'm doing some analysis on physical measurements of human beings. For each volunteer I measure the distance between the eyes, then length of each digit, the length of the shins, etc. And I record everything in a big table for some exploratory analysis.

If I decide to make my data bigger, I can do two things, I can make more measurements for each person (ie. more features). This is dangerous, as it increases the probability of spurious correlations.

If I decide to increase the number of instances, however, it should actually reduce the probability of spurious correlations, and although the correlations found may not imply causation, they will be more significant.

This is strongly related to the curse of dimensionality, which tells you that adding features (ie. dimensions) can cause an exponential increase in the number of instances required to reliably infer things from your data (unless your data has lower intrinsic dimension, ie. highly correlated features).

Personally, I see big data as an increase in the number of instances rather than the number of features, but this is a cause of confusion.

Answer 2

Another thing to consider is how people work with big data (as opposed to 'small' data). Big data usually requires multiple pre-processing steps before it is fed into analysis. And sometimes it is not clear what to test for exactly in these data sets to begin with. Both facts combined allow for considerable wiggle room when it comes to the final analysis. What often happens is that people run multiple analyses and then chose (or tend to chose) the one that either confirms their preconception or that returns a 'positive' result and not a hard-to-publish null-result. In other words, rather than the analysis techniques it is the humans who fall into the "traps of spurious correlations, multiple hypothesis testing, and false positive results".

Answer 3

'Big data' usually refers to data sets with gazillions of subjects, and relatively fewer measurements per subject (also called 'tall' data). For data that is wide, rather than tall, there is much work already done, a good source being Efron's recent book 'Large-Scale Inference: Empirical Bayes Methods for Estimation, Testing, and Prediction' which deals with (among other things) multiple hypothesis testing. For data that is truly tall, I haven't seen much theory, although there is tons of work relating to algorithms (see 'Mining of Massive Datasets' google it and you'll find a legally-free pdf). There is also some work on developing statistical methodology for tall data, like 'The Big Data bootstrap' by Kleiner, Talwalkar, Sarkar & Jordan.