41
$\begingroup$

"Big data" is everywhere in the media. Everybody says that "big data" is the big thing for 2012, e.g. KDNuggets poll on hot topics for 2012. However, I have deep concerns here. With big data, everybody seems to be happy just to get anything out. But aren't we violating all classic statistical principles such as hypothesis testing and representative sampling?

As long as we make only predictions about the same data set, this should be fine. So if I use Twitter data to predict Twitter user behavior, that is probably okay. However, using Twitter data to predict e.g. Elections completely neglects the fact that the Twitter users are not a representative sample for the whole population. Plus, most methods will actually not be able to differentiate between a true "grassroots" mood and a campaign. And twitter is full of campaigns. So when analyzing Twitter, you quickly end up just measuring campaigning and bots. (See for example "Yahoo Predicts America's Political Winners" which is full of poll bashing and "sentiment analysis is much better". They predicted "Romney has over a 90 percent likelihood of winning the nomination, and of winning the South Carolina primary" (he had 28%, while Gingrich had 40% in this primary).

Do you know other such big data fails? I remember roughly that one scientist predicted you could not maintain more than 150 friendships. He actually had only discovered a cap limit in friendster ...

As for twitter data, or actually any "big data" collected from the web, I believe that often people even introduce additional bias by the way they collect their data. Few will have all of Twitter. They will have a certain subset they spidered, and this is just yet another bias in their data set.

Splitting the data into a test set or for doing cross validation likely doesn't help much. The other set will have the same bias. And for big data, I need to "compress" my information so heavily that I'm rather unlikely to overfit.

I recently heard this joke, with the big data scientist that discovered there are approximately 6 sexes in the world... and I can this just so imagine to happen... "Male, Female, Orc, Furry, Yes and No".

So what methods do we have to get some statistical validity back into the analysis, in particular when trying to predict something outside of the "big data" dataset?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
32
$\begingroup$

Your fears are well founded and perceptive. Yahoo and probably several other companies are doing randomized experiments on users and doing it well. But observational data are frought with difficulties. It is a common misperception that problems diminish as the sample size increases. This is true for variance, but bias stays constant as n increases. When the bias is large, a very small truly random sample or randomized study can be more valuable than 100,000,000 observations.

Improve this answer
$\endgroup$
1
  • 8
    $\begingroup$ Big data is probably one area where bias variance decomposition is not helpful - data quality and data management are more important. This is because we cannot hope to know every data point or even special cases - just too many of them $\endgroup$
    – probabilityislogic
    Commented Feb 12, 2012 at 13:39
25
$\begingroup$

There are a number of techniques in experimental design and analysis that can help you reduce your bias, but this again always boils down to the same thing: One has to know what one is doing. Big data analysis has the same problem as any other data analysis; it suffers from a lack of hypotheses.

A clear example is multiple regression with stepwise variable selection. Very nice, one say, but with 100 variables measured statistical laws dictate that some of them will show a significant relation when evaluated by looking whether the respective coefficient differs significantly from zero. So the more variables in your dataset, the more chance of finding two that show some (meaningless) relation. And the bigger your dataset, the more chance for meaningless models due to eg a small confounding effect. If you test many models (and with even only 10 variables that can be a whole lot of models), you're very likely to find at least one significant. Does it mean something? No.

What should one do then? Use your brain:

  • formulate a hypothesis before collecting the data and test that hypothesis. That's the only way to make sure your statistics actually tell a story.
  • Use your covariates to stratify your sampling before doing some tests. Stupid example: If you have 1000 males and 100 females in your dataset, randomly select 50 each if you want to talk about an average population. That's actually something where big data comes in handy: You have more than enough to sample from.
  • Describe the test population thoroughly, so it's clear for which population your conclusions are formulated.
  • If you use your big dataset for explorative purposes, test the hypotheses you come up with during this exploration on a new and different dataset, not just a subset of what you collected. And test them again using all the necessary precautions.

These things are all obvious and well-known. Heck, already in 1984 Rosenbaum and Rubin illustrated how to use propensity scores to reduce bias in observational studies, and that's what most big datasets are: observational data. In more recent work of Feng et al, the use of the Mahalanobis distance is also advocated. And in fact, one of my statistical heroes, Cochran, wrote a review about that problem already in 1973! Or what about Rubin, who introduced multivariate matched sampling and regression correcting already in 1979. Old publications are seriously underestimated and far too often ignored, certainly in a field like statistics.

All these techniques have pros and cons, and one has to understand that reducing bias is not the same as eliminating bias. But if you are aware of :

  • what you want to test, and
  • how you are doing it

Big data is not an excuse to come with bogus results.

Edited after the (correc) remark of @D.W. who pointed out I used the term 'overfitting' in a wrong context.

Improve this answer
$\endgroup$
5
  • 12
    $\begingroup$ "the bigger your dataset, the more chance for meaningless overfitting" - Actually, that's backwards. The bigger the set of possible models, the greater the chance of overfitting (all else being equal). The larger the dataset, the smaller the chance of overfitting (all else being equal). $\endgroup$
    – D.W.
    Commented Feb 11, 2012 at 17:36
  • $\begingroup$ @D.W. How that so? In fact, if there's absolute independence in a simulation, there's as much chance on a significant model with small and big datasets (simple simulation shows you that). Alas, I have yet to meet a dataset where the independence is perfect. The moment you have eg a very small confounding effect, big datasets are more likely to give meaningless significant results than small datasets. $\endgroup$
    – Joris Meys
    Commented Feb 12, 2012 at 13:21
  • $\begingroup$ Nice answer - your comment about finding significant effects provides a good rationale for shrinkage methods over "in-or-out" methods of model selection. $\endgroup$
    – probabilityislogic
    Commented Feb 12, 2012 at 13:52
  • 10
    $\begingroup$ @D.W. is making a statement about overfitting, and seems correct -- particularly since the larger the data set, the more chance for humbling cross-validation on subsets of the data. Joris Meys is making a statement about statistical significance. That's also correct. But in large data sets statistical significance is moot -- it's effect size that matters because nearly everything is "statistically significant". $\endgroup$
    – zbicyclist
    Commented Feb 15, 2012 at 1:29
  • 1
    $\begingroup$ @zbicyclist Very correct observation. I admit I misinterpreted D.W. and used the term overfitting in a wrong context. I stand corrected. $\endgroup$
    – Joris Meys
    Commented Feb 15, 2012 at 9:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.