6
$\begingroup$

  • It's been many years since I've done any statistics (or any serious math), but I do remember that the sampling error decreases more slowly for larger sample sizes (like n^-1/2, at least for some statistics).

  • I also remember (from numerical analysis) that for processes modeled as linear ODEs, errors in constant coefficients or initial conditions increase exponentially with time (and at least as bad for non-linear processes), i.e. further reduction of initial condition errors only buy us a logarithmic increase in predictive precision over time.

  • While a lot has been said about Big Data and bias errors, one thing is certain: whatever data you can already collect (along with whatever biases it may contain), you can sample it randomly without introducing additional bias errors. In short: if you can store it -- you can sample it (without bias).

In light of these (and I could be wrong, of course), it seems that any additional sample we collect has diminishing returns on statistics as well as predictions (and the benefits diminish quite rapidly). It seems, then, that even if storing and analyzing Big Data is relatively cheap, it still doesn't pay. We get lots of data that buys us little if any additional knowledge: very little extra statistic accuracy, no less bias, and virtually no added predictive ability[1]. What, then, are the benefits of Big Data?

(This question is most not a duplicate of Is sampling relevant in the time of 'big data'?, and in any event, the answers to that question don't answer mine)

[1] This last point -- predictive ability -- seems the most pertinent, as that is what many commercial uses for Big Data are for. But user behavior changes all the time -- probably with some complicated feedback, and possibly non-linearly -- so whatever extra accuracy we get, say n^-1/2, this meager gain then becomes logarithmic when predictions are concerned. In fact, it can be argued that to get better predictions, it's preferable to reduce the time it takes to calculate the statistics (by sampling), than to enhance precision by increasing the sample size, because time has an exponential effect on "knowledge", while sample size has a mere polynomial effect.

Improve this question
$\endgroup$
2
  • 10
    $\begingroup$ (i) if you fix your models, then 100x more data may not buy even quite 10x the gain (depending on what you're looking at and what you're using it for) ... but in many cases, it vastly widens the kinds of models you can consider, and the extent to which you can let the data drive your models (if you properly deal with that); (ii) big data is not just large-n, but also large-p; (iii) in any case, sometimes, a standard error that's only say one fourth as large really matters... In some cases, companies gladly pay millions for even a 10% improvement in what they get out of their present models. $\endgroup$
    – Glen_b
    Oct 26, 2014 at 23:46
  • 1
    $\begingroup$ With your question, it comes to mind an example of famous incorrect prediction by The Literary Digest of 1936 US Presidential election results due to misuse of big data sampling. The story of this and the contrasting story of correct prediction by George Gallup's opinion poll, were some of the illustrating points of economist and writer Tim Harford in his recent 2014 Significance Lecture talk on big data trap: statslife.org.uk/science-technology/…. $\endgroup$
    – Aleksandr Blekh
    Jan 4, 2015 at 12:49

1 Answer 1

Reset to default
6
$\begingroup$

it isn't only the data that is big, also the problem is big.

Indeed, the benefits of increasing the sampling size aren't great if you were computing the mean of terabytes of data. Only that nobody is interested in the 10th digit of the mean anyways...

More often than not, big data problems are more like a big amount of problems to be solved at once. You have millions of users, thousands of products. The sample sizes for each of them aren't big data, but you have a lot of them... similarly, in image recognition, you have lots of pixels, lots of labels (imagenet has some 20000 categories or so), so more often than not you don't even have a single training example that is really similar...

When searching a large space of hypotheses, you also need to adjust for multiple testing problems. Say you are testing for problems with an $\alpha=0.999$ certainty. But you are testing just 100 hypotheses, then you end up with a certainty of only $\bar\alpha=0.90$ that the result is really correct. And this confidence drops quickly - at 1000 tests, there is a 2 in 3 chance of having a false positive. A (at least theoretical) way out is to use a larger $\alpha$ such as $\alpha=0.99999$ in the individual tests. But then you may need to get a larger sample to be able to reach such a confidence ever...

Improve this answer
$\endgroup$
3
  • 2
    $\begingroup$ Point (ii) in the comment by @Glen_b is particularly apt--don't let the OP's focus on sampling errors mislead you. BTW, terabytes of data won't yield 20 significant digits additional precision in a mean, but only around 6. For large-$p$ problems with nearly-collinear predictors, you might not even get one significant digit of precision out of terabytes of data. $\endgroup$
    – whuber ♦
    Jan 4, 2015 at 5:39
  • $\begingroup$ With large-$p$ you mean extreme $p$-values, to adjust for multiple testing? $p>1-10^{-8}$? $\endgroup$
    – Has QUIT--Anony-Mousse
    Jan 4, 2015 at 12:11
  • 2
    $\begingroup$ @anony-mousse "large p" means "large numbers of parameters". This was part of Glen_b's point about one benefit of Big Data being the ability to consider more complex models. (whuber's point is that in complex models with many predictors, it can still be difficult to distinguish the effect of highly correlated predictors even when sample sizes are huge.) $\endgroup$
    – Silverfish
    Jan 4, 2015 at 13:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.