30
$\begingroup$

From "Why should I be Bayesian when my model is wrong?", one of the key benefits of Bayesian inference to be able to inject exogenous domain knowledge into the model, in the form of a prior. This is especially useful when you don't have enough observed data to make good predictions.

However, the prior's influence diminishes (to zero?) as the dataset grows larger. So if you have enough data, the prior provides very little value.

What's the benefit of using Bayesian analysis in this case?

Maybe that we still get a posterior distribution over parameter values? (But for large enough data, wouldn't the posterior just collapse to the MLE?)

$\endgroup$
8
  • $\begingroup$ What is the goal of your analysis? Inference? Accurate predictions? Something else? $\endgroup$
    – Adrian
    Oct 6, 2020 at 20:29
  • 1
    $\begingroup$ "However, the prior's influence diminishes (to zero?) as the dataset grows larger." -- This is not true in general. Every new data point is not necessarily "more useful." It can keep telling you about what you already know, and not about what you don't. For example, imagine that you're trying to estimate your position by measuring your bearing (angle) to a bunch of known landmarks you can see in the distance. Your prior is a Gaussian centered where you think you are. If all of the landmarks are roughly co-linear, your posterior will never reduce uncertainty / spread along that axis. $\endgroup$
    – jnez71
    Oct 8, 2020 at 16:32
  • 1
    $\begingroup$ Check out the last gif in this answer for a visualization of that Bayesian behavior. One cool thing about Bayesian reasoning is pretty much that is doesn't (necessarily) behave the way your question suggests. The remaining uncertainty in one's posterior can make clear what your data can't seem to tell you, no matter how much you get. This can be a specific combination of the variables, or some equi-dimensional irreducible variance. It tends to be very relevant for timeseries models, check out "observability analysis." $\endgroup$
    – jnez71
    Oct 8, 2020 at 16:41
  • $\begingroup$ Your dataset is never large ;-) $\endgroup$
    – kutschkem
    Oct 9, 2020 at 8:12
  • $\begingroup$ @kutschkem in the demo from that other rather unrelated question, yes the data set is small; it was just for visualization purposes. If the landmarks are actually all in a straight line though, the uncertainty will never reduce. And perhaps even if the deviations from a straight line are much smaller than the error variance of the angle measurements. $\endgroup$
    – jnez71
    Oct 9, 2020 at 17:57

4 Answers 4

40
$\begingroup$
  1. Being Bayesian is not only about information fed through the prior. But even then: Where the prior is zero, no amount of data will turn that over.

  2. Having a full Bayesian posterior distribution to draw from opens loads and loads of ways to make inference from.

  3. It is easy to explain a credible interval to any audience whilst you know that most audiences have a very vague understanding of what a confidence interval is.

  4. Andrew Gelman said in one of his youtube videos, that $p$ is always slightly lower then $0.05$ because if it wasn't smaller then we would not read about it and if it was much smaller they'd examine subgroups. While that is not an absolute truth, indeed when you have large data you will be tempted to investigate defined subgroups ("is it still true when we only investigate caucasian single women under 30?") and that tends to shrink even large data quite a lot.

  5. $p$-values tend to get worthless with large data as in real life no null hypthesis holds true in large data sets. It is part of the tradition about $p$ values that we keep the acceptable alpha error at $.05$ even in huge datasets where there is absolutely no need for such a large margin of error. Baysian analysis is not limited to point hyptheses and can find that the data is in a region of practical equivalence to a null hypotheses, a Baysian factor can grow your believe in some sort of null hypothesis equivalent where a $p$ value can only accumulate evidence against it. Could you find ways to emulate that via confidence intervals and other Frequentist methods? Probably yes, but Bayes comes with that approach as the standard.

  6. "But for large enough data, wouldn't the posterior just collapse to the MLE" - what if a posterior was bimodal or if two predictors are correlated so you could have different combinations of e.g. $\beta_8$ and $\beta_9$ - a posterior can represent these different combinations, an MLE point estimator does not.

$\endgroup$
4
  • 3
    $\begingroup$ I'm not sure about point 5: Composite null hypothesis are frequently true, I reckon. The point that $p$-values are worthless in large samples is expected: It's just the property of consistency of the test (its power approaches one to detect false nulls). It doesn't make sense to me to say a tests is okay in small samples but worthless in large samples. $\endgroup$ Oct 6, 2020 at 7:57
  • 2
    $\begingroup$ Fair point taken. However, Frequentism is not only what you could do with it but also its tradition, what people usually do. Personally I was taught never to publish a one-tailed test (the obvious case of a composite null hypothesis) and I was taught to always setalpha at $.05$ or $.01$ or one of those multiplied by some Bonferroni correction. Neither rejection of one-tailed testing nor fixing alpha at some constant level for all research is inherent to Frequentism in theory, however in large parts of practice and teaching. $\endgroup$
    – Bernhard
    Oct 6, 2020 at 8:25
  • $\begingroup$ Oh yeah, absolutely. I think you're raising some excellent points in your answer (+1). $\endgroup$ Oct 6, 2020 at 8:32
  • 2
    $\begingroup$ +1. I think point 1 could use expansion. It's a two-edged sword: where the prior is literally zero you're saying it's impossible and if you mean "highly-unlikely" you are in error. However, if you literally mean "impossible" (negative variance, negative weight, etc), then a Bayesian prior can enforce this strictly where parametric approximations/estimates of point values can't. $\endgroup$
    – Wayne
    Oct 6, 2020 at 20:01
7
$\begingroup$

I'd like to echo some of the points in the other answer with slightly different emphasis.

To me the most important issue is that the Bayesian view of uncertainty/probability/randomness is the one that directly answers the questions we probably care about, whereas the Frequentist view of uncertainty directly answers other questions that are often somewhat besides the point. Bayesian inferences try to tell us what we (or an algorithm, machine, etc.) should believe given the data we have seen, or in other words "what can I learn about the world from this data?" Frequentist inferences try to tell us how different our results would be if the data that we actually saw were "re-generated" or "repeatedly sampled" an infinite number of times. Personally I sometimes find Frequentist questions interesting, but I can't think of a scenario where the Bayesian questions aren't what matter most (since at the end of the day I want to make a decision about what to believe or do now that I've seen new data). It's worth noting that often people (statisticians included) incorrectly interpret Frequentist analyses as answering Bayesian questions, probably betraying their actual interests. And while people get worried about the subjectivity inherent in Bayesian methods, I think of the Tukey line, "Far better an approximate answer to the right question, which is often vague, than an exact answer to the wrong question, which can always be made precise." For what it's worth, Frequentist methods are also subjective, and arguably in ways that are less obvious and convenient to critique.

Getting off my Bayesian high horse, you're right that answers to Frequentist questions (especially MLE) sometimes coincide closely (and in rare cases, exactly) with answers to Bayesian questions.

However, large data is a vague notion in a few senses that can make Bayesian and Frequentist (MLE) answers remain different:

  1. Most results about large data are asymptotic as the sample size goes to infinity, meaning that they don't tell us when our sample size is actually large enough for the asymptotic result to be accurate enough (up to some known level of error). If you go through the trouble to do both Bayesian and Frequentist analyses of your data and find they're numerically similar then it doesn't matter so much.
  2. Often with "large" data (e.g. many observations) we also have a large number of questions or parameters of interest. This is basically Bernhard's point #4.
  3. A lot of large data sets are not perfectly designed and relate to our interests indirectly because of issues like measurement error or sampling bias. Treated honestly, these complications may not go away even asymptotically, meaning that the models that realistically relate the data to what we care about have non-identifiable sensitivity parameters that are most natural to deal with using priors and the Bayesian machinery.

Of course, the flip-side of this question is "Why should I be Frequentist when my dataset is large?"

$\endgroup$
3
$\begingroup$

The other answers address what's probably your actual question. But just to add a more concrete viewpoint: if you're already a Bayesian (for small/medium datasets) and you get a large data, why not use the methodology you're familiar with? It will be relatively slow but you are familiar with the steps so you're less likely to make mistakes and you're more likely to spot problems. And a Bayesian workflow includes things like posterior predictive checks, etc, which are useful for understanding your model.

$\endgroup$
3
$\begingroup$

One place where Bayesian approach meets large datasets is Bayesian deep learning. When using Bayesian approach to neural networks people usually use rather simplistic priors (Gaussians, centered at zero), this is mostly for computational reasons, but also because there is not much prior knowledge (neural network parameters are black-boxish). The reason why Bayesian approach is used, is because out-of-the-box it gives us uncertainty estimates.

$\endgroup$
8
  • $\begingroup$ Is there some intuition of why the uncertainty estimates of BNNs don't collapse to certainty for large datasets? (Maybe they do?) $\endgroup$
    – kennysong
    Oct 6, 2020 at 23:12
  • $\begingroup$ @kennysong BNNs do collapse to certainty for small datasets; I think larger datasets make that type of collapse less probable rather than necessarily prevent it. $\endgroup$ Oct 7, 2020 at 1:23
  • $\begingroup$ @JeremyList Why is that the case? It seems counterintuitive to me $\endgroup$
    – kennysong
    Oct 7, 2020 at 3:41
  • 1
    $\begingroup$ @kennysong if your data is small, then the model can memorize the data (overfitt) and give exact predictions per each datapoint. In such case it would "feel" certain about it's predictions. If the data is big, than it cannot be certain, because same parameters need to be re-used for different data, so there is a variability. $\endgroup$
    – Tim
    Oct 7, 2020 at 8:51
  • $\begingroup$ @Tim Machine learning is not restricted to neural networks. There are several methods which do not require using a bayesian framework. What makes me rethink of using bayes in ML and when large data is the variance-bias trade-off. Bayesian methods tend to reduce variance but not bias, whereas data driven methods usually reduce bias but have larger scatter. Big data makes it possible to reduce the scatter in the latter, not sure the bias in the former. $\endgroup$
    – jpcgandre
    Nov 20, 2020 at 12:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.