26
$\begingroup$

When performing Bayesian inference, we operate by maximizing our likelihood function in combination with the priors we have about the parameters. Because the log-likelihood is more convenient, we effectively maximize $\sum \ln (\text{prior}) + \sum \ln (\text{likelihood})$ using an MCMC or otherwise which generates the posterior distributions (using a pdf for each parameter's prior and each data point's likelihood).

If we have a lot of data, the likelihood from that is going to overwhelm any information that the prior provides, by simple mathematics. Ultimately, this is good and by design; we know that the posterior will converge to just the likelihood with more data because it is supposed to.

For problems defined by conjugate priors, this is even provable exactly.

Is there a way to decide when priors don't matter for a given likelihood function and some sample size?

$\endgroup$
  • 3
    $\begingroup$ Your first sentence is not right. Bayesian inference, and the MCMC algorithm, do not maximize the likelihood. $\endgroup$ – niandra82 Mar 10 '16 at 16:44
  • 5
    $\begingroup$ Are you familiar with marginal likelihood, Bayes factors, prior/posterior predictive distribution, prior/posterior predictive check? these are the type of things you would use to compare models in a Bayesian Paradigm. I think this question boils down to whether or not the Bayes factor, between models that only differ by their prior, will converge to 1 as sample size goes to infinity. You may also want to put aside priors that are truncated within the parameter space implied by the likelihood, as this could potentially deny the target from converging to the maximum likelihood estimate. $\endgroup$ – Zachary Blumenfeld Mar 10 '16 at 18:02
  • $\begingroup$ @ZacharyBlumenfeld: this could qualify as a proper answer! $\endgroup$ – Xi'an Mar 10 '16 at 18:05
  • $\begingroup$ Is the corrected form "maximizing Bayes' rule"? Also, the models I am working with are physically based, so truncated parameter spaces are a necessity for the work. (I also agree that your comments are probably an answer, could you flesh them out @ZacharyBlumenfeld?) $\endgroup$ – pixels Mar 10 '16 at 19:31
37
$\begingroup$

It is not that easy. Information in your data overwhelms prior information not only your sample size is large, but when your data provides enough information to overwhelm the prior information. Uninformative priors get easily persuaded by data, while strongly informative ones may be more resistant. In extreme case, with ill-defined priors, your data may not be able at all to overcome it (e.g. zero density over some region).

Recall that by Bayes theorem we use two sources of information in our statistical model, out-of-data, prior information, and information conveyed by data in likelihood function:

$$ \color{violet}{\text{posterior}} \propto \color{red}{\text{prior}} \times \color{lightblue}{\text{likelihood}} $$

When using uninformative prior (or maximum likelihood), we try to bring minimal possible prior information into our model. With informative priors we bring substantial amount of information into the model. So both, the data and prior, inform us what values of estimated parameters are more plausible, or believable. They can bring different information and each of them can overpower the other one in some cases.

Let me illustrate this with very basic beta-binomial model (see here for detailed example). With "uninformative" prior, pretty small sample may be enough to overpower it. On the plots below you can see priors (red curve), likelihood (blue curve), and posteriors (violet curve) of the same model with different sample sizes.

enter image description here

On another hand, you can have informative prior that is close to the true value, that would also be easily, but not that easily as with weekly informative one, persuaded by data.

enter image description here

The case is very different with informative prior, when it is far from what the data says (using the same data as in first example). In such case you need larger sample to overcome the prior.

enter image description here

So it is not only about sample size, but also about what is your data and what is your prior. Notice that this is a desired behavior, because when using informative priors we want to potentially include out-of-data information in our model and this would be impossible if large samples would always discard the priors.

Because of complicated posterior-likelihood-prior relations, it is always good to look at the posterior distribution and do some posterior predictive checks (Gelman, Meng and Stern, 1996; Gelman and Hill, 2006; Gelman et al, 2004). Moreover, as described by Spiegelhalter (2004), you can use different priors, for example "pessimistic" that express doubts about large effects, or "enthusiastic" that are optimistic about estimated effects. Comparing how different priors behave with your data may help to informally assess the extent how posterior was influenced by prior.


Spiegelhalter, D. J. (2004). Incorporating Bayesian ideas into health-care evaluation. Statistical Science, 156-174.

Gelman, A., Carlin, J. B., Stern, H. S., and Rubin, D. B. (2004). Bayesian data analysis. Chapman & Hall/CRC.

Gelman, A. and Hill, J. (2006). Data analysis using regression and multilevel/hierarchical models. Cambridge University Press.

Gelman, A., Meng, X. L., and Stern, H. (1996). Posterior predictive assessment of model fitness via realized discrepancies. Statistica sinica, 733-760.

$\endgroup$
  • 2
    $\begingroup$ Nice contribution, thank you Tim. I'd like to add that the contrast you lay out so nicely here may present itself even within one and the same model as related to different parameters of that model. There may be some parameters about which the data offer negligible information, in which case priors may serve critically to provide identifying restrictions. $\endgroup$ – David C. Norris Mar 15 '16 at 19:26
  • $\begingroup$ In the first 3x3 matrix of graphs, are the graphs correct? The posterior is completely flat up to and including n=25? $\endgroup$ – MichiganWater Aug 21 at 2:50
  • 1
    $\begingroup$ @MichiganWater each 9-plot collection uses same scale for y-axis so that the largest values don't go out of the screen. So they are flat relatively to the case where you have more data. If you "zoomed in", they won't be flat. $\endgroup$ – Tim Aug 21 at 4:44
11
$\begingroup$

When performing Bayesian inference, we operate by maximizing our likelihood function in combination with the priors we have about the parameters.

This is actually not what most practitioners consider to be Bayesian inference. It is possible to estimate parameters this way, but I would not call it Bayesian inference.

Bayesian inference uses posterior distributions to calculate posterior probabilities (or ratios of probabilities) for competing hypotheses.

Posterior distributions can be estimated empirically by Monte Carlo or Markov-Chain Monte Carlo (MCMC) techniques.

Putting these distinctions aside, the question

Do Bayesian priors become irrelevant with large sample size?

still depends on the context of the problem and what you care about.

If what you care about is prediction given an already very large sample, then the answer is generally yes, the priors are asymptotically irrelevant*. However, if what you care about is model selection and Bayesian Hypothesis testing, then the answer is no, the priors matter a lot, and their effect will not deteriorate with sample size.

*Here, I am assuming that the priors aren't truncated/censored beyond the parameter space implied by the likelihood, and that they aren't so ill-specified as to cause convergence issues with near zero-density in important regions. My argument is also asymptotic, which comes with all the regular caveats.

Predictive Densities

As an example, let $\mathbf{d}_N = (d_1, d_2,...,d_N)$ be your data, where each $d_i$ signifies an observation. Let the likelihood be denoted as $f(\mathbf{d}_N\mid \theta)$, where $\theta$ is the parameter vector.

Then suppose we also specify two separate priors $\pi_0 (\theta \mid \lambda_1)$ and $\pi_0 (\theta \mid \lambda_2)$, which differ by the hyper-parameter $\lambda_1 \neq \lambda_2$.

Each prior will lead to different posterior distributions in a finite sample, $$ \pi_N (\theta \mid \mathbf{d}_N, \lambda_j) \propto f(\mathbf{d}_N\mid \theta)\pi_0 ( \theta \mid \lambda_j)\;\;\;\;\;\mathrm{for}\;\;j=1,2 $$

Letting $\theta^*$ be the suito true parameter value, $\theta^{j}_N \sim \pi_N(\theta\mid \mathbf{d}_N, \lambda_j)$, and $\hat \theta_N = \max_\theta\{ f(\mathbf{d}_N\mid \theta) \}$, it is true that $\theta^{1}_N$, $\theta^{2}_N$, and $\hat \theta_N$ will all converge in probability to $\theta^*$. Put more formally, for any $\varepsilon >0$;

$$ \begin{align} \lim_{N \rightarrow \infty} Pr(|\theta^j_N - \theta^*| \ge \varepsilon) &= 0\;\;\;\forall j \in \{1,2\} \\ \lim_{N \rightarrow \infty} Pr(|\hat \theta_N - \theta^*| \ge \varepsilon) &= 0 \end{align} $$

To be more consistent with your optimization procedure, we could alternatively define $\theta^j_N = \max_\theta \{\pi_N (\theta \mid \mathbf{d}_N, \lambda_j)\} $ and although this parameter is very different then the previously defined, the above asymptotics still hold.

It follows that the predictive densities, which are defined as either $f(\tilde d \mid \mathbf{d}_N, \lambda_j) = \int_{\Theta} f(\tilde d \mid \theta,\lambda_j,\mathbf{d}_N)\pi_N (\theta \mid \lambda_j,\mathbf{d}_N)d\theta$ in a proper Bayesian approach or $f(\tilde d \mid \mathbf{d}_N, \theta^j_N)$ using optimization, converge in distribution to $f(\tilde d\mid \mathbf{d}_N, \theta^*)$. So in terms of predicting new observations conditional on an already very large sample, the prior specification makes no difference asymptotically.

Model Selection and Hypothesis Testing

If one is interested in Bayesian model selection and hypothesis testing they should be aware that the effect of the prior does not vanish asymptotically.

In a Bayesian setting we would calculate posterior probabilities or Bayes factors with marginal likelihoods. A marginal likelihood is the likelihood of the data given a model i.e. $f(\mathbf{d}_N \mid \mathrm{model})$.

The Bayes factor between two alternative models is the ratio of their marginal likelihoods; $$ K_N = \frac{f(\mathbf{d}_N \mid \mathrm{model}_1)}{f(\mathbf{d}_N \mid \mathrm{model}_2)} $$ The posterior probability for each model in a set of models can also be calculated from their marginal likelihoods as well; $$ Pr(\mathrm{model}_j \mid \mathbf{d}_N) = \frac{f(\mathbf{d}_N \mid \mathrm{model}_j)Pr(\mathrm{model}_j)}{\sum_{l=1}^L f(\mathbf{d}_N \mid \mathrm{model}_l)Pr(\mathrm{model}_l)} $$ These are useful metrics used to compare models.

For the above models, the marginal likelihoods are calculated as; $$ f(\mathbf{d}_N \mid \lambda_j) = \int_{\Theta} f(\mathbf{d}_N \mid \theta, \lambda_j)\pi_0(\theta\mid \lambda_j)d\theta $$

However, we can also think about sequentially adding observations to our sample, and write the marginal likelihood as a chain of predictive likelihoods; $$ f(\mathbf{d}_N \mid \lambda_j) = \prod_{n=0}^{N-1} f(d_{n+1} \mid \mathbf{d}_n , \lambda_j) $$ From above we know that $f(d_{N+1} \mid \mathbf{d}_N , \lambda_j)$ converges to $f(d_{N+1} \mid \mathbf{d}_N , \theta^*)$, but it is generally not true that $f(\mathbf{d}_N \mid \lambda_1)$ converges to $f(\mathbf{d}_N \mid \theta^*)$, nor does it converge to $f(\mathbf{d}_N \mid \lambda_2)$. This should be apparent given the product notation above. While latter terms in the product will be increasingly similar, the initial terms will be different, because of this, the Bayes factor $$ \frac{f(\mathbf{d}_N \mid \lambda_1)}{ f(\mathbf{d}_N \mid \lambda_2)} \not\stackrel{p}{\rightarrow} 1 $$ This is an issue if we wished to calculate a Bayes factor for an alternative model with different likelihood and prior. For example consider the marginal likelihood $h(\mathbf{d}_N\mid M) = \int_{\Theta} h(\mathbf{d}_N\mid \theta, M)\pi_0(\theta\mid M) d\theta$; then $$ \frac{f(\mathbf{d}_N \mid \lambda_1)}{ h(\mathbf{d}_N\mid M)} \neq \frac{f(\mathbf{d}_N \mid \lambda_2)}{ h(\mathbf{d}_N\mid M)} $$ asymptotically or otherwise. The same can be shown for posterior probabilities. In this setting the choice of the prior significantly effects the results of inference regardless of sample size.

$\endgroup$
5
$\begingroup$

Another issue to keep in mind is you can have a lot of data, but still have very little information about certain parameters in your model. In such cases, even a mildly informative prior can be extremely helpful when performing inference.

As a silly example, suppose you were comparing means of two groups and you had 1,000,000 samples of group 1 and 10 samples of group 2. Then clearly having an informative prior about group 2 can improve inference, even though you've collected over a million samples.

And while that example may be trivial, it starts to lead some very important implications. If we want to understand some complex phenomena, the smart thing to do is collect a lot of information regarding the parts we don't understand and less information about the parts we do understand. If we collect a lot of data in such a manner, throwing out the prior because we have a lot of data is a really bad choice; we've just set back our analysis because we didn't waste time collecting data on things we already know!

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.