Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

If we have a small sample size, will the prior distribution influence the posterior distribution a lot?

share|improve this question
3  
The intuition is clear: the more data you have, the less you have to rely on your priors. Not just a statistics lesson, but a life lesson! ;) –  Lucas Reis Jun 13 '12 at 18:21
add comment

2 Answers 2

Yes. The posterior distribution for a parameter $\theta$, given a data set ${\bf X}$ can be written as

$$ p(\theta | {\bf X}) \propto \underbrace{p({\bf X} | \theta)}_{{\rm likelihood}} \cdot \underbrace{p(\theta)}_{{\rm prior}} $$

or, as is more commonly displayed on the log scale,

$$ \log( p(\theta | {\bf X}) ) = c + L(\theta;{\bf X}) + \log(p(\theta)) $$

The log-likelihood, $L(\theta;{\bf X}) = \log \left( p({\bf X}|\theta) \right)$, scales with the sample size, since it is a function of the data, while the prior density does not. Therefore, as the sample size increases, the absolute value of $L(\theta;{\bf X})$ is getting larger while $\log(p(\theta))$ stays fixed (for a fixed value of $\theta$), thus the sum $L(\theta;{\bf X}) + \log(p(\theta))$ becomes more heavily influenced by $L(\theta;{\bf X})$ as the sample size increases.

Therefore, to directly answer your question - the prior distribution becomes less and less relevant as it becomes outweighed by the likelihood. So, for a small sample size, the prior distribution plays a much larger role. This agrees with intuition since, you'd expect that prior specifications would play a larger role when there isn't much data available to disprove them whereas, if the sample size is very large, the signal present in the data will outweigh whatever a priori beliefs were put into the model.

share|improve this answer
4  
+1 Note that $c$ also depends on $n$. –  user10525 Jun 13 '12 at 16:38
add comment

Here is an attempt to illustrate the last paragraph in Macro's excellent (+1) answer. It shows two priors for the parameter $p$ in the ${\rm Binomial}(n,p)$ distribution. For a few different $n$, the posterior distributions are shown when $x=n/2$ has been observed. As $n$ grows, both posteriors become more and more concentrated around $1/2$.

For $n=2$ the difference is quite big, but for $n=50$ there is virtually no difference.

The two priors below are ${\rm Beta(1/2,1/2)}$ (black) and ${\rm Beta(2,2)}$ (red). The posteriors have the same colours as the priors that they are derived from.

Posterior distributions

(Note that for many other models and other priors, $n=50$ won't be enough for the prior not to matter!)

share|improve this answer
4  
Very cool illustrations, @MånsT. I de-italicized the words 'Beta' and 'Binomial' in your answer - I hope you don't mind. –  Macro Jun 13 '12 at 19:32
    
Of course not, @Macro! I agree that it looks better this way. –  MånsT Jun 13 '12 at 19:34
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.