What is the relationship between sample size and the influence of prior on posterior?

Question

If we have a small sample size, will the prior distribution influence the posterior distribution a lot?

Answer 1

Yes. The posterior distribution for a parameter $\theta$, given a data set ${\bf X}$ can be written as

$$ p(\theta | {\bf X}) \propto \underbrace{p({\bf X} | \theta)}_{{\rm likelihood}} \cdot \underbrace{p(\theta)}_{{\rm prior}} $$

or, as is more commonly displayed on the log scale,

$$ \log( p(\theta | {\bf X}) ) = c + L(\theta;{\bf X}) + \log(p(\theta)) $$

The log-likelihood, $L(\theta;{\bf X}) = \log \left( p({\bf X}|\theta) \right)$, scales with the sample size, since it is a function of the data, while the prior density does not. Therefore, as the sample size increases, the absolute value of $L(\theta;{\bf X})$ is getting larger while $\log(p(\theta))$ stays fixed (for a fixed value of $\theta$), thus the sum $L(\theta;{\bf X}) + \log(p(\theta))$ becomes more heavily influenced by $L(\theta;{\bf X})$ as the sample size increases.

Therefore, to directly answer your question - the prior distribution becomes less and less relevant as it becomes outweighed by the likelihood. So, for a small sample size, the prior distribution plays a much larger role. This agrees with intuition since, you'd expect that prior specifications would play a larger role when there isn't much data available to disprove them whereas, if the sample size is very large, the signal present in the data will outweigh whatever a priori beliefs were put into the model.

Answer 2

Here is an attempt to illustrate the last paragraph in Macro's excellent (+1) answer. It shows two priors for the parameter $p$ in the ${\rm Binomial}(n,p)$ distribution. For a few different $n$, the posterior distributions are shown when $x=n/2$ has been observed. As $n$ grows, both posteriors become more and more concentrated around $1/2$.

For $n=2$ the difference is quite big, but for $n=50$ there is virtually no difference.

The two priors below are ${\rm Beta(1/2,1/2)}$ (black) and ${\rm Beta(2,2)}$ (red). The posteriors have the same colours as the priors that they are derived from.

(Note that for many other models and other priors, $n=50$ won't be enough for the prior not to matter!)