17
$\begingroup$

How do we go about calculating a posterior with a prior N~(a, b) after observing n data points? I assume that we have to calculate the sample mean and variance of the data points and do some sort of calculation that combines the posterior with the prior, but I'm not quite sure what the combination formula looks like.

$\endgroup$
22
$\begingroup$

The basic idea of Bayesian updating is that given some data $X$ and prior over parameter of interest $\theta$, where the relation between data and parameter is described using likelihood function, you use Bayes theorem to obtain posterior

$$ p(\theta \mid X) \propto p(X \mid \theta) \, p(\theta) $$

This can be done sequentially, where after seeing first data point $x_1$ prior $\theta$ becomes updated to posterior $\theta'$, next you can take second data point $x_2$ and use posterior obtained before $\theta'$ as your prior, to update it once again etc.

Let me give you an example. Imagine that you want to estimate mean $\mu$ of normal distribution and $\sigma^2$ is known to you. In such case we can use normal-normal model. We assume normal prior for $\mu$ with hyperparameters $\mu_0,\sigma_0^2:$

\begin{align} X\mid\mu &\sim \mathrm{Normal}(\mu,\ \sigma^2) \\ \mu &\sim \mathrm{Normal}(\mu_0,\ \sigma_0^2) \end{align}

Since normal distribution is a conjugate prior for $\mu$ of normal distribution, we have closed-form solution to update the prior

\begin{align} E(\mu' \mid x) &= \frac{\sigma^2\mu + \sigma^2_0 x}{\sigma^2 + \sigma^2_0} \\[7pt] \mathrm{Var}(\mu' \mid x) &= \frac{\sigma^2 \sigma^2_0}{\sigma^2 + \sigma^2_0} \end{align}

Unfortunately, such simple closed-form solutions are not available for more sophisticated problems and you have to rely on optimization algorithms (for point estimates using maximum a posteriori approach), or MCMC simulation.

Below you can see data example:

n <- 1000
set.seed(123)
x     <- rnorm(n, 1.4, 2.7)
mu    <- numeric(n)
sigma <- numeric(n)

mu[1]    <- (10000*x[i] + (2.7^2)*0)/(10000+2.7^2)
sigma[1] <- (10000*2.7^2)/(10000+2.7^2)
for (i in 2:n) {
  mu[i]    <- ( sigma[i-1]*x[i] + (2.7^2)*mu[i-1] )/(sigma[i-1]+2.7^2)
  sigma[i] <- ( sigma[i-1]*2.7^2                  )/(sigma[i-1]+2.7^2)
}

If you plot the results, you'll see how posterior approaches the estimated value (it's true value is marked by red line) as new data is accumulated.

Updating prior in subsequent steps in normal-normal model

For learning more you can check those slides and Conjugate Bayesian analysis of the Gaussian distribution paper by Kevin P. Murphy. Check also Do Bayesian priors become irrelevant with large sample size? You can also check those notes and this blog entry for accessible step-by-step introduction to Bayesian inference.

$\endgroup$
  • $\begingroup$ Thank you, this is very helpful. How would we go about solving this simple example (unknown variance, unlike your example)? Suppose we have a prior distribution of N~(5, 4) and then we observe 5 data points (8, 9, 10, 8, 7). What would be the posterior after these observations? Thank you in advance. Much appreciated. $\endgroup$ – statstudent Sep 27 '16 at 8:16
  • $\begingroup$ @Kelly you can find examples for cases when either variance is unknown and mean known, or both are unknown in the Wikipedia entry on conjugate priors and the links I provided in the end of my answer. If both mean and variance are unknown it becomes slightly more complicated. $\endgroup$ – Tim Sep 27 '16 at 8:18
  • $\begingroup$ @Kelly btw, you can check here for example of estimating both $\mu$ and $\sigma^2$. $\endgroup$ – Tim Sep 27 '16 at 14:17
4
$\begingroup$

If you have a prior $P(\theta)$ and a likelihood function $P(x \mid \theta)$ you can calculate the posterior with:

$$ P(\theta \mid x) = \frac{\sum_\theta P(x \mid \theta) P(\theta)}{P(x)} $$

Since $P(x)$ is just a normalization constant to make probabilities sum to one, you could write:

$$P(\theta \mid x) \sim \sum_\theta P(x \mid \theta)P(\theta) $$

Where $\sim$ means "is proportional to."

The case of conjugate priors (where you often get nice closed form formulas)

This Wikipedia article on conjugate priors may be informative. Let $\boldsymbol{\theta}$ be a vector of your parameters. Let $P(\boldsymbol{\theta})$ be a prior over your parameters. Let $P(\mathbf{x} \mid \boldsymbol{\theta})$ be the likelihood function, the probability of the data given the parameters. The prior is a conjugate prior for the likelihood function if the prior $P(\boldsymbol{\theta})$ and the posterior $P(\boldsymbol{\theta} \mid \mathbf{x})$ are in the same family (eg. both Gaussian).

The table of conjugate distributions may help build some intuition (and also give some instructive examples to work through yourself).

$\endgroup$
1
$\begingroup$

This is the central computation issue for Bayesian data analysis. It really depends on the data and distributions involved. For simple cases where everything can be expressed in closed form (e.g., with conjugate priors), you can use Bayes's theorem directly. The most popular family of techniques for more complex cases is Markov chain Monte Carlo. For details, see any introductory textbook on Bayesian data analysis.

$\endgroup$
  • $\begingroup$ Thank you so much! Sorry if this is a really stupid follow-up question, but in the simple cases that you mentioned, how exactly would we use Bayes's theorem directly? Would the distribution created by the sample mean and variance of the data points become the likelihood function? Thank you very much. $\endgroup$ – statstudent Sep 26 '16 at 18:44
  • $\begingroup$ @Kelly Again, it depends on the distribution. See e.g. en.wikipedia.org/wiki/Conjugate_prior#Example . (If I answered your question, don't forget to accept my answer by clicking on the check mark under the voting arrows.) $\endgroup$ – Kodiologist Sep 26 '16 at 18:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.