bayesian estimation

I saw this image in a presentation and I don't know if I get it right, but does it mean that:

  • if I have a prior distribution, like the blue distribution in the image
  • if I have a likelihood distribution (red one)
  • then I am able to calculate the posterior distribution. Maybe it is difficult to calculate a parametric distribution, then is it possible to do simulations? If you could give a simple example to get the posterior in R, it will help to understand.

Let's say:

  • we assume that the prior is a normal law (mean=1,sd=2)
  • the likelihood is another normal law (mean=3,sd=1.5). Yes it is wired, but it is just an example
  • then can we calculate a posterior ?

If I am totally wrong, could you please tell me where I get it wrong? Thank you very much.


closed as too broad by Sycorax, kjetil b halvorsen, John, Tim, Nick Cox Feb 19 '16 at 8:41

Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. Avoid asking multiple distinct questions at once. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • $\begingroup$ The answer is an emphatic "yes" -- if conjugacy fails, then you're in the domain of Bayesian computing. The workhorse method is Markov Chain Monte Carlo, but Hamiltonian Monte Carlo is the current state-of-the-art. But this topic is very, very large, so I'm voting to close as too broad. It's kind of like asking "If I have some measurements, is it possible to make inferences about the larger population?" $\endgroup$ – Sycorax Feb 18 '16 at 15:35
  • 1
    $\begingroup$ Nevertheless, your example is strange. The posterior of what parameter are your refering ? A unknown parameter is missing in your example. $\endgroup$ – peuhp Feb 18 '16 at 15:37

From Bayes' theorem, we have

$$\text{posterior}(\theta) \propto \text{prior}(\theta)\times \mathcal{L}(\theta)$$

That is, at any point ($\theta=\theta_0$, say), you can (up to a scaling constant) evaluate the posterior by taking the product of the prior and the likelihood at that point.

So you could, for example, evaluate the posterior over a fine grid of values simply by calculating the prior and the likelihood over that grid and taking the product at each point.

You could also evaluate the scaling constant from such a grid of values, say via some reasonable numerical quadrature method (possibly one for values available on a grid; Simpson's rule might suffice) for example.

Here's an example of such a numerically calculated posterior from a likelihood and a prior:

enter image description here

In the above example, I created a function to take the product of the prior and the likelihood, and passed that to R's integrate function (which doesn't simply integrate over a grid, its approach is a bit more involved); I then divided the product function by that (numeric) integral to produce a properly-scaled posterior density for the plot (accurate to a teensy fraction of a pixel, much more than we need for the plot).

In the case of conjugate priors, as in your Gaussian-Gaussian example, you can determine the scaling constant algebraically.

 plot(xx,dgamma(xx,5,.2),type="l",col=4,ylim=c(0,0.06),ylab="f",xlab="theta") # prior
 lines(xx,dgamma(xx,20,.4),col=2) # likelihood (e.g. exponential, n=20)

 f=function(xx) dgamma(xx,20,.4)*dgamma(xx,5,.2)
 integrate(f,0,Inf)  #gives: "0.006575191 with absolute error < 1.3e-06"

 lines(xx,dgamma(xx,20,.4)*dgamma(xx,5,.2)/0.006575191 ,col=3)  # scaled posterior
 abline(h=0,lty=3,col="dimgrey") #dot in x-axis
  • $\begingroup$ Thank you very much. could you please share your R code? $\endgroup$ – XR SC Feb 19 '16 at 17:19
  • $\begingroup$ @ZShiliao done... $\endgroup$ – Glen_b Feb 19 '16 at 23:42

Not the answer you're looking for? Browse other questions tagged or ask your own question.