How can I calculate a probability from a likelihood, e.g. in the Metropolis-Hastings algorithm?

Question

This is a follow-up to my previous question, how can I compute a posterior density estimate from a prior and a likelihood

I am having difficulty understanding how it is possible to calculate the probability of a parameter set given its likelihood.

For example, in the answer to the previous question, the likelihood is used to weight the probability of sampling from the prior distributions.

However, the answer suggested I use a sequential sampling algorithm, so I have chosen to try a Metropolis-Hastings approach. However, it is not clear to me how I can use likelihoods to determine the probability that I will accept a proposed parameter set.

I can empirically estimate (using the R fitdistr function) that the likelihood is approximately $L\sim\textrm{log-}N(8, 0.7)$, I suppose that I could calculate the probability as plnorm(L, 8, 0.7) and the probability of acceptance as $P(L_\textrm{new})/P(L_\textrm{old})$. But is there a way to do this without assuming that the likelihood has some particular distribution? Or is it standard practice to make such an assumption?

Answer 1

You always need some form of a likelihood. Usually knowledge of the science leads to the likelihood, things like: will the data be discrete or continuous? what is the valid range of the data? what is the likely range of the data? What are reasonable shapes for the data? will contribute to choosing a likelihood function.

The likelihood function only need to show the probabilistic relationship between the parameter(s) and the data, it does not need estimates of the parameters, just knowing that if you knew the parameters the data would follow the normal (at least approximately) or other distribution.

The fitdistr function is an alternative to the Bayesian methodology, not usually a preparation for it. For both the Bayesian and the fitdistr approaches you specify a likelihood/distribution (it looks like you tried the log-normal) then estimate the parameters. If you use a difuse enough prior in the Bayesian method then you will find the answers match very closely between the 2 methods. But neither the fitdistr function or the Bayesian methods tell you whether the log-normal (or other) liklihood is the best or even appropriate one, for that you need to understand the science behind the data.

Once you have a prior and the liklihood (with the data), then you can start to apply things like the Metropolis-Hastings algorithm. Start with a guess or guesses for your parameters (need to be plausible), then generate a new candidate point (often adding a random normal to the current guess, but there are other ways), if the new candidate point has higher likelihood then accept the new point, if it has lower likelihood then you accept it with probability equal to the ratio between the likelihoods of the new and old point. If you don't accept it then you keep the old point another iteration. Do this a bunch of times and the set of points are draws from the posterior distribution. You can program this by hand to help yourself learn, but there are tools that make this much easier when looking for answers on real data.