Priors in Bayesian MCMC

Question

I am trying to understand how the choice of priors affects a Bayesian model estimated using MCMC. At a basic level I understand that the product of the prior and the likelihood are proportional to the posterior. However, I do not fully understand (a) how to place more or less weight on the prior or (b) how the prior in this example influences the results. The prior in this example is defined as:

# Prior distribution
prior <- function(param){
    a = param[1]
    b = param[2]
    sd = param[3]
    aprior = dunif(a, min=0, max=10, log = T)
    bprior = dnorm(b, sd = 5, log = T)
    sdprior = dunif(sd, min=0, max=30, log = T)
    return(aprior+bprior+sdprior)
}

As I understand the example, whatever value of b was drawn by the proposal function is considered by the prior to be the most probable. Is that correct? Doesn't this prior essentially place all of the action on the likelihood function because all values of b will have the same density under the prior that the proposed b is the mean of b's distribution?

More generally, how do I place more or less weight on the prior in the posterior. If $\text{posterior} \propto \text{prior} \times \text{likelihood}$ I don't know how to adjust the weight of the prior. If it's in log form as in the example the log of the posterior is defined as:

posterior <- function(param){
   return (likelihood(param) + prior(param))
}

Here it seems like I could put a weight in front of either the likelihod or the prior. For example, likelihood(param) + 2 * prior(param) would give the prior twice as much weight as the likelihood. Is that intuition correct or am I on the wrong track?

Answer 1

A prior is a distribution.

You don't really weight that distribution, Bayes' rule does.

However, if you're thinking of your prior distribution as representing uncertainty "about" some value you expect a priori, so that the mode (or whatever measure of the center) of the prior expresses your prior knowledge of where you think it lies and the spread represents the uncertainty about where it is, then increasing the spread puts less weight near that prior central value and decreasing the spread puts more weight near that prior central value.

So choose a more concentrated prior to put more weight near that value.

Answer 2

You should definitely invest some time in learning the bases of Bayesian statistics and MCMC methods from textbooks or on-line courses.

The title and the wording of the question seem to indicate some confusion between the prior modelling [which pertains to the statistical model] and the MCMC implementation [which pertains to the computational resolution].

As I understand the example, whatever value of b was drawn by the proposal function is considered by the prior to be the most probable. Is that correct?

This question is about the MCMC algorithm, not about the prior. The proposal distribution is used in an MCMC algorithm to propose a move in the parameter space. For instance in Gibbs sampling you could propose a new value of $b$, $b'$ say, using a random walk proposal, $$b'\sim\text{N}(b,5^2)$$This new value of $b$ will then be compared to the previous value of $b$ by a Metropolis--Hastings ratio$$\frac{\pi(b')f(x|a,b',\tau)}{\pi(b)f(x|a,b,\tau)}$$So the new value will be accepted for certain if it is more likely (or `more probable' in your wording) than the previous one. And it may still be accepted if less likely.

Doesn't this prior essentially place all of the action on the likelihood function because all values of b will have the same density under the prior that the proposed b is the mean of b's distribution?

I do not understand what `this prior' is but judging from your code, it appears that$$b\sim\text{N}(0,5^2)$$so the different values of $b$ you can encounter have a different prior density. When you run the Metropolis--Hastings with the random walk proposal$$b'\sim\text{N}(b,5^2)$$ it is indeed centred at the previous value $b$. But the result of this simulation can be any number compatible with this distribution, rather than its mean $b$. Hence $\pi(b)\ne\pi(b')$ with probability one. And therefore the prior has a say in deciding whether or not accepting $b'$. For instance, if $b'>3\times 5$, 3 times the standard deviation, $\pi(b')$ is essentially zero.