1
$\begingroup$

I am trying to understand how the choice of priors affects a Bayesian model estimated using MCMC. At a basic level I understand that the product of the prior and the likelihood are proportional to the posterior. However, I do not fully understand (a) how to place more or less weight on the prior or (b) how the prior in this example influences the results. The prior in this example is defined as:

# Prior distribution
prior <- function(param){
    a = param[1]
    b = param[2]
    sd = param[3]
    aprior = dunif(a, min=0, max=10, log = T)
    bprior = dnorm(b, sd = 5, log = T)
    sdprior = dunif(sd, min=0, max=30, log = T)
    return(aprior+bprior+sdprior)
}

As I understand the example, whatever value of b was drawn by the proposal function is considered by the prior to be the most probable. Is that correct? Doesn't this prior essentially place all of the action on the likelihood function because all values of b will have the same density under the prior that the proposed b is the mean of b's distribution?

More generally, how do I place more or less weight on the prior in the posterior. If $\text{posterior} \propto \text{prior} \times \text{likelihood}$ I don't know how to adjust the weight of the prior. If it's in log form as in the example the log of the posterior is defined as:

posterior <- function(param){
   return (likelihood(param) + prior(param))
}

Here it seems like I could put a weight in front of either the likelihod or the prior. For example, likelihood(param) + 2 * prior(param) would give the prior twice as much weight as the likelihood. Is that intuition correct or am I on the wrong track?

$\endgroup$
12
  • 3
    $\begingroup$ Not at all clear what you are doing. $\endgroup$
    – jaradniemi
    Apr 24, 2015 at 1:08
  • 2
    $\begingroup$ To place more weight on the prior, decrease its variance... $\endgroup$
    – Sycorax
    Apr 24, 2015 at 2:11
  • 2
    $\begingroup$ @Michael it seems you get -1's because your question is not really clear and it is hard to understand it. The main problem is: why do you want to weight your prior? What do you want to achieve by that? $\endgroup$
    – Tim
    Apr 24, 2015 at 6:25
  • 3
    $\begingroup$ A prior $\pi$ is a probability density. A powered prior $\pi^2$ or $\pi^{1/2}$ is no longer a probability density. By weighting the prior, you modify the whole paradigm. $\endgroup$
    – Xi'an
    Apr 24, 2015 at 6:49
  • 3
    $\begingroup$ Michael, if you have questions regarding my blog post, why don't you ask them at the blog where the context is a lot more clear. As said below, priors are not getting weighted, but you can increase weight by choosing more narrow priors. $\endgroup$ Apr 24, 2015 at 12:18

2 Answers 2

14
$\begingroup$

A prior is a distribution.

You don't really weight that distribution, Bayes' rule does.

However, if you're thinking of your prior distribution as representing uncertainty "about" some value you expect a priori, so that the mode (or whatever measure of the center) of the prior expresses your prior knowledge of where you think it lies and the spread represents the uncertainty about where it is, then increasing the spread puts less weight near that prior central value and decreasing the spread puts more weight near that prior central value.

So choose a more concentrated prior to put more weight near that value.

$\endgroup$
4
  • $\begingroup$ That makes sense in general. What about the specific prior selected in the above example. Doesn't it essentially treat all draws in the MCMC as the same within the prior because it shifts the mean of the prior to the value of the current draw for b? $\endgroup$
    – Michael
    Apr 24, 2015 at 4:24
  • 4
    $\begingroup$ Well, I don't plan to try to read your code, and your question just now doesn't make sense to me. Why would you be changing priors as you're sampling? $\endgroup$
    – Glen_b
    Apr 24, 2015 at 4:53
  • $\begingroup$ "Why would you be changing priors as you're sampling?" Agreed, this seems odd to me. But it's not my code, it's an example I'm trying to learn from, which is why I'm asking (a) did I understand the code correctly and (b) is this code correct? $\endgroup$
    – Michael
    Apr 24, 2015 at 17:14
  • 2
    $\begingroup$ The code in your post isn't changing priors while sampling (it's simply sampling from the prior by the look). You seem to want to either change Bayes' rule or to modify the prior at the sampling step, but it's not clear. You can validly change your prior as indicated in the answer, but you can't change the way the prior enters the posterior and still call it Bayesian. $\endgroup$
    – Glen_b
    Apr 25, 2015 at 2:06
23
$\begingroup$

You should definitely invest some time in learning the bases of Bayesian statistics and MCMC methods from textbooks or on-line courses.

The title and the wording of the question seem to indicate some confusion between the prior modelling [which pertains to the statistical model] and the MCMC implementation [which pertains to the computational resolution].

As I understand the example, whatever value of b was drawn by the proposal function is considered by the prior to be the most probable. Is that correct?

This question is about the MCMC algorithm, not about the prior. The proposal distribution is used in an MCMC algorithm to propose a move in the parameter space. For instance in Gibbs sampling you could propose a new value of $b$, $b'$ say, using a random walk proposal, $$b'\sim\text{N}(b,5^2)$$This new value of $b$ will then be compared to the previous value of $b$ by a Metropolis--Hastings ratio$$\frac{\pi(b')f(x|a,b',\tau)}{\pi(b)f(x|a,b,\tau)}$$So the new value will be accepted for certain if it is more likely (or `more probable' in your wording) than the previous one. And it may still be accepted if less likely.

Doesn't this prior essentially place all of the action on the likelihood function because all values of b will have the same density under the prior that the proposed b is the mean of b's distribution?

I do not understand what `this prior' is but judging from your code, it appears that$$b\sim\text{N}(0,5^2)$$so the different values of $b$ you can encounter have a different prior density. When you run the Metropolis--Hastings with the random walk proposal$$b'\sim\text{N}(b,5^2)$$ it is indeed centred at the previous value $b$. But the result of this simulation can be any number compatible with this distribution, rather than its mean $b$. Hence $\pi(b)\ne\pi(b')$ with probability one. And therefore the prior has a say in deciding whether or not accepting $b'$. For instance, if $b'>3\times 5$, 3 times the standard deviation, $\pi(b')$ is essentially zero.

$\endgroup$
6
  • 9
    $\begingroup$ "You should definitely invest some time in learning" qualifies as the most patronizing start to an answer I have ever received on a Stack Exchange site. I've looked through a number of textbooks and am working through an online course. I am trying to learn. I have a much easier time though learning from examples with code that implements an estimator. Everything in your answer is in the textbook, but your answer does not seem to describe the code. In particular, while b is drawn as you describe the code appears to always shift the center of the prior to b bprior = dnorm(b, sd = 5, log = T) $\endgroup$
    – Michael
    Apr 24, 2015 at 17:33
  • 2
    $\begingroup$ @Michael In case you have not yet looked at it, may I recommend Michael Clark's Bayesian Basics? $\endgroup$
    – Fr.
    Apr 30, 2015 at 7:44
  • 4
    $\begingroup$ Michael's question was uninformed and based on a lack of knowledge. Just by reading his question (and his comment here), it is clear that the problem is a lack of knowledge of R: "because all values of b will have the same density under the prior that the proposed b is the mean of b's distribution". Equally clearly, the OP is not trying to reinvent Bayesian statistics, as he is lacking knowledge of that, too. However, I think the only right answer would have been to politely point out that this is a SO question about using R and distributions correctly, and not a XV question about MCMC. $\endgroup$
    – fnl
    Apr 30, 2015 at 8:44
  • 7
    $\begingroup$ @Michael Dude. You're getting free advice from a leading mind in Bayesian statistics. You should listen when he speaks -- even if you think he's being rude (and he's not, in my opinion!), you could learn so much with an attitude more open to criticism. $\endgroup$
    – Sycorax
    Apr 30, 2015 at 18:08
  • 7
    $\begingroup$ People mean well, but take for example "Michael's question was uninformed and based on a lack of knowledge." This is factually true and yet still very rude. You have to be willing to ask questions that betray a lack of understanding so that you can learn. Shaming people or berating people for asking such questions is an incredibly ineffective way to teach. Similarly, "it is clear that the problem is a lack of knowledge of R." No, perfectly comfortable programming in R. Just misunderstood the snytax of a single function absent the explicit declaration of the default value for a parameter. $\endgroup$
    – Michael
    May 7, 2015 at 22:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.