0
$\begingroup$

I am trying to understand how the choice of priors affects a Bayesian model estimated using MCMC. At a basic level I understand that the product of the prior and the likelihood are proportional to the posterior. However, I do not fully understand (a) how to place more or less weight on the prior or (b) how the prior in this example influences the results. The prior in this example is defined as:

# Prior distribution
prior <- function(param){
    a = param[1]
    b = param[2]
    sd = param[3]
    aprior = dunif(a, min=0, max=10, log = T)
    bprior = dnorm(b, sd = 5, log = T)
    sdprior = dunif(sd, min=0, max=30, log = T)
    return(aprior+bprior+sdprior)
}

As I understand the example, whatever value of b was drawn by the proposal function is considered by the prior to be the most probable. Is that correct? Doesn't this prior essentially place all of the action on the likelihood function because all values of b will have the same density under the prior that the proposed b is the mean of b's distribution?

More generally, how do I place more or less weight on the prior in the posterior. If $\text{posterior} \propto \text{prior} \times \text{likelihood}$ I don't know how to adjust the weight of the prior. If it's in log form as in the example the log of the posterior is defined as:

posterior <- function(param){
   return (likelihood(param) + prior(param))
}

Here it seems like I could put a weight in front of either the likelihod or the prior. For example, likelihood(param) + 2 * prior(param) would give the prior twice as much weight as the likelihood. Is that intuition correct or am I on the wrong track?

$\endgroup$
  • 3
    $\begingroup$ Not at all clear what you are doing. $\endgroup$ – jaradniemi Apr 24 '15 at 1:08
  • 2
    $\begingroup$ To place more weight on the prior, decrease its variance... $\endgroup$ – Sycorax Apr 24 '15 at 2:11
  • 2
    $\begingroup$ @Michael it seems you get -1's because your question is not really clear and it is hard to understand it. The main problem is: why do you want to weight your prior? What do you want to achieve by that? $\endgroup$ – Tim Apr 24 '15 at 6:25
  • 3
    $\begingroup$ A prior $\pi$ is a probability density. A powered prior $\pi^2$ or $\pi^{1/2}$ is no longer a probability density. By weighting the prior, you modify the whole paradigm. $\endgroup$ – Xi'an Apr 24 '15 at 6:49
  • 3
    $\begingroup$ Michael, if you have questions regarding my blog post, why don't you ask them at the blog where the context is a lot more clear. As said below, priors are not getting weighted, but you can increase weight by choosing more narrow priors. $\endgroup$ – Florian Hartig Apr 24 '15 at 12:18
12
$\begingroup$

A prior is a distribution.

You don't really weight that distribution, Bayes' rule does.

However, if you're thinking of your prior distribution as representing uncertainty "about" some value you expect a priori, so that the mode (or whatever measure of the center) of the prior expresses your prior knowledge of where you think it lies and the spread represents the uncertainty about where it is, then increasing the spread puts less weight near that prior central value and decreasing the spread puts more weight near that prior central value.

So choose a more concentrated prior to put more weight near that value.

$\endgroup$
  • $\begingroup$ That makes sense in general. What about the specific prior selected in the above example. Doesn't it essentially treat all draws in the MCMC as the same within the prior because it shifts the mean of the prior to the value of the current draw for b? $\endgroup$ – Michael Apr 24 '15 at 4:24
  • 4
    $\begingroup$ Well, I don't plan to try to read your code, and your question just now doesn't make sense to me. Why would you be changing priors as you're sampling? $\endgroup$ – Glen_b Apr 24 '15 at 4:53
  • $\begingroup$ "Why would you be changing priors as you're sampling?" Agreed, this seems odd to me. But it's not my code, it's an example I'm trying to learn from, which is why I'm asking (a) did I understand the code correctly and (b) is this code correct? $\endgroup$ – Michael Apr 24 '15 at 17:14
  • 2
    $\begingroup$ The code in your post isn't changing priors while sampling (it's simply sampling from the prior by the look). You seem to want to either change Bayes' rule or to modify the prior at the sampling step, but it's not clear. You can validly change your prior as indicated in the answer, but you can't change the way the prior enters the posterior and still call it Bayesian. $\endgroup$ – Glen_b Apr 25 '15 at 2:06
18
$\begingroup$

You should definitely invest some time in learning the bases of Bayesian statistics and MCMC methods from textbooks or on-line courses.

The title and the wording of the question seem to indicate some confusion between the prior modelling [which pertains to the statistical model] and the MCMC implementation [which pertains to the computational resolution].

As I understand the example, whatever value of b was drawn by the proposal function is considered by the prior to be the most probable. Is that correct?

This question is about the MCMC algorithm, not about the prior. The proposal distribution is used in an MCMC algorithm to propose a move in the parameter space. For instance in Gibbs sampling you could propose a new value of $b$, $b'$ say, using a random walk proposal, $$b'\sim\text{N}(b,5^2)$$This new value of $b$ will then be compared to the previous value of $b$ by a Metropolis--Hastings ratio$$\frac{\pi(b')f(x|a,b',\tau)}{\pi(b)f(x|a,b,\tau)}$$So the new value will be accepted for certain if it is more likely (or `more probable' in your wording) than the previous one. And it may still be accepted if less likely.

Doesn't this prior essentially place all of the action on the likelihood function because all values of b will have the same density under the prior that the proposed b is the mean of b's distribution?

I do not understand what `this prior' is but judging from your code, it appears that$$b\sim\text{N}(0,5^2)$$so the different values of $b$ you can encounter have a different prior density. When you run the Metropolis--Hastings with the random walk proposal$$b'\sim\text{N}(b,5^2)$$ it is indeed centred at the previous value $b$. But the result of this simulation can be any number compatible with this distribution, rather than its mean $b$. Hence $\pi(b)\ne\pi(b')$ with probability one. And therefore the prior has a say in deciding whether or not accepting $b'$. For instance, if $b'>3\times 5$, 3 times the standard deviation, $\pi(b')$ is essentially zero.

$\endgroup$
  • 5
    $\begingroup$ "You should definitely invest some time in learning" qualifies as the most patronizing start to an answer I have ever received on a Stack Exchange site. I've looked through a number of textbooks and am working through an online course. I am trying to learn. I have a much easier time though learning from examples with code that implements an estimator. Everything in your answer is in the textbook, but your answer does not seem to describe the code. In particular, while b is drawn as you describe the code appears to always shift the center of the prior to b bprior = dnorm(b, sd = 5, log = T) $\endgroup$ – Michael Apr 24 '15 at 17:33
  • 2
    $\begingroup$ @Michael In case you have not yet looked at it, may I recommend Michael Clark's Bayesian Basics? $\endgroup$ – Fr. Apr 30 '15 at 7:44
  • 4
    $\begingroup$ Michael's question was uninformed and based on a lack of knowledge. Just by reading his question (and his comment here), it is clear that the problem is a lack of knowledge of R: "because all values of b will have the same density under the prior that the proposed b is the mean of b's distribution". Equally clearly, the OP is not trying to reinvent Bayesian statistics, as he is lacking knowledge of that, too. However, I think the only right answer would have been to politely point out that this is a SO question about using R and distributions correctly, and not a XV question about MCMC. $\endgroup$ – fnl Apr 30 '15 at 8:44
  • 3
    $\begingroup$ @Michael Dude. You're getting free advice from a leading mind in Bayesian statistics. You should listen when he speaks -- even if you think he's being rude (and he's not, in my opinion!), you could learn so much with an attitude more open to criticism. $\endgroup$ – Sycorax Apr 30 '15 at 18:08
  • 1
    $\begingroup$ "you could learn so much with an attitude more open to criticism" I am open to criticism and I did learn something from this answer. As a teacher, however, it's generally ineffective to insult your students. New users on Stack Exchange sites are often berated for how stupid their question is rather than helped to ask the right question. $\endgroup$ – Michael May 7 '15 at 22:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.