I am now working with the Gibbs sampling. One problem that puzzled me is that when we use the Gibbs sampling, we always sample randomly from the conditional probability. What will happen if we sample the most probable value instead? Thank you in advance.

  • 1
    $\begingroup$ Are you asking what happens if the most probable value is sampled or are you asking what would happen if we always just take the mode of the conditional distribution at that step? $\endgroup$ – user44764 May 29 '14 at 23:36
  • $\begingroup$ Thank you Matthew. Yes, I am asking what would happen if we always just take the mode of the conditional distribution at each step. $\endgroup$ – user46414 May 30 '14 at 19:05
  • $\begingroup$ You would certainly become overconfident in your posterior distributions for all parameters, because you are no longer accounting for as much uncertainty in one parameter. The impact of this is that you would have too low (i.e., too low compared to the true value) variance in your posterior distributions for all parameters (and especially so in the one). I can't really speak with any authority on the other impacts of this design choice. With that said, I have used this approach in testing phase to see how close my sampler is to the most likely values to see if my code is making sense. $\endgroup$ – user44764 May 30 '14 at 19:28
  • $\begingroup$ Your sampler will also tend to get stuck on a local maxima. $\endgroup$ – jerad May 31 '14 at 18:31
  • $\begingroup$ I was about to ask the same question and then I found this thread. I would like to add a sub-question here: would it be equivalent to some other method like some EM to find the MAP...? $\endgroup$ – alberto Jan 6 '15 at 7:18

This would no longer be an algorithm for sampling from the posterior, but rather be an algorithm for optimization of the posterior, i.e., finding the MAP estimator.

In fact, this algorithm would be exactly that of coordinate descent (although technically coordinate ascent). In this algorithm, one at a time, you optimize each parameter while keeping the others fixed. In general, this is considered a rather slow algorithm, especially if the parameters are highly correlated, but somewhat surprisingly, it is the algorithm used by LASSO.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.