I am now working with the Gibbs sampling. One problem that puzzled me is that when we use the Gibbs sampling, we always sample randomly from the conditional probability. What will happen if we sample the most probable value instead? Thank you in advance.
5
-
1$\begingroup$ Are you asking what happens if the most probable value is sampled or are you asking what would happen if we always just take the mode of the conditional distribution at that step? $\endgroup$– user44764Commented May 29, 2014 at 23:36
-
$\begingroup$ Thank you Matthew. Yes, I am asking what would happen if we always just take the mode of the conditional distribution at each step. $\endgroup$– user46414Commented May 30, 2014 at 19:05
-
$\begingroup$ You would certainly become overconfident in your posterior distributions for all parameters, because you are no longer accounting for as much uncertainty in one parameter. The impact of this is that you would have too low (i.e., too low compared to the true value) variance in your posterior distributions for all parameters (and especially so in the one). I can't really speak with any authority on the other impacts of this design choice. With that said, I have used this approach in testing phase to see how close my sampler is to the most likely values to see if my code is making sense. $\endgroup$– user44764Commented May 30, 2014 at 19:28
-
$\begingroup$ Your sampler will also tend to get stuck on a local maxima. $\endgroup$– jeradCommented May 31, 2014 at 18:31
-
$\begingroup$ I was about to ask the same question and then I found this thread. I would like to add a sub-question here: would it be equivalent to some other method like some EM to find the MAP...? $\endgroup$– albertoCommented Jan 6, 2015 at 7:18
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
This would no longer be an algorithm for sampling from the posterior, but rather be an algorithm for optimization of the posterior, i.e., finding the MAP estimator.
In fact, this algorithm would be exactly that of coordinate descent (although technically coordinate ascent). In this algorithm, one at a time, you optimize each parameter while keeping the others fixed. In general, this is considered a rather slow algorithm, especially if the parameters are highly correlated, but somewhat surprisingly, it is the algorithm used by LASSO.
