I am now working with the Gibbs sampling. One problem that puzzled me is that when we use the Gibbs sampling, we always sample randomly from the conditional probability. What will happen if we sample the most probable value instead? Thank you in advance.
This would no longer be an algorithm for sampling from the posterior, but rather be an algorithm for optimization of the posterior, i.e., finding the MAP estimator.
In fact, this algorithm would be exactly that of coordinate descent (although technically coordinate ascent). In this algorithm, one at a time, you optimize each parameter while keeping the others fixed. In general, this is considered a rather slow algorithm, especially if the parameters are highly correlated, but somewhat surprisingly, it is the algorithm used by LASSO.