Is Coordinate Ascent algorithm related to Gibbs Sampling in some way? I wonder if only me feel there are certain connections between them, I googled it for a long time, but found no where mentioned these two method. But to me, they indeed looked so related, Could anyone help?
 A: The purpose of coordinate ascent is to maximize a function, whereas the purpose of Gibbs sampling is to draw samples from a probability distribution. The two methods are loosely related in the sense that they both fix all coordinates except for the one being updated, and take turns updating the individual coordinates. They differ in that coordinate ascent always steps uphill, whereas Gibbs sampling draws a sample from the marginal distribution. The probability density associated with the new point may be higher or lower than the previous point. If you were to apply coordinate ascent in place of Gibbs sampling, you'd converge to a local mode of the distribution, as in maximum likelihood or MAP estimation (as long as coordinate ascent doesn't get stuck). The sequence of points would no longer be a valid sample from the distribution. It doesn't make sense to talk about applying Gibbs sampling in place of coordinate ascent, because coordinate ascent deals with functions that aren't necessarily probability distributions.
You might also be interested to think about the similarities/differences between Metropolis-Hastings and simulated annealing.
A: I was reorgnizing my study notes and came across this old post. I guess there's no harm adding some details to user20160's answer.
Assume in a Bayesian setup where you want to make inference about the parameter $\theta = (\theta_1,\theta_2)^T$, and you know the form of the (probably unnormalized) posterior distribution, say $p(\theta)$. Your goal is either:

*

*get the MAP estimate of $\theta$ by finding $\theta_{MAP} =\arg \max_\theta p(\theta)$

*or, draw samples of $\theta$ from the (unnormalized) density function $p(\theta)$.

You can use coordinate ascent for 1 and Gibbs sampling for 2.
Coordinate ascent

*

*step1: randomly initialize $\theta^{(0)}=(\theta_1^{(0)},\theta_2^{(0)})$

*step2: iterate between:
$$
\theta_1^{(i)} = \theta_1^{(i-1)} + \alpha \frac{\partial p(\theta_1,\theta_2=\theta_2^{(i-1)})}{\partial \theta_1} \\
\theta_2^{(i)} = \theta_2^{(i-1)} + \alpha \frac{\partial p(\theta_1=\theta_1^{(i)},\theta_2)}{ \partial \theta_2}
$$
$\theta^{(n)}$ approaches a single point $\theta_{MAP}$ as $n$ increases.

Gibbs sampling

*

*step1: randomly initialize $\theta^{(0)}=(\theta_1^{(0)},\theta_2^{(0)})$

*step2: iterate between:
$$
\theta_1^{(i)} = \text{sample }\theta_1\text{ from }p(\theta_1,\theta_2=\theta_2^{(i-1)}) \\
\theta_2^{(i)} =  \text{sample }\theta_2\text{ from }p(\theta_1=\theta_1^{(i)},\theta_2)
$$
$\{\theta^{(i)};i=1:n\}$ resemble draws from $p(\theta)$ as $n$ increases.

As you can see, Gibbs sampling can somehow be seen as a randomized version of the coordinate ascent. Coordinate ascent is a fixed direction version of Gibbs sampling. If describe from a sampling perspective,  coordinate ascent "draws samples" towards the modes of the conditionals in each iteration (the conditionals are $p(\theta_1,\theta_2^{(i-1)})$ and $p(\theta_1^{(i)},\theta_2)$) by moving the previous sample in the direction of the gradient, eventually the latest "sample" will approach the mode of the joint distribution.
