I came across the following quote on enter link description here

if we made MCMC as reliable as stochastic gradient descent now is for deep networks, that could mean a resurgence of more explicit Bayesian/probabilistic models and hierarchical graphical models, of which RBMs are just one example.

Is it really true that MCMC is not as reliable as SGD?


In practice, the reliability of both methods will depend on the situation. But, here are a few points to consider:

MCMC must sample the space in a way that's representative of the distribution. This can require a large number of points, particularly in high dimensions. In contrast, SGD doesn't care about the overall structure of the objective function; it only needs to find the minimum, and tries to do this by stepping downhill at each point. In a sense, this is a simpler problem.

However, when the objective function has multiple local minima, SGD can easily get stuck--once we enter the basin surrounding a local minimum, there may be no escape. So, SGD isn't a feasible strategy for problems where there are multiple local minima but we need the global minimum. In the context of deep learning, objective functions are highly nonconvex and contain many local minima. But, the saving grace is that many of these local minima can correspond to networks with good generalization performance. We don't need the global minimum and, in fact, we may not even want it (as it can have worse generalization performance than many of the local minima). SGD can also be trapped by saddle points (or at least take exponential time to escape from them). These are quite prevalent in cost functions for deep nets.

MCMC doesn't get stuck in the same sense because it doesn't seek extrema but, rather, aims to sample from or integrate over the entire distribution. But, MCMC can become trapped if the distribution contains widely separated modes, as the probability of transitioning to the low density region separating the modes will be small. There are various approaches for dealing with this situation.

Compared to SGD, MCMC can require more tuning to work on a particular problem. First, there's a choice of which MCMC algorithm to use, which is problem dependent. Then, there's a choice of algorithm-specific parameters, which can also be highly problem dependent. Finally, there's burn-in, discarding correlated samples, and number of iterations (perhaps not that big a deal in comparison to the previous choices). SGD is a single algorithm that tends to work in many circumstances (even if other optimization methods might be more efficient). There are only a few parameters, which are straightforward to set. Chief among them is learning rate, which can be set using cross validation. Batch size can typically be chosen a priori, and many choices work. Number of iterations can be chosen by looking for convergence or monitoring performance on the validation set (early stopping).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.