Why can't we use backpropagation and gradient descent on a Restricted Boltzmann Machine

Question

Can someone please explain why we cannot use the backpropagation algorithm and gradient descent to train a Restricted Boltzmann Machine. In other words, why can't we train an RBM in the same manner that we train a feedforward network?

Whenever I have googled around for answers to this question, I keep seeing answers that say the partition function for the RBM is intractable and so you need to use something like Gibbs sampling and Constrastive Divergence to train the RBM.

The problem with this answer is that no one really explains why you don't need to use the partition function when training a feedforward network? Whenever they teach feedforward networks in textbooks nowadays, they just start by showing the backpropagation algorithm. I have not seen anyone try to explain why you don't need to worry about a partition function for a feedforward network. Can someone explain this part?

I have used Gibbs sampling and Metropolis-Hastings when training hierarchical Bayesian models. So I understand the reason why the partition function is so complicated in that circumstance because you are really trying to use MCMC to estimate the posterior distribution of the parameter. But aren't we trying to do the same thing in a feedforward network when we are trying to learn weights from the data?

So that is where I am getting confused. Why MCMC with RBMs but not feedforward networks?  

Answer 1

An RBM is a graphical model with unobserved latent variables, making the marginal distribution intractable.

A typical neural network is not a graphical model. It does not have any unobserved latent variables. Sometimes, a hidden layer in the network is referred to as the "latent space", but nonetheless it is a deterministic function of the input without any real probabilistic interpretation. Therefore, there are no unobserved variables which need to be integrated over, and no partition function.

Whereas RBM is a generative model which directly models the distribution of the data $p(x)$, most neural networks are used in a discriminative fashion and model $p(y|x)$, so they are not really comparable.

When neural networks are used for modeling $p(x)$, it is usually as a product of conditionals $p(\mathbf x) = p(x_0)\prod p(x_i|x_{< i}; \theta)$ which does not use any unobserved variables -- see language modeling -- or a VAE model, where the intractable parts are avoided by indirectly optimizing a lower bound on $p(x)$.

Answer 2

Backpropagation is the reverse-mode differentiation and it only applies to acyclic networks with directed edges(as stated in this article: Back-propagation, an introduction).

RBMs are equivalent to infinite directed belief networks(see this article: Learning Deep Architectures for AI), then we cannot apply backpropagation to the infinitely deep neural network.