I'm trying to understand the difference between a restricted Boltzmann machine (RBM), and a feed-forward neural network (NN). I know that an RBM is a generative model, where the idea is to reconstruct the input, whereas an NN is a discriminative model, where the idea is the predict a label. But what I am unclear about, is why you cannot just use a NN for a generative model? In particular, I am thinking about deep belief networks and multi-layer perceptrons.

Suppose my input to the NN is a set of notes called x, and my output of the NN is a set of nodes y. In a discriminative model, my loss during training would be the difference between y, and the value of y that I want x to produce (e.g. probabilities for class labels). However, what about if I just made the output have the same number of nodes as the input, and then set the loss to be the difference between x and y? In this way, the network would learn to reconstruct the input, like in an RBM.

So, given that a NN (or a multi-layer perceptron) can be used to train a generative model, why would you use an RBM (or a deep belief network) instead? Or in this case, would they be exactly the same?

  • $\begingroup$ Discriminate models do not have an inherent "will" to create data; that is, they operate on data they are fed in, they do not create data by themselves. You would have to switch to a recurrent model (even just feeding some outputs of the network back as inputs makes it a simple recurrent network) for there to be any working memory to create with, and you would need to connect a random number generator to seed creation with. At this point you have half-reinvented what Boltzmann machines do. $\endgroup$ – Skrylar Jan 13 '17 at 4:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.