I'm working on an implementation of a Restricted Boltzman Machine (RBM). I made some tests on the MNIST dataset trying to learn a representation of the digit 2. My inputs are binary images. My aim is to get results similar to the one of Geoffrey Hinton presented in this video: (12:26).Here are the different features learnt by my implementation (each image represent the weights linking the visible layer to one hidden neuron):

As you can see they are all very similar. It's probably a mistake in my code but somehow I don't see why the features should be different. They are all representing a sort of average of the input so in a sense they are "correctly" approximating the data.

My implementation works as follows:

  • First we select a minibatch of input images.
  • Step1: For each of the input in the minibatch we sample the hidden layer. Comparing the visible layer and the sampled hidden layer we compute <hj.vi>data and average over all the minibatch.
  • Step2: I use Persistent Contrastive Divergence (PCD) so in order to resample my visible layer I use the version of the hidden layer previously sampled at step2. I sample once more the hidden layer before computing <hj.vi>recon and averaging over all the minibatch.
  • Step3: I update all the weights wij <- wij + l*(<hj.vi>data - <hj.vi>recon)

Does it seems coherent to you? Do you see anything I could add to encourage diversity among my features, or are those results incoherent and I should recheck my implementation?

EDIT 1: Comparison with LDA

Another generative model that I know is Latent Dirichlet Allocation (LDA). The outcome of LDA is very similar to the one of RBM in the sense that you get for one document (one image for RBM), a probability of activating each topic (each feature for RBM), as well as a probability knowing the topic, to generate each word of the vocabulary (each visible unit for RBM). So I used LDA on my images, each document is an image, each pixel is a word. The images being binary you can only have at most one occurrence of a word in a document. Here is the result:

The quality of those topics is far better than those obtained with my RBM. I guess I'll have to check the code again. However my question still holds, the results obtained with LDA are very different from the ones of Hinton. They seem closer to topics than to features, which makes sense since LDA is a topic modeling method. What pushes a RBM to generate local features instead of global features like LDA ?

  • 1
    $\begingroup$ Were you able to find why you were losing diversity in the RBM? What if you decrease the size of your minibatches? How about just one image per batch? I'm guessing that might do it. $\endgroup$ – highBandWidth Jun 10 '16 at 3:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.