0
$\begingroup$

I'm currently trying to debug my own RBM implementation by applying it to an autoencoder exercise I've done before. In this exercise I feed 8-dimensional unit vectors to 3 hidden nodes and force the network to learn a compressed binary representation, i.e.:

####### visible ########    ## hidden ##
[1, 0, 0, 0, 0, 0, 0, 0] ->  [0, 0, 0]
[0, 1, 0, 0, 0, 0, 0, 0] ->  [1, 0, 0]
[0, 0, 1, 0, 0, 0, 0, 0] ->  [0, 1, 0]
[0, 0, 0, 1, 0, 0, 0, 0] ->  [1, 1, 0]
          ...                   ...

With the three hidden nodes the network should be able to memorize / distinguish 2^3 = 8 different input vectors. Of course this requires heavy overfitting and the autoencoder only succeeded when I disabled the regularization term in the weight update rule.

Now when I train the RBM on this example, it is only able to reconstruct 3 vectors. All other inputs get scrambled, i.e.:

####### visible ########    ## hidden ##  #### reconstruction ####
[1, 0, 0, 0, 0, 0, 0, 0] ->  [0, 0, 0] -> [1, 0, 0, 1, 0, 1, 0, 0]
[0, 1, 0, 0, 0, 0, 0, 0] ->  [1, 0, 0] -> [0, 1, 0, 0, 0, 0, 0, 0] *correct
[0, 0, 1, 0, 0, 0, 0, 0] ->  [0, 0, 1] -> [0, 0, 1, 0, 0, 0, 0, 0] *correct
[0, 0, 0, 1, 0, 0, 0, 0] ->  [0, 0, 0] -> [1, 0, 1, 1, 1, 0, 1, 1]
[0, 0, 0, 0, 1, 0, 0, 0] ->  [0, 1, 0] -> [0, 0, 0, 0, 1, 0, 0, 0] *correct
          ...                   ...                 ...

At first I thought that I had a bug in my RBM implementation, so I've tried others (e.g. this implementation by edwin chen), but the results were similar.

I've heard that the binary units of an RBM also act as regularizers. Is this the reason why the example doesn't work well? Or should I continue searching for bugs in my code?

$\endgroup$
1
$\begingroup$

I'm not totally sure but yes, binary RBM models comes with a builtin regularizer. When you compare the activation value in one unit with a number sampled from a uniform distribution you're limiting the amount of information this unit can transmit from $[0, 1]$ to $\{0, 1\}$ and this acts as a regularizer.

ref: A Practical Guide to Training Restricted Boltzmann Machines https://www.cs.toronto.edu/~hinton/absps/guideTR.pdf

Maybe if you train for a very long time you can achieve the same result as the auto encoder.

$\endgroup$
  • $\begingroup$ Thanks for your answer. Hinton's guide was what I was referring to, but I missed the link here. I also think, that this is the reason, but it doesn't seem to improve with longer training. I trained for 100,000 epochs and there was almost no improvement after ~20,000 epochs. $\endgroup$ – Griddo Jan 28 '16 at 9:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.