I have written some code that basically does the following:-

1: creates a matrix of binary features and a matrix of scaled features ( in the range 0 to 1 )

2: RBM trains separately on each of the above features matrices to get weight matrices

3: horizontally stacks a random row vector for bias unit weights and the two weight matrices from step 2 to create a single, initial input layer to hidden layer weight matrix

4: matrix multiplies the input features by the matrix from step 3 and feeds forward through the logistic function hidden layer to create a hidden layer output

5: RBM trains on the hidden layer output from step 4 to get a weight matrix

6: horizontally stacks a random row vector for hidden layer bias unit weights and the weight matrix from step 5 to create a single, initial hidden layer to Softmax weight matrix

7: uses the initial weight matrices from steps 3 and 6 instead of random initialisation for backpropagation training of a Feedforward neural network

However, I am now wondering whether it's wise/correct to exclude the weights of the bias units from the RBM training?

  • $\begingroup$ You should definately be learning the "weights" of the biases. I use quotes around weights as it is normally easier (and quicker) to use a bias vector than append a 1 to every input. That being said I find your question very confusing. I'm not sure why you would be stacking random vectors onto anything. Are you talking about initialization? $\endgroup$ – alto Jan 9 '14 at 19:28
  • $\begingroup$ @alto Indeed I am talking about initialization. I'm using RBM training to extract features/initial weights prior to back prop training of a feed forward NN. I'm not using RBM to initialize the weights attached to the bias unit because a bias unit does not contain information in the same way the data inputs do, hence the stacking of a random weight vector for the weights that attach to the bias unit. However, after your comment I shall adjust my code to include the bias weights in the RBM training - it is a simple enough tweak. $\endgroup$ – babelproofreader Jan 10 '14 at 21:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.