I am Pradeep pursuing M.tech at S.I.T, Tumkur, India. I have stuck in a problem while implementing Deep Belief Networks for Isolated Word Recognition. The procedure i have adopted is described below, please do help me in analysing the stage where i went wrong.
Back-propagation algorithm implementation : The Normal back-propagation algorithm that I have used for Isolated Word Recognition contains the following details of the feed forward architecture;
- Input matrix -180×132 i.e., 180 inputs are fed to the FFNN and 33 such vectors belonging to each class. So there are 4 classes hence 33×4=132.
- I have used a single hidden layer with 30 hidden nodes and the activation function is ‘sigmoid’ (1/1+exp(-x))
- The output layer has 4 nodes so as to classify 4 different classes with the same “sigmoid” activation function.
- The target labels for 1st class is [1;0;0;0] which is repeated 33 times, 2nd class has [0;1;0;0] repeating 33 times , 3rd class has [0;0;1;0] and 4th class has [0;0;0;1] each repeating 33 times.
The procedure that I followed for classification is through usage of Back-propagation learning Algorithm. Once the training is completed, I give a test input of dimension 180×1, correspondingly find the output by using the updated weights, the position corresponding to minimum mean square error between expected and obtained output is said to be the recognized word. The Classification result is approximately 92%.
Deep Belief Network Implementation
Now my intension is to perform the same task using Deep Belief Networks 1. Initially, the weight initialization is random but 1st pair of layers (input-hidden1) is treated as an RBM and the RBM update procedure is trough contrastive divergence procedure. 2. The procedure that I have followed is referred from Yoshua Bengio in “Learning Deep Architectures for AI, 2009”.
- But the changes that I have made here is instead of using “sigm” at visible layer, I have used “normpdf” following Gaussian distribution with 0 mean and unit variance . This is because for real valued speech features, its necessary to use Gaussian distribution as provided in the literatures.
- Once the first layer RBM is trained using the above algorithm, the activations at the hidden layer are fed as the input to the next layer (stacking RBM to form DBN) and is pre-trained in the same fashion as that followed by first layer. Now here are my further steps that I have implemented with lot of doubts.
- I have added the output layer with 4 nodes so as to classify 4 different classes with “softmax activation function”
- The cost function here is “Cross entropy” instead of “mean squared error” between the target (as illustrated in step 4 of back –prop algorithm) and the obtained output.
- But from the results the error increases after every epoch (iteration) and becomes “NaN” (undetermined or undefined numerical results) after a few iterations.
Here are my following queries about the above implementation
- Is the procedure adopted is correct?
- Logically I have used 2-hidden layers in the above described DBN implementation. Is that step correct? Because in the normal Back-prop algorithm I have used only one hidden layer.
- Should I add immediately a output layer after pre-training 1st RBM? I kindly request you to give suggestions in correcting the implemented topics.