I want to implement my rbm to classifications application. I want to classify MNIST dataset.

I understand that , to classify we need to add a softmax layer. can anyone elaborate more ?


1 Answer 1


You can use an RBM for classification in a few ways. Normally, you'll use RBMs as the first few layers to reduce dimensionality, then drop a neural network (or 'fully connected') layer on the top to learn the labels. Attaching a softmax to this means that the output is guaranteed to sum to 1.0, meaning it's a valid probability distribution.

With that said, a softmax isn't inherently necessary for classification, nor is a FC layer. If you want to use ONLY RBMs for classification, you can use this trick: Add the label (via concatenation) to your training data set.

So if your input was [1 0 1] and your label was [0 1], you smash them together, and your new training example becomes [1 0 1 0 1]. In the future, when you're trying to determine what label something gets, you do a similar process, take your input [0 1 1] but attach an empty output [0 0], to produce [0 1 1 0 0], then push it up and down your network. The resulting value (depending on whether or not you're using the activities or activations) will be something like [-0.1, 0.99 0.98 0.1 0.91]. Now, split off those last two 'label' bits from the output and you get [0.1 0.91], so the chances of being category two are around 91x that of cat 1. Of course, you'll note that these don't add up to 1. Madness! So if we pass that through a softmax function, we instead get [0.307, 0.693], which DOES add to one, making it a for-real probability distribution.

So why do we care about softmax in this case? Well, if you want to select an item from a given distribution rather than just picking the max all the time, you do have to be sure that it's a probability distribution. In your case, just classifying, it's safe to pick the maximum-value category.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.