I have come against a small issue when implementing some deep learning techniques.

Sometimes, if one is implementing a classic Bernoulli-Bernoulli Restricted Boltzmann Machine (RBM), the sampling functions (i.e. particularly the function that samples the hidden units given the visible units, $p(h|v)$) requires using a binomial distribution.

However, I have found in my code that it's entirely possible to generate a row in matrix using the binomial distribution (using the visible means computed from a propagate down function as the p value and $n = 1$) with all 0's or all 1's. This can lead to certain numerical problems (akin to numerical analysis issues such as NaN's appearing in the computation given that later matrix operations use division or Log) in later computations.

My question is:

What is done in practice (obviously most deep learning papers to my knowledge, as well as tutorial sites, do not address these small technical issues) in such cases?

A binomial distribution sampling could generate a row in a matrix with all 0's. What I've done is decided that I need to check for such cases and add some very slight noise to one of these 0's (let's say I write a small sub-routine to check that a matrix doesn't contain any rows of all 0's and will then randomly select a cell in any such bad-case-row and then, insert a noise value such as $10^{-2}$).

I know I'm giving up some accuracy in introducing some slight noise to nudge the computation in the right direction, but once I do I get correct predictions/label constructions (for all cases) so it appears my code works fine. So while I believe I may have found one technical workaround, I would like to know what is actually done to handle this scenario?

Update: Is this a technical challenge no one really encounters? Is there something special about Python (and usu. Theano) that magically ensures there are never any sample rows in a sampled matrix that are all 0's or all 1's? even though a binomial sampling is used)?


Answer: Well, it turns out the answer is simple: after I spent more time working with stochastic ANN architectures I found that it all has to do with the mean or probability of the unit turning on. When a hidden layer unit has a lower probability of turning on (such as a value under 0.5), it is more likely to be 0.0 than 1.0 when you use the unit probability to sample from the Bernoulli distribution. This means that if hidden unit activities given some input vector are generally all off as opposed to on, you can get hidden vector representations of all 0.0's--i.e. a meaningless/useless transformation of the data if you intend on building deep networks.

So what have I learned that I can impart to those who read my posts regarding DL? Make sure your network is learning well during training: if you want to go with stochastic nets (as opposed to mean-field nets, where you can use the probabilities directly), pay attention to the probabilities of the hidden units and see what hidden layer "binary codes" your model is learning at each layer (beware the all 0.0's layer). I found that my dirty trick of adding noise to these "empty" vectors (the one in the original post) does stop the NaN problem from cropping up but can potentially hurt the net's performance.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.