First I'll sum up my intuitive (beginner) understanding of his deep-learning architecture. A short summary can be listened to on Coursera in the 5 minute video.

We start with several layers of autoencoder networks in an unsupervised fashion, to arrive to a good representation/compression of the problem itself. (Mind, compression is almost equivalent to intelligence, see for example the Hutter Prize).

Then we add the actual labels to the data, i.e. we start a gradient algorithm to tune the weights of our pre-trained network to arrive to a good prediction, in a supervised fashion. This is helpful because we started gradient training from a network which is already good at representing the problem, so we have less chance to get stuck in a simplistic local optima as simple multilayer networks do.

Why does using stochastic activation functions help in the whole process? We could do the same (with the same intuition above, at least) with deterministic autoencoder networks too, then applying gradient.


Intuitively it's because you don't want your auto-encoder to learn what is exactly in your training dataset, but you want it to learn what it is alike.

Failling to use a strategy to learn what is around your training dataset will lead to overfitting.

  • $\begingroup$ I suspected something like this. Although there are a number of other strategies to avoid overfitting: less weights, L1, L2 regularization (penalty on weights), resampling, out of sample validation, etc... why is the stochastic approach better? $\endgroup$ – Mark Horvath Aug 1 '15 at 16:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.