If you have a half page to explain dropout, how would you proceed? Which is the rationale behind this technique?
$\begingroup$
$\endgroup$
Add a comment
|
4 Answers
$\begingroup$
$\endgroup$
1
The abstract of the dropout article seems perfectly serviceable.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, Ruslan Salakhutdinov, "Dropout: A Simple Way to Prevent Neural Networks from Overfitting", Journal of Machine Learning Research, 2014.
Deep neural nets with a large number of parameters are very powerful machine learning systems. However, overfitting is a serious problem in such networks. Large networks are also slow to use, making it difficult to deal with overfitting by combining the predictions of many different large neural nets at test time. Dropout is a technique for addressing this problem. The key idea is to randomly drop units (along with their connections) from the neural network during training. This prevents units from co-adapting too much. During training, dropout samples from an exponential number of different “thinned” networks. At test time, it is easy to approximate the effect of averaging the predictions of all these thinned networks by simply using a single unthinned network that has smaller weights. This significantly reduces overfitting and gives major improvements over other regularization methods. We show that dropout improves the performance of neural networks on supervised learning tasks in vision, speech recognition, document classification and computational biology, obtaining state-of-the-art results on many benchmark data sets.
If you read the paper, you'll find a description of what co-adapting behavior means in the context of drop-out.
In a standard neural network, the derivative received by each parameter tells it how it should change so the final loss function is reduced, given what all other units are doing. Therefore, units may change in a way that they fix up the mistakes of the other units. This may lead to complex co-adaptations. This in turn leads to overfitting because these co-adaptations do not generalize to unseen data. We hypothesize that for each hidden unit, dropout prevents co-adaptation by making the presence of other hidden units unreliable. Therefore, a hidden unit cannot rely on other specific units to correct its mistakes. It must perform well in a wide variety of different contexts provided by the other hidden units. To observe this effect directly, we look at the first level features learned by neural networks trained on visual tasks with and without dropout.
-
$\begingroup$ Kudos to the original author(s), that explanation is serenely simple $\endgroup$ Jan 27 at 18:10
$\begingroup$
$\endgroup$
This answer is a follow-up to Sycorax' great answer, for readers who would like to see how dropout is implemented.
When applying dropout in artificial neural networks, one needs to compensate for the fact that at training time a portion of the neurons were deactivated. To do so, there exist two common strategies:
- Inverting the dropout during the training phase:
- Scaling the activation at test time:
The /p is moved from the training to the predicting code, where it becomes *p:
These three slides came from lecture 6 from Standford CS231n: Convolutional Neural Networks for Visual Recognition.
answered Jan 21, 2017 at 17:02
$\begingroup$
$\endgroup$
Dropout momentarily (in a batch of input data) switches off some neurons in a layer so that they do not contribute any information or learn any information during those updates, and the onus falls on other active neurons to learn harder and reduce the error.
If I have to explain drop-out to a 6-year-old, this is how: Imagine a scenario, in a classroom, a teacher asks some questions but always same two kids are answering, immediately. Now, the teacher asks them to stay quiet for some time and let other pupils participate. This way other students get to learn better. Maybe they answer wrong, but the teacher can correct them(weight updates). This way the whole class(layer) learn about a topic better.
$\begingroup$
$\endgroup$
You can look at drop-out as a prior probability on whether a feature (or latent feature in some intermediate layer) does not matter - i.e. a spike (point mass at zero = feature does not matter) and slab (flat = non-reglarized prior across the whole parameter space) prior.
Importantly, this allows you to not just regularize model fitting, but also to obtain uncertainty about inference. This is discussed in the dissertation and papers (also this) of Yarin Gal.
