Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
I'm learning about dropout from these sources:
At test time, the trained weights are scaled by a factor of $p$, to replace them with their mean. According to the original paper this is kind of like an approximation to an ensemble of the $2^n$ possible dropout models that "works well in practice".
It's an approximation because the network isn't a linear function of its weights.
Is there any justification for this approximation?
It's not even clear to me that dropout is a consistent estimator.
From Goodfellow, Bengio and Courville's book, Deep Learning:
A key insight (Hinton et al., 2012c) involved in dropout is that we can approximate $p_\mathrm{ensemble}$ by evaluating $p(y|\mathbf{x})$ in one model: the model with all units, but with the weights going out of unit $i$ multiplied by the probability of including unit $i$. The motivation for this modification is to capture the right expected value of the output from that unit. We call this approach the weight scaling inference rule. There is not yet any theoretical argument for the accuracy of this approximate inference rule in deep nonlinear networks, but empirically it performs very well.
(Emphasis mine)
