4
$\begingroup$

I'm learning about dropout from these sources:

At test time, the trained weights are scaled by a factor of $p$, to replace them with their mean. According to the original paper this is kind of like an approximation to an ensemble of the $2^n$ possible dropout models that "works well in practice".

It's an approximation because the network isn't a linear function of its weights.

Is there any justification for this approximation?

It's not even clear to me that dropout is a consistent estimator.

$\endgroup$
5
$\begingroup$

From Goodfellow, Bengio and Courville's book, Deep Learning:

A key insight (Hinton et al., 2012c) involved in dropout is that we can approximate $p_\mathrm{ensemble}$ by evaluating $p(y|\mathbf{x})$ in one model: the model with all units, but with the weights going out of unit $i$ multiplied by the probability of including unit $i$. The motivation for this modification is to capture the right expected value of the output from that unit. We call this approach the weight scaling inference rule. There is not yet any theoretical argument for the accuracy of this approximate inference rule in deep nonlinear networks, but empirically it performs very well.

(Emphasis mine)

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.