3
$\begingroup$

In the dropout method of regularization, we randomly delete half of the hidden neurons, leaving the input and output layers the same.

In a theoretical sense, wouldn't the same effect occur if we just randomly assign half of the hidden neurons a weight of 0, since this would effectively null out that neuron?

$\endgroup$

1 Answer 1

1
$\begingroup$

If you set all the output weights for certain neurons to 0, yes, it is the same. But just to clarify, dropout doesn't actually delete neurons, it just deactivate them temporarily and randomly for each input. But if you deactivate a portion of your weights without regard to specific neurons (you may cut only some weights from one neuron), you end with DropConnect.

$\endgroup$
3
  • $\begingroup$ Ah, the "DropConnect" resource you pointed me to mentioned that dropout is technically setting the activations of certain neurons to 0. So DropConnect would actually be setting the weights to 0. This means that dropout deactivates entire neurons, while DropConnect only deactivates a subset of activations within a layer, not neuron-specific. Am I interpreting this correctly? $\endgroup$ Commented Apr 3, 2016 at 20:49
  • $\begingroup$ If by "a subset of activations within a layer" you mean "a subset of weights within a layer", yes, it is correct. $\endgroup$
    – rcpinto
    Commented Apr 3, 2016 at 20:53
  • $\begingroup$ @rcpinto Still I am confused. If some neuron produces zero output but is not dropped out, how the subsequent neurons differentiate between this neuron (non-random output 0) and a dropped out neuron (random output 0)? $\endgroup$
    – ado sar
    Commented Jun 25 at 19:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.