3
$\begingroup$

In the dropout method of regularization, we randomly delete half of the hidden neurons, leaving the input and output layers the same.

In a theoretical sense, wouldn't the same effect occur if we just randomly assign half of the hidden neurons a weight of 0, since this would effectively null out that neuron?

$\endgroup$
1
$\begingroup$

If you set all the output weights for certain neurons to 0, yes, it is the same. But just to clarify, dropout doesn't actually delete neurons, it just deactivate them temporarily and randomly for each input. But if you deactivate a portion of your weights without regard to specific neurons (you may cut only some weights from one neuron), you end with DropConnect.

$\endgroup$
2
  • $\begingroup$ Ah, the "DropConnect" resource you pointed me to mentioned that dropout is technically setting the activations of certain neurons to 0. So DropConnect would actually be setting the weights to 0. This means that dropout deactivates entire neurons, while DropConnect only deactivates a subset of activations within a layer, not neuron-specific. Am I interpreting this correctly? $\endgroup$ Apr 3 '16 at 20:49
  • $\begingroup$ If by "a subset of activations within a layer" you mean "a subset of weights within a layer", yes, it is correct. $\endgroup$
    – rcpinto
    Apr 3 '16 at 20:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.