1
$\begingroup$

I have read some general statements about the usefulness of Dropout but the issue is still very vague to me. It is always said that it prohibits co-adaptability of neurons, but why it should be a bad thing? We build the network of neurons and train all of them together, otherwise what is the point in a connected network? And what we expect is that one specific neuron is activated by a specific input. Dropout exactly opposes this. When we randomly set on and off the neurons, how we do expect that it learn something?

$\endgroup$
  • $\begingroup$ Thanks for reply, but what is the point in discussing symmetry here? I can not relate it to my question. $\endgroup$ – Shahriar49 Apr 4 '18 at 23:39
1
$\begingroup$

There are quite a few questions asked in this single question. I would try to answer as much as possible in a concise way.

We build the network of neurons and train all of them together, otherwise what is the point in a connected network? True, we train them together. Though this is not the correct terminology to use, a network of neurons is called a neural network (again, not the exact definition, but I am trying to match your description) and we train a neural network. When we train a neural network, the weights associated with these connections between different layers of neural nets are "learnt". These weights can take up any value, it could be very small or larger than others.

Now, dropout doesn't set the weight to zero completely. Remember, dropout is applied on a whole layer and has a probability associated and can keep changing from batch to batch*. So when we feed our first batch, $n_1, n_2, n_3$ might have there outputs zero, but other neurons still output values. [$n_1, n_2, n_3, n_4,.. n_{10}$ are neurons in some layer]. For next iteration, we might have $n_2, n_4, n_5$ with their outputs zero. The network is forced to learn with such perturbations and eventually adapts to learn even in absence of some data.

As you pointed out, a neuron might learn to fire only when it sees a particular value. We would like to avoid neurons becoming specialised since this would lead to poor generalisation. It might not perform well against unseen data. Dropout is, thus, essentially a type of regularization technique and is used to avoid overfitting. It has been hard to explain mathematically why it works, but it is one of the successful techniques.

$\endgroup$
  • $\begingroup$ Thank you very much for your reply, but I still have problems understanding it. My first question is that they say co-adaptability of neurons are bad.Why? I don't see anything harmful in that. The second question: it is claimed that network neurons are learned to be activated by specific features in input (such as corners, colors, etc). This is critical specially in deep networks and it is always claimed as their main benefit. Dropout will ruin and disrupt this systematic learning by randomly killing neurons. Why we should do such a thing? $\endgroup$ – Shahriar49 Apr 4 '18 at 23:48
  • $\begingroup$ 1. Could you expand on what do you mean/understand by co-adaptability? I would be able tp better explain in terms of that then 2. Deep learning techniques are effective in idenitfying features such as corners, colors etc. True. But in order to learn these hidden features, the network needs to be robust against different types of input. Dropout doesn't kill a neuron. It just switches it off for meanwhile, so that other neurons can learn too. Imagine scenario: In a class a teacher asks a question, same student always answers. So they encourage someone else to answer and this kid sits quietly. $\endgroup$ – Shubham Agrawal Apr 9 '18 at 22:04
  • $\begingroup$ Co-adaptability was not my word. I saw it in some other posts and my understanding is that they mean the neurons are learning all together, i.e. learning on any neuron is affected by what the others learn and will affect the other neurons as well. About your example: if we have smart students that always answer, why do we need dumb students? (we don't have any social responsibility towards neurons :) All I say is that the neurons' activation is based on their feature detection. Dropout disrupts this process. Robustness doesn't mean that we should have every neuron learn every feature. $\endgroup$ – Shahriar49 Apr 10 '18 at 13:15
  • $\begingroup$ Feature detection is not by a neuron, it is collective effort of the whole layer. EDIT: Also: "All I say is that the neurons' activation is based on their feature detection." Wrong. $\endgroup$ – Shubham Agrawal Apr 10 '18 at 16:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.