# How many neurons are actually dropped when using dropout?

I understand that when using dropout, a single neuron can be described using Bernoulli random variable and for a set of neurons it can be described as Binomial random variable

When using Dropout, we define a fixed Dropout probability p for a chosen layer and we expect that a proportional number of neurons are dropped from it.

For example, if the layer we apply Dropout to has n=1024 neurons and p=0.5, we expect that 512 get dropped. Let’s verify this statement:

Thus, the probability of dropping out exactly np=512 neurons is of only 0.025!

why dont we see the probability of 0.5 what am I missing here?

As you add more Bernoulli trials (neurons in this case) the probability of having a proportion within some small distance of 0.5 (say between 0.49 and 0.51) goes up, but the probability of exactly $$n/2$$ goes down.

The variance of $$X\sim\text{Bin}(n,p)$$ is $$np(1-p)$$. With $$p=0.5$$ that's $$n/4$$ ($$\sigma_X=\sqrt{n}/2$$). So as you add more neurons, the typical (RMS) distance of X from n/2 will go up and the probability of an individual outcome nearest the mean goes down. But the variance of a binomial proportion $$\hat{p}=X/n$$ is $$p(1-p)/n$$ ($$\sigma_\hat{p}=0.5/\sqrt{n}$$), so as you add more trials (neurons) the probability of being within a small interval of the mean proportion goes up.

This is a similar kind of error to the Gambler's fallacy; people expect that an excess of heads will be compensated by an excess of tails in the short run, so that the number of heads will be pushed toward $$\frac12 n$$. But there's no such effect. In $$n$$ tosses of a fair coin, let's call the number of heads $$N_H$$, (similarly for $$N_T$$, so $$N_H+N_T=n$$), and let the observed proportion of heads be $$p_H$$, where $$p_H=N_H/n$$.

What happens to $$E|N_H-N_T|$$ as $$n$$ grows? It doesn't go to zero (as the fallacious gambler reasons) -- as $$n\to\infty$$, it actually goes to infinity (it grows as $$n^\frac12$$). However, $$E|p_H-p_T|$$ does go to zero (as indeed it must, because of the weak law of large numbers).

• When did we all suddenly start talking taking about deep learning..?
– Tim
Mar 3 '19 at 6:51
• When a lot of the questions started being stats questions? Mar 3 '19 at 6:54

When $$n$$ gets large, individual probabilities gets smaller, because options are many. For example, if we are dealing with lots of neurons, let's say $$n=10^{30}$$, $$P(Y=5.10^{29})$$ will be much much smaller, right? Because, when you have an experiment of such nature, probability is also spreading across the neighbourhood of $$np$$. In your case, the probability of $$Y=511,513 \ ...$$ have also some volume. So, you'll consider ranges, not individual probabilities. When you select some from $$n$$ elements, it won't be that hard to get a number around $$np \pm \sqrt{np(1-p)}$$, i.e. one deviation away from the mean. Specifically, for one $$\sigma=16$$, you have a high probability: $$P(496\leq Y\leq 528)\approx 0.7$$. If we consider $$2\sigma$$, $$P(480\leq Y \leq 544)\approx0.96$$. So, it's highly likely that you end up with a number in $$[480,544]$$. In any case, don't confuse the expected value with individual scores.