I understand that when using dropout, a single neuron can be described using Bernoulli random variable and for a set of neurons it can be described as Binomial random variable

When using Dropout, we define a fixed Dropout probability p for a chosen layer and we expect that a proportional number of neurons are dropped from it.

For example, if the layer we apply Dropout to has n=1024 neurons and p=0.5, we expect that 512 get dropped. Let’s verify this statement:

enter image description here

Thus, the probability of dropping out exactly np=512 neurons is of only 0.025!

why dont we see the probability of 0.5 what am I missing here?


As you add more Bernoulli trials (neurons in this case) the probability of having a proportion within some small distance of 0.5 (say between 0.49 and 0.51) goes up, but the probability of exactly $n/2$ goes down.

The variance of $X\sim\text{Bin}(n,p)$ is $np(1-p)$. With $p=0.5$ that's $n/4$ ($\sigma_X=\sqrt{n}/2$). So as you add more neurons, the typical (RMS) distance of X from n/2 will go up and the probability of an individual outcome nearest the mean goes down. But the variance of a binomial proportion $\hat{p}=X/n$ is $p(1-p)/n$ ($\sigma_\hat{p}=0.5/\sqrt{n}$), so as you add more trials (neurons) the probability of being within a small interval of the mean proportion goes up.

This is a similar kind of error to the Gambler's fallacy; people expect that an excess of heads will be compensated by an excess of tails in the short run, so that the number of heads will be pushed toward $\frac12 n$. But there's no such effect. In $n$ tosses of a fair coin, let's call the number of heads $N_H$, (similarly for $N_T$, so $N_H+N_T=n$), and let the observed proportion of heads be $p_H$, where $p_H=N_H/n$.

What happens to $E|N_H-N_T|$ as $n$ grows? It doesn't go to zero (as the fallacious gambler reasons) -- as $n\to\infty$, it actually goes to infinity (it grows as $n^\frac12$). However, $E|p_H-p_T|$ does go to zero (as indeed it must, because of the weak law of large numbers).

  • $\begingroup$ When did we all suddenly start talking taking about deep learning..? $\endgroup$
    – Tim
    Mar 3 '19 at 6:51
  • 1
    $\begingroup$ When a lot of the questions started being stats questions? $\endgroup$
    – Glen_b
    Mar 3 '19 at 6:54

When $n$ gets large, individual probabilities gets smaller, because options are many. For example, if we are dealing with lots of neurons, let's say $n=10^{30}$, $P(Y=5.10^{29})$ will be much much smaller, right? Because, when you have an experiment of such nature, probability is also spreading across the neighbourhood of $np$. In your case, the probability of $Y=511,513 \ ...$ have also some volume. So, you'll consider ranges, not individual probabilities. When you select some from $n$ elements, it won't be that hard to get a number around $np \pm \sqrt{np(1-p)}$, i.e. one deviation away from the mean. Specifically, for one $\sigma=16$, you have a high probability: $P(496\leq Y\leq 528)\approx 0.7$. If we consider $2\sigma$, $P(480\leq Y \leq 544)\approx0.96$. So, it's highly likely that you end up with a number in $[480,544]$. In any case, don't confuse the expected value with individual scores.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.