# Variance introduced by dropout

I just read this paper Sampling-free Epistemic Uncertainty Estimation Using Approximated Variance Propagation and stumbled upon two points at the end of section 3.1 that I don't understand. Here, the authors state that:

For the special case of the first noise layer in the network, we can either model the input noise from prior knowledge (i.e. sensor noise) or simplify it by assuming zero noise. In the latter case, the resulting covariance matrix will be diagonal given independent noise, resulting in: $$\Sigma_{\vec{X}^i} = \text{diag}(\sigma_1^2, \cdots, > \sigma_n^2)$$ where $$n$$ is the dimensionality of the activation vector and $$\sigma_i^2$$ is the variance of activation $$i$$.

Here I don't understand the following point: they say that you can assume zero noise but the resulting covariance matrix will be diagonal. How is this possible? If there is a signal (in this case the random vector $$\vec{X}^i$$) with zero noise, how can the covariance matrix have any nonzero elements?

The variance introduced by the regular dropout, which follows a Bernoulli distribution, is given by $$p(1 − p)a_i^2$$, with $$p$$ defining the dropout rate and $$a_i$$ the mean activation of node $$i$$.

I find this part of the section even more confusing. How is it, that the variance introduced by dropout is $$p(1 − p)a_i^2$$ and not just $$p(1 − p)$$? Where does this expression for the variance come from? Why is the variance squared by the squared mean activation? And where do they get the mean activation $$a_i$$ from in the first place?

In this section, they're discussing the behavior of a 'noise layer', which combines its input $$X$$ with random noise $$Z$$. For the first noise layer in the network, $$X \in \mathbb{R}^n$$ is assumed to be a fixed vector. $$Z$$ is an $$n$$-dimensional random vector with i.i.d. entries. For an additive noise layer, the activations $$Y$$ are produced by adding the noise to the input:

$$Y = X + Z$$

Alternatively, the noise layer may implement dropout. In this case, the entries of $$Z$$ are i.i.d. Bernoulli (with dropout rate $$p$$), and the activations are given by the Hadamard product of the input and noise vectors (i.e. by multiplying them element-wise):

$$Y = X \circ Z$$

Here I don't understand the following point: they say that you can assume zero noise but the resulting covariance matrix will be diagonal. How is this possible? If there is a signal (in this case the random vector $$\vec{X}^i$$) with zero noise, how can the covariance matrix have any nonzero elements?

They're referring to the covariance matrix of the noise layer activations $$Y$$, not the input $$X$$. For the first additive noise layer, since $$Y=X+Z$$ and $$X$$ is assumed constant, the covariance matrix of $$Y$$ is equal to the covariance matrix of $$Z$$. And, since $$Z$$ has i.i.d. entries, the covariance matrix is diagonal.

I find this part of the section even more confusing. How is it, that the variance introduced by dropout is $$p(1 − p)a_i^2$$ and not just $$p(1 − p)$$? Where does this expression for the variance come from? Why is the variance squared by the squared mean activation? And where do they get the mean activation $$a_i$$ from in the first place?

The wording of the paper seems slightly ambiguous to me, but I think we can understand it as follows. Consider the first dropout layer. The activation of the $$i$$th unit in the noise layer is:

$$Y_i = X_i Z_i$$

Suppose that that $$X_i=a_i$$ is the activation of the $$i$$th input unit (which is assumed constant, as above). Therefore:

$$\operatorname{Var}(Y_i) = \operatorname{Var}(a_i Z_i) = a_i^2 \operatorname{Var}(Z_i)$$

Since $$Z_i$$ is Bernoulli as above, $$\operatorname{Var}(Z_i) = p(1-p)$$ and:

$$\operatorname{Var}(Y_i) = p (1-p) a_i^2$$

• Is it possible to derive the expression for $Var(Y_i)$ using $\Sigma^{(i)} = J \Sigma_{\vec{Z}} J^T$ where $J(\vec{Y}^{(i-1)})$ is a diagonal matrix with diagonal elements $X_i$ and $\Sigma_{\vec{Z}}$ also a diagonal matrix with elements $p(1-p)$? Commented Jul 1, 2021 at 13:34