# What is the probability that sample variance decreases by adding random Gaussian noise to the variable?

If we assume WLOG that our variable X has mean zero (mean-centered), then this can be stated

$$Pr \bigg(\sum x^2 > \sum (x-n)^2 \bigg)$$

for some random variable $$n$$ distributed under $$N \sim N(0, \sigma_N^2)$$.

Expanding the quadratic expression and arranging terms gives

$$Pr \bigg( \sum (2 x n ) > \sum ( n^2 ) \bigg)$$

Under what conditions is it possible to derive this probability? In this case, I know from domain context that var(X) > var(N), but var(N) is not negligible.

This feels strange because, intuitively, adding noise should be far more likely to increase variance, given the nature of squared error terms. Looking at this equation, the LHS has expected value = 0 due to independence of X and N, while the RHS has expected value = $$\sigma_N^2 \cdot n_{obs}$$. So the greater $$\sigma_N^2$$ becomes, the less likely this value, but I'm wondering how to bound or quantify this changing probability.

I'm interested to at least see what can be derived by assuming a Gaussian distribution for X, but I'm more interested to see if any generalities (e.g. lower or upper bounds) can be made overall.

Given that we're dealing with summations, is it reasonable to invoke the CLT on the LHS, and claim that var(XN) = var(X)var(N) given that X and N both have mean zero? What are the cautions against invoking the CLT in this case?

• To obtain an answer, you need to specify the joint distribution of $x$ and $n$. Do you have any particular distribution(s) in mind? But this doesn't really seem like the right way to approach the question. If you're truly interested in the issue of the effect of adding noise on variance, have you considered the fact that when $x$ and $n$ are independent, $\operatorname{Var}(x+n)=\operatorname{Var}(x)+\operatorname{Var}(n) \gt \operatorname{Var}(x)$?
– whuber
Oct 31 '18 at 18:51
• I figured that exact answers are unattainable, but I was wondering if there's some way to approximate or bound the terms, through something like Chebyshev's Inequality. The theoretical variance is certainly greater, but I'm wondering what the effects on sample values are. Oct 31 '18 at 18:58
• I've thrown a couple more musings into the original post. Given that we're dealing with a summation and a product of independent variables, can we utilize the CLT to get some asymptotic results? Oct 31 '18 at 19:08

OK I think I've got it, so I'll post this as an answer.

Taking it up from the desired probability $$Pr \bigg( \sum(2xn) > \sum (n^2) \bigg)$$:

$$= Pr \bigg( \displaystyle\sum_{i=0}^m(2x_i n_i - n_i^2) > 0 \bigg)$$ for $$m$$ total sample observations

$$= Pr \bigg( \displaystyle\sum_{i=0}^m A_i > 0 \bigg)$$ for random variable $$A_i = 2x_i n_i - n_i^2$$

By the CLT, this LHS sum should converge to a normal distribution with mean $$m \cdot E[A_i]$$ and variance $$m \cdot Var[A_i]$$.

So we can work out the mean and variance of random variable $$A$$:

(1) $$E[A]$$

$$\qquad \cdot E[A] = E[ 2XN - N^2 ] = 2 E[X] E[N] - E[N^2] = - E[N^2]$$ since $$X, N$$ are independent with mean zero

$$\qquad \cdot N^2 \sim \sigma_N^2 \chi^2(1)$$ as the square of a random variable, so $$E[N^2] = \sigma_N^2$$

$$\qquad \therefore E[A] = - \sigma_N^2$$

(2) $$Var[A]$$

$$\qquad \cdot Var[A] = Var[2XN - N^2] = Var[2XN] + Var[N^2] + 2Cov[2XN, -N^2]$$

$$\qquad \cdot Var[2XN] = 4 Var[X] Var[N] = 4 \sigma_X^2 \sigma_N^2$$ given that X,N are independent with mean zero

$$\qquad \cdot Var[N^2] = Var[\sigma_N^2 \chi^2(1)] = 2 \sigma_N^4$$

$$\qquad \cdot Cov[2XN, -N^2] = E\bigg[ \Big(2XN - E[2XN]\Big)\Big(N^2 - E[N^2]\Big)\bigg]$$

$$\qquad \qquad = E\bigg[ 2XN \cdot \Big(N^2 - \sigma_N^2\Big)\bigg]$$

$$\qquad \qquad = E\bigg[ 2XN^3 - 2XN\sigma_N^2\bigg]$$

$$\qquad \qquad = 0$$

$$\qquad \therefore Var[A] = 4\sigma_X^2 \sigma_N^2 + 2 \sigma_N^4$$

$$\therefore Pr \bigg( \displaystyle\sum_{i=0}^m A_i > 0 \bigg)$$ can be written as

$$= Pr \bigg( B > 0 \bigg) =$$ for random variable $$B \sim N\Big(- m \cdot \sigma_N^2, \quad m \cdot 4\sigma_X^2 \sigma_N^2 + 2m \sigma_N^4 \Big)$$

$$= Pr \bigg( B_0 > \frac{\sqrt{m} \cdot \sigma_N^2 }{\sqrt{4\sigma_X^2 \sigma_N^2 + 2 \sigma_N^4}} \bigg)$$ for standard normal random variable $$B_0 \sim N(0,1)$$

• You seem to have neglected the fact that $A$ and $B$ are not independent. This cannot be ignored.
– whuber
Oct 31 '18 at 20:30
• Hmmm. Given that sample mean and variance are independent, can it be demonstrated that A and B might be independent? Given that A is somewhat akin to the (randomly weighted) sample mean, and B is by definition the sample variance. Oct 31 '18 at 20:40
• I'm afraid not: take a look at their definitions. $A=xn$ and $B=n(n)$ are very strongly interconnected and will remain so no matter the sample size. You can tackle this by considering $(2x-n)$ conditional on the sign of $n.$
– whuber
Oct 31 '18 at 20:43
• I'm not convinced by that logic, given that A and B are actually summations that are very similar to sample mean and variance [which certainly don't appear independent at first glance], but I suppose the lack of sample-mean subtraction in my "variance term" B could lend itself to bias. I'll take another look under your suggestion of $n(2x-n)$ though. Oct 31 '18 at 20:56
• You don't have to be convinced: but it demonstrates that you have to convince your readers that $A$ and $B$ are asymptotically independent.
– whuber
Oct 31 '18 at 21:06

If $$var(X) >> var(N)$$, then $$\sum n^2$$ is negligible, and we can simply take the probability of $$\sum 2xn >0$$. This is then 50%. Looking at it geometrically, we can treat $$u= [x_1,x_2, ...]$$ and $$v = [n_1,n_2,...]$$ as vectors. We want the probability that $$|u+v|>|u|$$. This is equivalent drawing a hypersphere with radius $$u$$ and dimension $$k$$, where $$k$$ is the number of observations (normally that would be called $$n$$, but you've represented the noise with that variable), then asking what the probability of $$u+v$$ being inside that hypersphere is. To find that, we draw a hypersphere centered at the head of $$u$$ with radius $$v$$, and calculate what percentage of that smaller hypersphere is inside the larger hypersphere. For $$|u| >>|v|$$, the larger hypersphere cuts through the smaller circle almost like a straight hyperplane, so the probability is close to half. As $$|v|$$ gets larger, the larger hypersphere curves more, and a smaller portion of the smaller hypersphere is inside the larger hypersphere.

• Yeah, I noticed this too - - but in practice, with my data, I'm not getting a value of 1/2, so this assumption of mine about "negligibility" was false. I'm going to remove that part from the OP. Oct 31 '18 at 23:01