What is the probability that sample variance decreases by adding random Gaussian noise to the variable?

Question

If we assume WLOG that our variable X has mean zero (mean-centered), then this can be stated

$Pr \bigg(\sum x^2 > \sum (x-n)^2 \bigg)$

for some random variable $n$ distributed under $N \sim N(0, \sigma_N^2) $.

Expanding the quadratic expression and arranging terms gives

$Pr \bigg( \sum (2 x n ) > \sum ( n^2 ) \bigg)$

Under what conditions is it possible to derive this probability? In this case, I know from domain context that var(X) > var(N), but var(N) is not negligible.

This feels strange because, intuitively, adding noise should be far more likely to increase variance, given the nature of squared error terms. Looking at this equation, the LHS has expected value = 0 due to independence of X and N, while the RHS has expected value = $\sigma_N^2 \cdot n_{obs}$. So the greater $\sigma_N^2$ becomes, the less likely this value, but I'm wondering how to bound or quantify this changing probability.

I'm interested to at least see what can be derived by assuming a Gaussian distribution for X, but I'm more interested to see if any generalities (e.g. lower or upper bounds) can be made overall.

Given that we're dealing with summations, is it reasonable to invoke the CLT on the LHS, and claim that var(XN) = var(X)var(N) given that X and N both have mean zero? What are the cautions against invoking the CLT in this case?

Answer 1

OK I think I've got it, so I'll post this as an answer.

Taking it up from the desired probability $Pr \bigg( \sum(2xn) > \sum (n^2) \bigg)$:

$ = Pr \bigg( \displaystyle\sum_{i=0}^m(2x_i n_i - n_i^2) > 0 \bigg)$ for $m$ total sample observations

$ = Pr \bigg( \displaystyle\sum_{i=0}^m A_i > 0 \bigg)$ for random variable $A_i = 2x_i n_i - n_i^2$

By the CLT, this LHS sum should converge to a normal distribution with mean $m \cdot E[A_i]$ and variance $m \cdot Var[A_i]$.

So we can work out the mean and variance of random variable $A$:

(1) $E[A]$

$\qquad \cdot E[A] = E[ 2XN - N^2 ] = 2 E[X] E[N] - E[N^2] = - E[N^2]$ since $X, N$ are independent with mean zero

$\qquad \cdot N^2 \sim \sigma_N^2 \chi^2(1)$ as the square of a random variable, so $E[N^2] = \sigma_N^2$

$\qquad \therefore E[A] = - \sigma_N^2$

(2) $Var[A]$

$\qquad \cdot Var[A] = Var[2XN - N^2] = Var[2XN] + Var[N^2] + 2Cov[2XN, -N^2]$

$\qquad \cdot Var[2XN] = 4 Var[X] Var[N] = 4 \sigma_X^2 \sigma_N^2$ given that X,N are independent with mean zero

$\qquad \cdot Var[N^2] = Var[\sigma_N^2 \chi^2(1)] = 2 \sigma_N^4$

$\qquad \cdot Cov[2XN, -N^2] = E\bigg[ \Big(2XN - E[2XN]\Big)\Big(N^2 - E[N^2]\Big)\bigg]$

$\qquad \qquad = E\bigg[ 2XN \cdot \Big(N^2 - \sigma_N^2\Big)\bigg]$

$\qquad \qquad = E\bigg[ 2XN^3 - 2XN\sigma_N^2\bigg]$

$\qquad \qquad = 0$

$\qquad \therefore Var[A] = 4\sigma_X^2 \sigma_N^2 + 2 \sigma_N^4$

$\therefore Pr \bigg( \displaystyle\sum_{i=0}^m A_i > 0 \bigg) $ can be written as

$= Pr \bigg( B > 0 \bigg) = $ for random variable $B \sim N\Big(- m \cdot \sigma_N^2, \quad m \cdot 4\sigma_X^2 \sigma_N^2 + 2m \sigma_N^4 \Big)$

$= Pr \bigg( B_0 > \frac{\sqrt{m} \cdot \sigma_N^2 }{\sqrt{4\sigma_X^2 \sigma_N^2 + 2 \sigma_N^4}} \bigg)$ for standard normal random variable $B_0 \sim N(0,1)$

Answer 2

If $var(X) >> var(N)$, then $\sum n^2$ is negligible, and we can simply take the probability of $\sum 2xn >0$. This is then 50%. Looking at it geometrically, we can treat $u= [x_1,x_2, ...]$ and $v = [n_1,n_2,...]$ as vectors. We want the probability that $|u+v|>|u|$. This is equivalent drawing a hypersphere with radius $u$ and dimension $k$, where $k$ is the number of observations (normally that would be called $n$, but you've represented the noise with that variable), then asking what the probability of $u+v$ being inside that hypersphere is. To find that, we draw a hypersphere centered at the head of $u$ with radius $v$, and calculate what percentage of that smaller hypersphere is inside the larger hypersphere. For $|u| >>|v|$, the larger hypersphere cuts through the smaller circle almost like a straight hyperplane, so the probability is close to half. As $|v|$ gets larger, the larger hypersphere curves more, and a smaller portion of the smaller hypersphere is inside the larger hypersphere.