What is the probability that sample variance decreases by adding random Gaussian noise to the variable? - Cross Validated most recent 30 from stats.stackexchange.com 2019-06-17T15:21:20Z https://stats.stackexchange.com/feeds/question/374691 http://www.creativecommons.org/licenses/by-sa/3.0/rdf https://stats.stackexchange.com/q/374691 2 What is the probability that sample variance decreases by adding random Gaussian noise to the variable? Joey F. https://stats.stackexchange.com/users/200850 2018-10-31T18:37:32Z 2018-10-31T23:02:51Z <p>If we assume WLOG that our variable X has mean zero (mean-centered), then this can be stated</p> <p><span class="math-container">$Pr \bigg(\sum x^2 &gt; \sum (x-n)^2 \bigg)$</span></p> <p>for some random variable <span class="math-container">$n$</span> distributed under <span class="math-container">$N \sim N(0, \sigma_N^2)$</span>. </p> <p>Expanding the quadratic expression and arranging terms gives</p> <p><span class="math-container">$Pr \bigg( \sum (2 x n ) &gt; \sum ( n^2 ) \bigg)$</span></p> <p>Under what conditions is it possible to derive this probability? In this case, I know from domain context that var(X) > var(N), but var(N) is not negligible. </p> <p>This feels strange because, intuitively, adding noise should be far more likely to increase variance, given the nature of squared error terms. Looking at this equation, the LHS has expected value = 0 due to independence of X and N, while the RHS has expected value = <span class="math-container">$\sigma_N^2 \cdot n_{obs}$</span>. So the greater <span class="math-container">$\sigma_N^2$</span> becomes, the less likely this value, but I'm wondering how to bound or quantify this changing probability.</p> <p>I'm interested to at least see what can be derived by assuming a Gaussian distribution for X, but I'm more interested to see if any generalities (e.g. lower or upper bounds) can be made overall. </p> <p>Given that we're dealing with summations, is it reasonable to invoke the CLT on the LHS, and claim that var(XN) = var(X)var(N) given that X and N both have mean zero? What are the cautions against invoking the CLT in this case?</p> https://stats.stackexchange.com/questions/374691/-/374709#374709 1 Answer by Joey F. for What is the probability that sample variance decreases by adding random Gaussian noise to the variable? Joey F. https://stats.stackexchange.com/users/200850 2018-10-31T19:42:14Z 2018-10-31T22:34:27Z <p>OK I think I've got it, so I'll post this as an answer.</p> <p>Taking it up from the desired probability <span class="math-container">$Pr \bigg( \sum(2xn) &gt; \sum (n^2) \bigg)$</span>:</p> <p><span class="math-container">$= Pr \bigg( \displaystyle\sum_{i=0}^m(2x_i n_i - n_i^2) &gt; 0 \bigg)$</span> for <span class="math-container">$m$</span> total sample observations</p> <p><span class="math-container">$= Pr \bigg( \displaystyle\sum_{i=0}^m A_i &gt; 0 \bigg)$</span> for random variable <span class="math-container">$A_i = 2x_i n_i - n_i^2$</span></p> <p>By the CLT, this LHS sum should converge to a normal distribution with mean <span class="math-container">$m \cdot E[A_i]$</span> and variance <span class="math-container">$m \cdot Var[A_i]$</span>.</p> <p>So we can work out the mean and variance of random variable <span class="math-container">$A$</span>:</p> <p>(1) <span class="math-container">$E[A]$</span></p> <p><span class="math-container">$\qquad \cdot E[A] = E[ 2XN - N^2 ] = 2 E[X] E[N] - E[N^2] = - E[N^2]$</span> since <span class="math-container">$X, N$</span> are independent with mean zero</p> <p><span class="math-container">$\qquad \cdot N^2 \sim \sigma_N^2 \chi^2(1)$</span> as the square of a random variable, so <span class="math-container">$E[N^2] = \sigma_N^2$</span></p> <p><span class="math-container">$\qquad \therefore E[A] = - \sigma_N^2$</span></p> <p>(2) <span class="math-container">$Var[A]$</span></p> <p><span class="math-container">$\qquad \cdot Var[A] = Var[2XN - N^2] = Var[2XN] + Var[N^2] + 2Cov[2XN, -N^2]$</span></p> <p><span class="math-container">$\qquad \cdot Var[2XN] = 4 Var[X] Var[N] = 4 \sigma_X^2 \sigma_N^2$</span> given that X,N are independent with mean zero</p> <p><span class="math-container">$\qquad \cdot Var[N^2] = Var[\sigma_N^2 \chi^2(1)] = 2 \sigma_N^4$</span></p> <p><span class="math-container">$\qquad \cdot Cov[2XN, -N^2] = E\bigg[ \Big(2XN - E[2XN]\Big)\Big(N^2 - E[N^2]\Big)\bigg]$</span></p> <p><span class="math-container">$\qquad \qquad = E\bigg[ 2XN \cdot \Big(N^2 - \sigma_N^2\Big)\bigg]$</span></p> <p><span class="math-container">$\qquad \qquad = E\bigg[ 2XN^3 - 2XN\sigma_N^2\bigg]$</span></p> <p><span class="math-container">$\qquad \qquad = 0$</span></p> <p><span class="math-container">$\qquad \therefore Var[A] = 4\sigma_X^2 \sigma_N^2 + 2 \sigma_N^4$</span> </p> <p><span class="math-container">$\therefore Pr \bigg( \displaystyle\sum_{i=0}^m A_i &gt; 0 \bigg)$</span> can be written as </p> <p><span class="math-container">$= Pr \bigg( B &gt; 0 \bigg) =$</span> for random variable <span class="math-container">$B \sim N\Big(- m \cdot \sigma_N^2, \quad m \cdot 4\sigma_X^2 \sigma_N^2 + 2m \sigma_N^4 \Big)$</span></p> <p><span class="math-container">$= Pr \bigg( B_0 &gt; \frac{\sqrt{m} \cdot \sigma_N^2 }{\sqrt{4\sigma_X^2 \sigma_N^2 + 2 \sigma_N^4}} \bigg)$</span> for standard normal random variable <span class="math-container">$B_0 \sim N(0,1)$</span></p> https://stats.stackexchange.com/questions/374691/-/374737#374737 0 Answer by Acccumulation for What is the probability that sample variance decreases by adding random Gaussian noise to the variable? Acccumulation https://stats.stackexchange.com/users/179204 2018-10-31T22:44:50Z 2018-10-31T22:44:50Z <p>If <span class="math-container">$var(X) &gt;&gt; var(N)$</span>, then <span class="math-container">$\sum n^2$</span> is negligible, and we can simply take the probability of <span class="math-container">$\sum 2xn &gt;0$</span>. This is then 50%. Looking at it geometrically, we can treat <span class="math-container">$u= [x_1,x_2, ...]$</span> and <span class="math-container">$v = [n_1,n_2,...]$</span> as vectors. We want the probability that <span class="math-container">$|u+v|&gt;|u|$</span>. This is equivalent drawing a hypersphere with radius <span class="math-container">$u$</span> and dimension <span class="math-container">$k$</span>, where <span class="math-container">$k$</span> is the number of observations (normally that would be called <span class="math-container">$n$</span>, but you've represented the noise with that variable), then asking what the probability of <span class="math-container">$u+v$</span> being inside that hypersphere is. To find that, we draw a hypersphere centered at the head of <span class="math-container">$u$</span> with radius <span class="math-container">$v$</span>, and calculate what percentage of that smaller hypersphere is inside the larger hypersphere. For <span class="math-container">$|u| &gt;&gt;|v|$</span>, the larger hypersphere cuts through the smaller circle almost like a straight hyperplane, so the probability is close to half. As <span class="math-container">$|v|$</span> gets larger, the larger hypersphere curves more, and a smaller portion of the smaller hypersphere is inside the larger hypersphere.</p>