12
$\begingroup$

I am puzzled by the following statement:

" In order to increase the standard deviation of a set of numbers, you must add a value that is more than one standard deviation away from the mean"

What is the proof of that? I know of course how we define the standard deviation but that part I seem to miss somehow. Any comments?

$\endgroup$
  • 1
    $\begingroup$ Have you tried to work out the algebra involved? $\endgroup$ – Alecos Papadopoulos Oct 22 '13 at 17:44
  • $\begingroup$ Yes, I have. I have subtracted the sample variance of n values from the variance of n+1 values and I have required the difference to be greater than zero. Yet I cannt quite figure it out. $\endgroup$ – JohnK Oct 22 '13 at 18:02
  • 3
    $\begingroup$ One of the simplest ways is to differentiate Welford's algorithm with respect to the new value $x_n$ and then integrate to show that if introducing $x_n$ increases the variance, then $(x_n-\bar{x}_{n-1})^2 \ge \frac{n}{n-1}v_{n-1}$ where $\bar{x}_{n-1}$ is the mean of the first $n-1$ values and $v_{n-1}$ is their variance estimate. $\endgroup$ – whuber Oct 22 '13 at 19:11
  • $\begingroup$ Okay but can this be shown with simple algebra perhaps? My knowledge of statistics is not that advanced. $\endgroup$ – JohnK Oct 22 '13 at 19:24
20
$\begingroup$

For any $N$ numbers $y_1,y_2, \ldots, y_N$ with mean $\displaystyle \bar{y} = \frac{1}{N}\sum_{i=1}^N y_i$, the variance is given by $$\begin{align} \sigma^2 &= \frac{1}{N-1}\sum_{i=1}^N (y_i-\bar{y})^2\\ &= \frac{1}{N-1}\sum_{i=1}^N \left(y_i^2 - 2y_i\bar{y} + \bar{y}^2\right)\\ &= \frac{1}{N-1}\left[\left(\sum_{i=1}^Ny_i^2\right) - 2N(\bar{y})^2 + N(\bar{y})^2 \right] \\ \sigma^2 &=\frac{1}{N-1}\sum_{i=1}^N \left(y_i^2 - (\bar{y})^2\right) \tag{1} \end{align}$$ Applying $(1)$ to the given set of $n$ numbers $x_1, x_2, \ldots x_n$ which we take for convenience in exposition to have mean $\bar{x} = 0$, we have that $$\sigma^2 = \frac{1}{n-1}\sum_{i=1}^n \left(x_i^2-(\bar{x})^2\right) = \frac{1}{n-1}\sum_{i=1}^n x_i^2$$ If we now add in a new observation $x_{n+1}$ to this data set, then the new mean of the data set is $$\frac{1}{n+1}\sum_{i=1}^{n+1}x_i = \frac{n\bar{x} + x_{n+1}}{n+1} = \frac{x_{n+1}}{n+1}$$ while the new variance is $$\begin{align} \hat{\sigma}^2 &= \frac{1}{n}\sum_{i=1}^{n+1} \left(x_i^2-\frac{x_{n+1}^2}{(n+1)^2}\right)\\ &= \frac{1}{n}\left[\left((n-1)\sigma^2 + x_{n+1}^2\right) - \frac{x_{n+1}^2}{n+1}\right]\\ &= \left.\left.\frac{1}{n}\right[(n-1)\sigma^2 + \frac{n}{n+1}x_{n+1}^2\right]\\ &> \sigma^2 ~ \text{only if}~ x_{n+1}^2 > \frac{n+1}{n}\sigma^2. \end{align}$$ So $|x_{n+1}|$ needs to be larger than $\displaystyle\sigma\sqrt{1+\frac{1}{n}}$ or, more generally, $x_{n+1}$ needs to differ from the mean $\bar{x}$ of the original data set by more than $\displaystyle\sigma\sqrt{1+\frac{1}{n}}$, in order for the augmented data set to have larger variance than the original data set. See also Ray Koopman's answer which points out that the new variance is larger than, equal to, or smaller than, the original variance according as $x_{n+1}$ differs from the mean by more than, exactly, or less than $\displaystyle\sigma\sqrt{1+\frac{1}{n}}$.

$\endgroup$
  • 5
    $\begingroup$ +1 Finally somebody gets it right... ;-) The statement to be proved is correct; it's just not tight. Incidentally, you may also pick your units of measurement to make $\sigma^2=1$, which further simplifies the calculation, reducing it to about two lines. $\endgroup$ – whuber Oct 22 '13 at 22:28
  • $\begingroup$ I suggest you use S instead of sigma in the first set of equations and thanks for the derivation. It was good to know :) $\endgroup$ – Theoden Oct 4 '16 at 19:12
3
$\begingroup$

The puzzling statement gives a necessary but insufficient condition for the standard deviation to increase. If the old sample size is $n$, the old mean is $m$, the old standard deviation is $s$, and a new point $x$ is added to the data, then the new standard deviation will be less than, equal to, or greater than $s$ according as $|x-m|$ is less than, equal to, or greater than $s\sqrt{1+1/n}$.

$\endgroup$
  • 1
    $\begingroup$ Do you have a proof at hand? $\endgroup$ – JohnK Oct 22 '13 at 19:27
2
$\begingroup$

Leaving aside the algebra (which also works) think about it this way: The standard deviation is square root of the variance. The variance is the average of the squared distances from the mean. If we add a value that is closer to the mean than this, the variance will shrink. If we add a value that is farther from the mean than this, it will grow.

This is true of any average of values that are non-negative. If you add a value that is higher than the mean, the mean increases. If you add a value that is less, it decreases.

$\endgroup$
  • $\begingroup$ I would love to see a rigorous proof as well. While I understand the principle I am puzzled by the fact that the value has to be at least 1 deviation away from the mean. Why precisely 1? $\endgroup$ – JohnK Oct 22 '13 at 18:00
  • $\begingroup$ I don't see what is confusing. The variance is the average. If you add something greater than the average (that is, more than 1 sd) it increases. But I am not one for formal proofs $\endgroup$ – Peter Flom Oct 22 '13 at 18:02
  • $\begingroup$ It could be greater than the average by 0.2 standard deviations. Why wouldn't it increase then? $\endgroup$ – JohnK Oct 22 '13 at 18:14
  • $\begingroup$ No, not greater than the mean of the data, greater than the variance, which is the mean of the squared distances. $\endgroup$ – Peter Flom Oct 22 '13 at 18:23
  • 4
    $\begingroup$ It is confusing because including a new value changes the mean, so all the residuals change. It is conceivable that even when the new value is far from the old mean, its contribution to the SD could be compensated by reducing the sum of squares of the residuals of the other values. This is one of the many reasons why rigorous proofs are useful: they provide not only security in one's knowledge, but insight (and even new information) as well. For instance, the proof will show that you have to add a new value that is strictly further than one SD from the mean in order to increase the SD. $\endgroup$ – whuber Oct 22 '13 at 19:14
2
$\begingroup$

I'll get you started on the algebra, but won't take it quite all of the way. First, standardize your data by subtracting the mean and dividing by the standard deviation: $$ Z = \frac{x-\mu}{\sigma} .$$ Note that if $x$ is within one standard deviation of the mean, $Z$ is between -1 and 1. Z would be 1 if $x$ were exactly one sd away from the mean. Then look at your equation for standard deviation: $$\sigma = \sqrt{\frac{\sum_{i=1}^{N}Z_i^2}{N-1}}$$ What happens to $\sigma$ if $Z_N$ is between -1 and 1?

$\endgroup$
  • $\begingroup$ A number whose absolute value is less than 1, when squared it is also going to be less than 1 in abs. value. Yet what I do not understand is that even if Z_N falls into that category, we are adding a positive value to σ, so shouldn't it increase? $\endgroup$ – JohnK Oct 22 '13 at 18:13
  • $\begingroup$ Yes, you are adding a positive value, but it will be smaller than your average deviation from the mean and therefore reduce sigma. Maybe it would make more sense to consider the value as $Z_{N+1}$. $\endgroup$ – wcampbell Oct 22 '13 at 18:20
  • 1
    $\begingroup$ 1) Don't forget, when you add that value, you are also increasing $N$ by 1. 2) You are not adding that value to $\sigma$, you are adding it to $\sum Z_i^2$. $\endgroup$ – jbowman Oct 22 '13 at 18:24
  • $\begingroup$ Exactly what I was trying to express! $\endgroup$ – wcampbell Oct 22 '13 at 18:25
  • $\begingroup$ It's not that simple: in this answer you have computed the SD as if the new value were already part of the dataset. Instead, the $Z_i$ have to be standardized with respect to the SD and mean of the first $N-1$ values only, not all of them. $\endgroup$ – whuber Oct 22 '13 at 18:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.