# Intuition behind standard deviation

I'm trying to gain a better intuitive understanding of standard deviation.

From what I understand it is representative of the average of the differences of a set of observations in a data set from the mean of that data set. However it is NOT actually equal to the averages of the differences as it gives more weight to observations further from the mean.

Say I have the following population of values - $\{1, 3, 5, 7, 9\}$

The mean is $5$.

If I take a measure of spread based on absolute value I get

$$\frac{\sum_{i = 1}^5|x_i - \mu|}{5} = 2.4$$

If I take a measure of spread based using standard deviation I get

$$\sqrt{\frac{\sum_{i = 1}^5(x_i - \mu)^2}{5}} = 2.83$$

The result using standard deviation is larger, as expected, because of the extra weight it gives to values further from the mean.

But if I was just told that I was dealing with a population with a mean of $5$ and a standard deviation of $2.83$ how would I infer that the population was comprised of values something like the $\{1, 3, 5, 7, 9\}$? It just seems that the figure of $2.83$ is very arbitrary...I don't see how you are supposed to interpret it. Does $2.83$ mean the values are spread very wide or are they all tightly clustered around the mean...

When you are presented with a statement that you are dealing with a population with a mean of $5$ and a standard deviation of $2.83$ what does that tell you about the population?

• This question is related (though is not identical) to stats.stackexchange.com/q/81986/3277 and a further one linked to there. – ttnphns Feb 4 '14 at 12:11
• It tells you a 'typical' distance from the mean (the RMS distance). What makes that 'large' or 'small' depends on your criteria. If you're trying to measure engineering tolerances it might be huge. In other contexts the same standard deviation may be regarded as quite small. – Glen_b -Reinstate Monica Feb 4 '14 at 16:53

My intuition is that the standard deviation is: a measure of spread of the data.

You have a good point that whether it is wide, or tight depends on what our underlying assumption is for the distribution of the data.

Caveat: A measure of spread is most helpful when the distribution of your data is symmetric around the mean and has a variance relatively close to that of the Normal distribution. (This means that it is approximately Normal.)

In the case where data is approximately Normal, the standard deviation has a canonical interpretation:

• Region: Sample mean +/- 1 standard deviation, contains roughly 68% of the data
• Region: Sample mean +/- 2 standard deviation, contains roughly 95% of the data
• Region: Sample mean +/- 3 standard deviation, contains roughly 99% of the data

(see first graphic in Wiki)

This means that if we know the population mean is 5 and the standard deviation is 2.83 and we assume the distribution is approximately Normal, I would tell you that I am reasonably certain that if we make (a great) many observations, only 5% will be smaller than 0.4 = 5 - 2*2.3 or bigger than 9.6 = 5 + 2*2.3.

Notice what is the impact of standard deviation on our confidence interval? (the more spread, the more uncertainty)

Furthermore, in the general case where the data is not even approximately normal, but still symmetrical, you know that there exist some $\alpha$ for which:

• Region: Sample mean +/- $\alpha$ standard deviation, contains roughly 95% of the data

You can either learn the $\alpha$ from a sub-sample, or assume $\alpha=2$ and this gives you often a good rule of thumb for calculating in your head what future observations to expect, or which of the new observations can be considered as outliers. (keep the caveat in mind though!)

I don't see how you are supposed to interpret it. Does 2.83 mean the values are spread very wide or are they all tightly clustered around the mean...

I guess every question asking "wide or tight", should also contain: "in relation to what?". One suggestion might be to use a well-known distribution as reference. Depending on the context it might be useful to think about: "Is it much wider, or tighter than a Normal/Poisson?".

EDIT: Based on a useful hint in the comments, one more aspect about standard deviation as a distance measure.

Yet another intuition of the usefulness of the standard deviation $s_N$ is that it is a distance measure between the sample data $x_1,… , x_N$ and its mean $\bar{x}$:

$s_N = \sqrt{\frac{1}{N} \sum_{i=1}^N (x_i - \overline{x})^2}$

As a comparison, the mean squared error (MSE), one of the most popular error measures in statistics, is defined as:

$\operatorname{MSE}=\frac{1}{n}\sum_{i=1}^n(\hat{Y_i} - Y_i)^2$

The questions can be raised why the above distance function? Why squared distances, and not absolute distances for example? And why are we taking the square root?

Having quadratic distance, or error, functions has the advantage that we can both differentiate and easily minimise them. As far as the square root is concerned, it adds towards interpretability as it converts the error back to the scale of our observed data.

• Why do you say that a measure of spread is most 'helpful' when the data are normal? It seems to me that any set of data have a spread and the standard deviation is a summary of the spread even if it doesn't capture the shape of the spread. – Michael Lew Feb 4 '14 at 22:49
• Sure, you are right. But I wasn't claiming that the standard deviation depends on the shape of the distribution in any way. Merely pointing out that IF you have some knowledge about the shape (or you are ready to make this assumption), it is usually a much more helpful information. In a similar way, the sample mean is a good descriptor of your data, IF you can make certain general assumptions about the distribution. – means-to-meaning Feb 14 '14 at 17:27
• My favorite reason for using square instead of absolute value is that way it is a logarithm of probability of some Gaussian. So if you believe that errors are Gaussian in nature, and that bits are good way of measuring information, then it makes sense to use squared error. – qbolec Dec 18 '17 at 20:27

It may help to realize that the mean is analogous to the center of mass. The variance is the moment of inertia. The standard deviation is the radius of gyration.

For a historical perspective, take a look at:

Karl Pearson (1894) Contributions To the Mathematical theory of Evolution.

This plot from Airy 1875 shows the various measures of deviation which are easily interconverted (page 17). The standard deviation is called "error of mean square". It is also discussed pages 20-21 and he justifies its use on page 48, showing that it is easiest to calculate by hand because there is no need for separate calculation of negative and positive errors. The term standard deviation was introduced by Pearson in the paper cited above on page 75. As an aside: Note that the utility of the standard deviation is dependent on the applicability of the "law of errors", also known as the "normal curve", which arises from "a great many independent causes of error" (Airy 1875 pg 7). There is no reason to expect that deviations from a group mean of each individual should follow this law. In many cases for biological systems a log normal distribution is better assumption than normal. See:

Limpert et al (2001) Log-normal Distributions across the Sciences: Keys and Clues

It is further questionable whether it is appropriate to treat individual variation as noise, since the data generating process acts at the level of the individual and not group.

The standard deviation does, indeed, give more weight to those farther from the mean, because it is the square root of the average of the squared distances. The reasons for using this (rather than the mean absolute deviation that you propose, or the median absolute deviation, which is used in robust statistics) are partly due to the fact that calculus has an easier time with polynomials than with absolute values. However, often, we do want to emphasize the extreme values.

As to your question about the intuitive meaning - it develops over time. You are correct that more than one set of numbers can have the same mean and sd; this is because the mean and sd are just two pieces of information, and the data set may be 5 pieces (as 1,3,5,7,9) or much more.

Whether a mean 5 and sd of 2.83 is "wide" or "narrow" depends on the field you are working in.

When you have only 5 numbers, it is easy to look at the full list; when you have many numbers, more intuitive ways of thinking about spread include such things as the five number summary or, even better, graphs such as a density plot.

Standard deviation measures the distance of your population from the mean as random variables.

Let us suppose that your 5 numbers are equally likely to have occurred, so that each has probability .20. This is represented by the random variable $X: [0,1] \rightarrow \mathbb{R}$ given by

$$X(t) = \begin{cases} 1 & 0 \leq t < \frac{1}{5} \\ 3 & \frac{1}{5} \leq t < \frac{2}{5}\\ 5 & \frac{2}{5} \leq t < \frac{3}{5}\\ 7 & \frac{3}{5} \leq t < \frac{4}{5}\\ 9 & \frac{4}{5} \leq t \leq 1 \end{cases}$$

The reason we move to functions and measure theory is because we need to have a systematic way of discussing how two probability spaces are the same up to events which have zero chance of occurring. Now that we have moved to functions we need a sense of distance.

There are many senses of distance for functions, most notably the norms $$||Y||_p = \left(\int_{0}^1|Y(t)|^pdt\right)^{1/p}$$ for $Y: [0,1] \rightarrow \mathbb{R}$ and $1 \leq p < \infty$ induce the distance functions $d_p(Y,Z) = ||X - Z||_p$.

If we take the $p=1$ norm we get the naïve absolute value deviation that you mentioned: $$d_1(X,5) = ||X - \underline{5} ||_1 = 2.4.$$ If we take the $p=2$ norm we get the usual standard deviation $$d_2(X,5) = ||X-\underline{5}||_2 = 2.83.$$

Here $\underline{5}$ denotes the constant function $t \mapsto 5$.

Understanding the meaning of standard deviation is really understanding the meaning of the distance function $d_2$ and understanding why it is, in many senses, the best measure of distance between functions.

• This explanation includes some constructions that do not seem "intuitive." The principal one is the unwarranted appearance of a function defined on $[0,1]$, an interval which has nothing to do with the setting. (It is natural to define $X:\{1,3,5,7,9\}\to\mathbb{R}$ as $X(i)=i$ where the algebra is the power set of $\{1,3,5,7,9\}$.) Also, interpreting expressions like "$||X-5||_1$" is somewhat problematic because "$5$" represents a number--the mean of the population--not a random variable. In the end, after all this machinery is introduced, the question is restated but not actually answered. – whuber Feb 4 '14 at 19:47
• Yes the random variable you listed is standard for those comfortable with measure theory. I was hoping to narrow it down to understanding functions and integration for people with only calculus background. I will rewrite the mean as a function. – SomeEE Feb 4 '14 at 19:58
• Also, in that it is a restated question, are you suggesting to include comments about why $d_2$ is the best measure of distance between functions? – SomeEE Feb 4 '14 at 20:00
• The question asks for intuition in understanding the standard deviation. You have explained how it is the $L^2$ norm in some function space. Although that provides another mathematical formalization (and would be adequate intuition for a mathematician otherwise ignorant of the standard deviation), it seems to stop short of what the original poster was requesting. What would be most welcome is a follow-up paragraph explaining the "meaning of the distance function $d_2$" and elaborating, if only a little, on the senses in which it is a "best" measure of distance. – whuber Feb 4 '14 at 20:11