Intuition behind standard deviation

Question

I'm trying to gain a better intuitive understanding of standard deviation.

From what I understand it is representative of the average of the differences of a set of observations in a data set from the mean of that data set. However it is NOT actually equal to the averages of the differences as it gives more weight to observations further from the mean.

Say I have the following population of values - $\{1, 3, 5, 7, 9\}$

The mean is $5$.

If I take a measure of spread based on absolute value I get

$$\frac{\sum_{i = 1}^5|x_i - \mu|}{5} = 2.4$$

If I take a measure of spread based using standard deviation I get

$$\sqrt{\frac{\sum_{i = 1}^5(x_i - \mu)^2}{5}} = 2.83$$

The result using standard deviation is larger, as expected, because of the extra weight it gives to values further from the mean.

But if I was just told that I was dealing with a population with a mean of $5$ and a standard deviation of $2.83$ how would I infer that the population was comprised of values something like the $\{1, 3, 5, 7, 9\}$? It just seems that the figure of $2.83$ is very arbitrary...I don't see how you are supposed to interpret it. Does $2.83$ mean the values are spread very wide or are they all tightly clustered around the mean...

When you are presented with a statement that you are dealing with a population with a mean of $5$ and a standard deviation of $2.83$ what does that tell you about the population?

Answer 1

My intuition is that the standard deviation is: a measure of spread of the data.

You have a good point that whether it is wide, or tight depends on what our underlying assumption is for the distribution of the data.

Caveat: A measure of spread is most helpful when the distribution of your data is symmetric around the mean and has a variance relatively close to that of the Normal distribution. (This means that it is approximately Normal.)

In the case where data is approximately Normal, the standard deviation has a canonical interpretation:

• Region: Sample mean +/- 1 standard deviation, contains roughly 68% of the data
• Region: Sample mean +/- 2 standard deviation, contains roughly 95% of the data
• Region: Sample mean +/- 3 standard deviation, contains roughly 99% of the data

(see first graphic in Wiki)

This means that if we know the population mean is 5 and the standard deviation is 2.83 and we assume the distribution is approximately Normal, I would tell you that I am reasonably certain that if we make (a great) many observations, only 5% will be smaller than 0.4 = 5 - 2*2.3 or bigger than 9.6 = 5 + 2*2.3.

Notice what is the impact of standard deviation on our confidence interval? (the more spread, the more uncertainty)

Furthermore, in the general case where the data is not even approximately normal, but still symmetrical, you know that there exist some $\alpha$ for which:

• Region: Sample mean +/- $\alpha$ standard deviation, contains roughly 95% of the data

You can either learn the $\alpha$ from a sub-sample, or assume $\alpha=2$ and this gives you often a good rule of thumb for calculating in your head what future observations to expect, or which of the new observations can be considered as outliers. (keep the caveat in mind though!)

I don't see how you are supposed to interpret it. Does 2.83 mean the values are spread very wide or are they all tightly clustered around the mean...

I guess every question asking "wide or tight", should also contain: "in relation to what?". One suggestion might be to use a well-known distribution as reference. Depending on the context it might be useful to think about: "Is it much wider, or tighter than a Normal/Poisson?".

EDIT: Based on a useful hint in the comments, one more aspect about standard deviation as a distance measure.

Yet another intuition of the usefulness of the standard deviation $s_N$ is that it is a distance measure between the sample data $x_1,… , x_N$ and its mean $\bar{x}$:

$s_N = \sqrt{\frac{1}{N} \sum_{i=1}^N (x_i - \overline{x})^2}$

As a comparison, the mean squared error (MSE), one of the most popular error measures in statistics, is defined as:

$\operatorname{MSE}=\frac{1}{n}\sum_{i=1}^n(\hat{Y_i} - Y_i)^2$

The questions can be raised why the above distance function? Why squared distances, and not absolute distances for example? And why are we taking the square root?

Having quadratic distance, or error, functions has the advantage that we can both differentiate and easily minimise them. As far as the square root is concerned, it adds towards interpretability as it converts the error back to the scale of our observed data.

Answer 2

It may help to realize that the mean is analogous to the center of mass. The variance is the moment of inertia. The standard deviation is the radius of gyration.

For a historical perspective, take a look at:

George Airy (1875) On the algebraical and numerical theory of errors of observations and the combination of observations

Karl Pearson (1894) Contributions To the Mathematical theory of Evolution.

This plot from Airy 1875 shows the various measures of deviation which are easily interconverted (page 17). The standard deviation is called "error of mean square". It is also discussed pages 20-21 and he justifies its use on page 48, showing that it is easiest to calculate by hand because there is no need for separate calculation of negative and positive errors. The term standard deviation was introduced by Pearson in the paper cited above on page 75.

As an aside: Note that the utility of the standard deviation is dependent on the applicability of the "law of errors", also known as the "normal curve", which arises from "a great many independent causes of error" (Airy 1875 pg 7). There is no reason to expect that deviations from a group mean of each individual should follow this law. In many cases for biological systems a log normal distribution is better assumption than normal. See:

Limpert et al (2001) Log-normal Distributions across the Sciences: Keys and Clues

It is further questionable whether it is appropriate to treat individual variation as noise, since the data generating process acts at the level of the individual and not group.

Answer 3

The standard deviation does, indeed, give more weight to those farther from the mean, because it is the square root of the average of the squared distances. The reasons for using this (rather than the mean absolute deviation that you propose, or the median absolute deviation, which is used in robust statistics) are partly due to the fact that calculus has an easier time with polynomials than with absolute values. However, often, we do want to emphasize the extreme values.

As to your question about the intuitive meaning - it develops over time. You are correct that more than one set of numbers can have the same mean and sd; this is because the mean and sd are just two pieces of information, and the data set may be 5 pieces (as 1,3,5,7,9) or much more.

Whether a mean 5 and sd of 2.83 is "wide" or "narrow" depends on the field you are working in.

When you have only 5 numbers, it is easy to look at the full list; when you have many numbers, more intuitive ways of thinking about spread include such things as the five number summary or, even better, graphs such as a density plot.

Answer 4

Standard deviation measures the distance of your population from the mean as random variables.

Let us suppose that your 5 numbers are equally likely to have occurred, so that each has probability .20. This is represented by the random variable $X: [0,1] \rightarrow \mathbb{R}$ given by

$$X(t) = \begin{cases} 1 & 0 \leq t < \frac{1}{5} \\ 3 & \frac{1}{5} \leq t < \frac{2}{5}\\ 5 & \frac{2}{5} \leq t < \frac{3}{5}\\ 7 & \frac{3}{5} \leq t < \frac{4}{5}\\ 9 & \frac{4}{5} \leq t \leq 1 \end{cases}$$

The reason we move to functions and measure theory is because we need to have a systematic way of discussing how two probability spaces are the same up to events which have zero chance of occurring. Now that we have moved to functions we need a sense of distance.

There are many senses of distance for functions, most notably the norms $$||Y||_p = \left(\int_{0}^1|Y(t)|^pdt\right)^{1/p}$$ for $Y: [0,1] \rightarrow \mathbb{R}$ and $1 \leq p < \infty$ induce the distance functions $d_p(Y,Z) = ||X - Z||_p$.

If we take the $p=1$ norm we get the naïve absolute value deviation that you mentioned: $$d_1(X,5) = ||X - \underline{5} ||_1 = 2.4. $$ If we take the $p=2$ norm we get the usual standard deviation $$d_2(X,5) = ||X-\underline{5}||_2 = 2.83.$$

Here $\underline{5}$ denotes the constant function $t \mapsto 5$.

Understanding the meaning of standard deviation is really understanding the meaning of the distance function $d_2$ and understanding why it is, in many senses, the best measure of distance between functions.

Answer 5

The mean absolute deviation is the arithmetic average of all the absolute distances between the data and their mean.

By analogy, the standard deviation is a "kind of" weighted average of all the distances between the data and their mean, weighted by that distance. The fact that it's the squared root allows to transform its units.