Why Are Measures of Dispersion Less Intuitive Than Centrality?

Question

There seems to be something in our human understanding that creates difficulties in grasping intuitively the idea of variance. In a narrow sense the answer is immediate: squaring throws us off from our reflexive understanding. But, is it just variance that presents problems, or is it the whole idea of spread in the data? We seek refuge in the range, or just stating the minimum and maximum, but are we just avoiding the real difficulty? In the mean (mode or median) we find the center, the summary... a simplification; the variance spreads things around and makes them uncomfortable. The primitive man would certainly make use of the mean in hunting animals by triangulating to the pray, but I presume it was much later that we felt the need to quantify the spread of things. In fact, the term variance was first introduced by Ronald Fisher as recently as 1918 in the paper "The Correlation Between Relatives on the Supposition of Mendelian Inheritance."

Most people that follow the news would have heard the story of Larry Summers' unfortunate speech about math aptitudes by gender, which were possibly related to his departure from Harvard. In a nutshell, he suggested a broader variance in the distribution of math competency among males as compared to females, even though both genders enjoyed the same mean. Regardless of the appropriateness or political implications, this seems to be substantiated in the scientific literature.

More importantly, perhaps the understanding of issues like climate change - please forgive me for bringing up topics that could lead to completely uncalled for discussions - by the general population could be aided by improved familiarity with the idea of variance.

The issue gets compounded when we try to grasp covariance, as shown in this post, featuring a great, and colorful answer by @whuber here.

It may be tempting to dismiss this question as too general, but it is clear that we are discussing it indirectly, as in this post, where the mathematics are trivial, yet the concept keeps on being elusive, belying a more comfortable acceptance of range as opposed to the more nuanced idea variance.

In a letter from Fisher to E.B.Ford, referring to the controversy over his suspicion on the Mendelian experiments, we read: "Now, when data had been faked, I know very well how generally people underestimate the frequency of wide chance deviations, so that the tendency is always to make them agree too well with expectations... the deviations [in Mendel’s data] are shockingly small." The great RA Fisher is so keen on suspecting small variances in small samples that he writes: "it remains a possibility, among others that Mendel was deceived by some assistant who knew all too well what was expected."

And it is entirely possible that this bias towards understating or misunderstanding spread persists today. If so, is there any explanation for why we are more comfortable with centrality concepts than with dispersion? Is there anything we can do to internalize the idea?

Some concepts we "see" in a flash, and then we don't, yet we accept them, and move on. For example, $\small e^{i\pi}+1=0$ or $\small E=mc^2$, but we don't really need to even know about these identities to make decisions in our daily lives. The same is not true of variance. So, shouldn't it be more intuitive?

Nassim Taleb has made a fortune applying his (well, really Benoit Mandelbrot's) perception of flawed understanding of variance to exploiting times of crisis, and has tried to make the concept understandable to the masses with sentences like, "the variance of variance is, epistemologically, a measure of lack of knowledge about the lack of knowledge of the mean" - yes, there is more context to this mouthful... And to his credit, he has also made it simpler with the Thanksgiving Turkey idea. One may argue that the key to investing is understanding variance (and covariance).

So why is it so slippery, and how to remedy it? Without formulas... just the intuition of years of dealing with uncertainty... I don't know the answer, but it's not mathematical (necessarily, that is): for instance, I wonder if the idea of kurtosis interferes with variance. In the following plot we have two histograms overlapping with virtually the same variance; yet, my knee jerk reaction is that the one with the longest tails, and tallest peak (higher kurtosis) is more "spread out":

Answer 1

I share your feeling that variance is slightly less intuitive. More importantly, variance as a measure is optimized for certain distributions and has less worth for asymmetric distributions. Mean absolute difference from the mean is not much more intuitive in my view, because it requires one to choose the mean as the measure of central tendency. I prefer Gini's mean difference --- the mean absolute difference over all pairs of observations. It is intuitive, robust, and efficient. On efficiency, if the data come from a Gaussian distribution, Gini's mean difference with an appropriate rescaling factor applied to it is 0.98 as efficient as the sample standard deviation. There is an efficient computing formula for Gini's mean difference once the data are sorted. R code is below.

w <- 4 * ((1:n) - (n - 1)/2)/n/(n - 1)
sum(w * sort(x - mean(x)))

Answer 2

Here are some of my thoughts. It doesn't address every angle you could look at your question from, in fact, there is a lot it doesn't address (the question does feel a bit broad).

Why is it hard for laypeople to understand the mathematical calculation of Variance?

Variance is essentially how spread out things are. This is easy enough to understand, but the way it is calculated may seem counter-intuitive to a layperson.

The issue is that the differences from the mean are squared (then averaged), and then square rooted to get the Standard Deviation. We understand why this method is necessary - the squaring is to make the values positive and then they are square-rooted to get the original units. However, a layperson is likely to be confused with why the numbers are squared and square-rooted. This looks like it cancels itself out (it doesn't) so seems pointless/strange.

What is more intuitive to them is finding the spread by simply averaging the absolute differences between the mean and each point (called Mean absolute deviation). This method does not require squaring and square-rooting, so is far more intuitive.

Note that just because Mean Absolute Deviation is more straightforward, does not mean it is 'better'. The debate of whether to use Squares or Absolute values has been going on for a century involving many prominent statisticians, so a random person like me can't just show up here and say one is better. (Averaging squares to find variance is of course more popular)

In a nutshell: The Squaring to find variance seems less intuitive to laypeople who would find averaging the Absolute differences to be more straightforward. However, I don't think people have a problem with understanding the idea of spread itself

Answer 3

Here it goes my opinion on your question.

I will start by questioning an abovementioned answer to then try to make my point.

Question to previous hypothesis:

Is it really the squares makes dispersion measures such as the Square Mean Deviation hard to understand? I agree the square makes it harder by bringing mathematical complexity but if the answer was only the squares, the Mean Absolute Deviation would be as simple to understand and measures of centrality.

Opinion:

I think that what makes it hard for us to understand measures of dispersion is that dispersion itself is a 2-dimentional information. Trying to summarize a 2-dimensional information in one metric implies a partial loss of information which in consequence causes confusion.

Example:

An example that can help to explain the concept above is by the following. Let's get 2 different sets of data:

1. Follows a Gaussian distribution
2. Follows an unknown and asymetric distribution

Let's also assume the dispersion in terms of Standard Deviation is 1.0.

My mind tends to interpret the dispersion of set 1 much more clearer than that of set 2. In this specific case, the reason for my better understanding is explained knowing 2-dimensional shape of the distribution in advance allows me to understand the distribution measure in terms of a probability around the centralized Gaussian mean. In other words, the Gaussian distribution gave me the 2-dimensional hint I needed to better translate from the measure of dispersion.

Conclusion:

In sum, there no tangible way to capture in one Deviation Measure all there is in a 2-dimensional information. What I usually do to understand dispersion without looking directly at the distribution itself is to combine many measures that explain a certain distribution. They will set up the context for my mind to have a better grasp on the dispersion measure itself. If I could make use of graphs certainly box plots are really useful for visualing it.

Great discussion that made me think a lot on the issue. I would be glad to hear your opinion.

Answer 4

I think that a simple reason that people have a harder time with variability (whether variance, standard deviation, MAD, or whatever) is that you cannot really understand variability until after you understand the idea of center. This is because the measures of variability are all measured based on distance from the center.

Concepts like mean and median are parallel concepts, you could learn either one first and some people may have a better understanding of one and other people will understand the other better. But spread is measured from the center (for some definition of center), so cannot really be understood first.