It says in wikipedia

The role of the normal distribution in the central limit theorem is in part responsible for the prevalence of the variance in probability and statistics.

I understand this as
When we use variance/SD as a measure of dispersion, we are actually looking for the "scaling parameter" of a normal distribution, since a random random variable is likely to follow approximately a normal distribution to CLT.

In the case that the data is not normally distributed, is variance/SD still a reasonable measure of dispersion?

Say the data is uniformly distributed, the average absolute deviation seems to be a better measure of dispersion than the variance, because it can be seen as the "scaling parameter" for the uniform distribution, am I right?

I mean, say I have two sets of samples, one is {1,1,1,-1,-1,-1} and the other one is drawn from a normal distribution $N(0,1)$, their variances are both 1. The two sets will be considered as of the same degree of dispersion if we use variance as the measure.

But it feels like we are forcefully treating them both as Gaussian then work out the distribution parameters and say "yeah they're equal in terms of dispersion".

  • 1
    $\begingroup$ In what way do you mean "work" in the title? Work at doing what? In estimating the population variance, or something else? Measured how? In what sense do you intend the word "better" in the final paragraph? Better at what, exactly? If you're after a low-variance estimate of the spread in a general continuous uniform distribution I wouldn't use the average absolute deviation, but some function of the range. $\endgroup$ – Glen_b Dec 2 '16 at 3:42
  • $\begingroup$ @Glen_b i'm not sure maybe work as a measure of dispersion/deviation/discrepancy, maybe better in the sense that the scale of a uniform distribution with SD $\sigma$ is not twice as large as that of a uniform distribution with SD $0.5\sigma$, isn't the average absolute deviation just a function of the range? $\endgroup$ – dontloo Dec 2 '16 at 3:55
  • $\begingroup$ For the uniform, the population average absolute deviation and the population standard deviation are both functions of the population range (and vice versa -- if you know any of them you know all the others), but the sample average absolute deviation, the sample sd and the sample range are not equally good at estimating them. For example, if your measure of "working"/"goodness" is the variance of the estimator, then - at least in large samples - a multiple of the sample range is the best way to estimate all three. But if your criteria for what "good" is change, something else might do better. $\endgroup$ – Glen_b Dec 2 '16 at 5:09
  • 1
    $\begingroup$ @Glen_b thanks for your reply, I get the point that "variance is just what it's defined to be, and it doesn't necessarily have anything to do with a specific distribution", what I meant is, when it's used as a measure of dispersion, it seems not the best choice for non-Gaussian (or the like) distributions. $\endgroup$ – dontloo Dec 3 '16 at 7:08
  • 1
    $\begingroup$ Maybe you have linked the standard deviation and variance with the Gaussian because your first intense exposure is with the Gaussian distribution and its transformations, such as Student's t distribution. It might be better to think of variance as a feature as a nose is a feature of animals. If it doesn't have a nose, it might be a tree. Height is a feature. A tree can be five foot tall, when its young at least, and a human can be five foot tall. It is a descriptor of a distribution, but not the only descriptor of a distribution. $\endgroup$ – Dave Harris Dec 3 '16 at 22:15

Your question is a little vague, but no, variance isn't used because of its association with the normal distribution. Most distributions have at least a mean and a variance. Some do not have a variance. Some can either have or not have a variance. Some have no mean and so do not have a variance.

Just for mental clarification on your side, if a distribution has a mean then $\bar{x}\approx\mu,$ but if it does not then $\bar{x}\approx\text{nothing}$. That is it gravitates nowhere and any calculation just floats around the real number line. It doesn't mean anything. The same is true if you calculate a standard deviation for a distribution that does not have one. It has no meaning.

The variance is a property of a distribution. You are correct in that it can be used to scale the problem, but it is deeper than that. In some theoretical frameworks, it is a measure of our ignorance, or more precisely, uncertainty. In others, it measures how large of an effect chance can have on outcomes.

Although variance is a conceptualization of dispersion, it is an incomplete conceptualization. Both skew and kurtosis further explain how the dispersion operates on a problem.

For many problems in a null hypothesis framework of thinking, the Central Limit Theorem makes the discussion of problems simpler and so it doesn't hurt that there is a linkage between the normal distribution, with its very well defined distributional properties, and the use of the standard deviation. However, this is more true for simple problems than complex ones. This is also less true for Bayesian methods which do not use a null hypothesis and which do not depend on the sampling distribution of the estimator.

The average absolute deviation is a valuable tool in parameter free and distribution free methods, but less valuable for the uniform distribution. If you actually had a bounded uniform distribution, then the mean and the variance are known.

Let me give you a uniform distribution problem that may not be as simple as you think. Consider that a new enemy battle tank has appeared on the battlefield. You do not know how many they have, let alone that they existed. You want to estimate the total number of tanks.

Tanks have serial numbers on their engines, or used to before someone figured this out. The probability of capturing any one specific serial number is $1/N$ where $N$ is the total of the tanks. Of course you do not know $N$, so this is an interesting problem. You need to know N. You can only see the distribution of captured serial numbers and not know if the largest number captured is also the last tank built. It probably is not.

In that case, the mean and standard deviation provide the most powerful tools to solve the problem, despite the intuition that the standard deviation is a bad estimator.

It will be true that it is a bad estimator for certain problems, but you need to learn them on a case by case basis.

Statistical tools are chosen based on needs, rules of math and trade-offs between real world costs and limitations and the demands of the problem. Sometimes that is the variance, but sometimes it is not. The best thing to do is to learn why the rules are designed the way they are and that is too long for a posting here.

I would recommend a good practitioners book on non-parametric statistics and if you have had calculus a good introductory practitioners book on Bayesian methods.

  • 1
    $\begingroup$ Related to The German Tank Problem, in case someone likes both history, tanks, and statistics: en.wikipedia.org/wiki/German_tank_problem $\endgroup$ – Beyer Dec 2 '16 at 8:18
  • $\begingroup$ thank you lots for the answer, just I didn't quite follow how the SD of the samples helps in German Tank Problem? I only see the use of the SD of the estimation (from the link above). $\endgroup$ – dontloo Dec 3 '16 at 3:40
  1. First we need to be clear about the distinction between a measure of the variability of a distribution (such as its standard deviation or its mean deviation or its range) and the best way to estimate that measure from a sample. For example, if your distribution is uniform, the best sample estimate of the population mean deviation from the mean is not the sample mean deviation -- actually a fraction of the range is generally much better.

    (Of course if you really don't know what distribution you may be dealing with, such considerations may not be much help.)

  2. So why measure population variability by variance?

    Variance (and through it, standard deviation) has a very particular property that isn't shared by other measures of variability, which is a very simple form for the variance of sums (and more generally linear combinations) of variables.

    When you have independence, the simple form becomes much simpler still.

    Specifically, under independence, $\text{Var}(X+Y) = \text{Var}(X) + \text{Var}(Y)$ and because of that the standard deviation is also quite simple in form. The non-independence case is not much more complicated.

    Other measures of variability don't have such a simple property.

    This makes variance (and hence standard deviation) very attractive ways to measure variability of distributions.

  3. A second reason is that the mean (which is often seen as a natural location measure) is the location that minimizes a square error loss function -- and when you minimize it, you obtain the variance. Many people see a square-error loss function as natural or useful, and in that case the variance in turn becomes a natural measure of variation.

  • $\begingroup$ But I've always understood the statistical meaning of squared error loss as maximizing the log-likelihood under a Gaussian noise assumption, which again derives from the CLT. $\endgroup$ – dontloo Dec 6 '16 at 12:11
  • $\begingroup$ So the prevalence of the variance is mostly because of its mathematical convenience? $\endgroup$ – dontloo Dec 6 '16 at 12:11
  • 1
    $\begingroup$ @dontloo To answer a question phrased that way would require speculation, tractability is a reason to use variance. You're asking me to make a claim for which I don't have sufficient evidence (there are multiple reasons -- including some I haven't listed, such as a degree of status quo bias -- but to assert one as the primary cause would require evidence I don't possess). I would guess that reasons 2 and 3 above are strong reasons for prevalence, and arguably sufficient ones. $\endgroup$ – Glen_b Dec 6 '16 at 16:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.