I apologize in advance, I'm new to statistics. I have a large (millions) dataset (the US Census American Community Survey) with 286 attributes. I've calculated the mean, variance and standard deviation for each attribute, and I would like to order them by "variability" (roughly, that is - if the concept I'm driving at is not a statistical one, I'll settle for one that is just intuitively appealing.) Obviously I can rank them by sigma, highest to lowest (ranging from very high to very low in absolute terms, on the order of thousands down to <1.0). But is that meaningful? Does a large sigma (or variance) mean it's more "variable", or should I be looking for the largest RATIO of sigma to mean? I can't find any reference to a concept like that in my textbooks, but it seems to me to convey more meaning than a variance/sigma on its own. (The point, if you haven't guessed, is to reduce the dimensionality to something more manageable by discarding the attributes with the least variability.)
2 Answers
The standard deviation, by itself, is not very useful for comparing one variable to another. Suppose, for instance, you were comparing income (in \$) to number of children. The latter varies from, perhaps, 0 to 15; the former from \$0 to some millions.
One method is to, as you suggest, take the ratio of the sd to the mean; this is called the coefficient of variation.
Is this the best method? It depends on what you mean by "variability" of an attribute, how much you want to weight extreme values, and so on. Another possible method is the ratio of the interquartile range (or total range) to the median. Yet another idea would be the ratio of the range to the interquartile range. Other methods could be devised.
Considering income: Bill Gates and Warren Buffett will have a huge influence on the mean, the sd and the coefficient of variation. They will have no influence at all on the ratio of the interquartile range to the median.
One problem with the coefficient of variation is that it becomes very weird when some of the values are negative. e.g. Taking two series:
-3 -1 0 1 3
and
0 1 3 5 7
our intuition says the variability is the same. Yet the coefficient of variation for the first set is infinite!
-
1$\begingroup$ Thanks very much peter and @glen-b - I don't have enough points to up-vote your answer, but I would if I could, they were both very edifying and informative. I'm re-writing my code to add a column for coefficient of variation now, and try out the metrics that Peter suggests. I have a license to experiment with it, now that I know the rough boundaries. Thanks again - Ed $\endgroup$– EdBCommented May 25, 2014 at 20:43
The ratio of standard deviation to mean is called the coefficient of variation, and is widely used with positive variables.
Either that or the standard deviation may be relevant, depending on what your variables are and what it is you're interested in comparing.
http://en.wikipedia.org/wiki/Coefficient_of_variation
Coefficient of variation will be constant when the standard deviation is proportional to the mean (by the definition of CV).
Roughly speaking, it's like comparing standard deviations of logged variables (in fact for a number of commonly-used distributions, it's exactly that).
If it suits you to use the coefficient of variation, you might well do so.
