0
$\begingroup$

I have a series of numbers ranging from -0.69 to 14.703. These datapoints measure YoY change. I want to create a text descriptor based on where a value falls in a range determined by the Standard Deviation. My mean is 1.13144 and Std Dv is 2.1579. This leaves me with these standard deviation steps, to which I assign a value (i.e. if you fall between the mean and (negative) 1 SD, you are Below Average, between the mean and (positive) 1 SD, you are Above Average, and so on.

Text Description of Std Dev

This the list of values I got the mean and Std Dev from:

enter image description here

Nearly every item in this list falls 1 SD below the mean. I know the data is not normally distributed, but I would have expected something a little less skewed. Am I using this correctly?

Edit: Year over Year (YoY) change. I take last year's sales and this year's and find the percent difference.

Improve this question
$\endgroup$
10
  • $\begingroup$ Can you explain YoY change? $\endgroup$
    – kjetil b halvorsen ♦
    May 18 at 14:57
  • $\begingroup$ Year over Year change. I take last year's sales and this year's and find the percent difference. $\endgroup$
    – jabs
    May 18 at 14:58
  • 5
    $\begingroup$ Could you explain why you want to base your classification on the SD while at the same time you recognize the SD might not be appropriate or useful for this distribution? If your underlying problem is to partition the data, then consider reframing it as a clustering problem or, even more simply, just partition the data according to quantiles, such as 0 - 10% ("very low"), 10% - 25% ("low"), 25% - 75% ("medium"), etc. $\endgroup$
    – whuber ♦
    May 18 at 15:04
  • 1
    $\begingroup$ A graph would say much more about these numbers than arbitrary binning based on any criterion. I would illustrate, but the data are presented as an image. The graph could be combined with text annotation if something else were known about each value (that isn't confidential). $\endgroup$
    – Nick Cox
    May 18 at 17:09
  • 1
    $\begingroup$ Whether the distribution is normal should not be overemphasized. No distribution of data is or can be exactly normal. Everything depends on whether there is a reference distribution that acts as a standard comparison. For example, the exponential has SD equal to the mean, which means that values are expected to vary from about $-1$ SD below the mean to rather more SD above the mean. That's what it is. Use it if it is helpful and not otherwise. $\endgroup$
    – Nick Cox
    Jun 1 at 6:31

1 Answer 1

Reset to default
3
$\begingroup$

The standard deviation is a summary metric which can be calculated for any set of numbers, but that doesn't mean that captures something "useful" about that set of numbers in all cases. What you're seeing here is that you have a set of numbers with little variation for the most part, plus a few outlier samples which are the main contributors to the standard deviation. Summary metrics that are highly dependent on a small set of samples tend to not be very useful, since they say more about that small set of samples than they do about the population as a whole, rather defeating the purpose of the summary. On its face, this doesn't seem like a terribly useful way of categorizing YoY changes, since 90% of the values get categorized the same.

The use of "1 SD from the mean" is convenient for normally distributed populations as it's a fixed threshold that provides "reasonably sized" groups on either side of the threshold. But for an arbitrary distribution, 1 SD may or may not have any particular useful meaning at all. Some distributions are entirely contained within 1 SD of the mean, in which case trying to threshold values 1 SD from the mean is pointless.

You see a similar effect in other outlier-dependent summary statistics like the mean. For example, the mean net worth in the US is nearly $1M, but this is dragged way up a small number of billionaires, so many would view it as a misleading summary statistic. The median net worth is barely \$100k, which most would view as a better summary of the population.

Whenever using summary statistics, it's worthwhile to make sure that it's summarizing a useful aspect of the population as a whole.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thanks for the basic and clear write-up. It helped this beginner a lot. $\endgroup$
    – jabs
    May 19 at 19:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.