I was wondering how reliable is a large standard deviation whose negative values are below 0? If we were calculating 2 standard deviations away from the mean to catch at least 75% of the observations over a non-normal distribution and end up with (say) -600 standard deviation over a mean of 300 knowing that the measure we are applying the standard deviation to does not accept negative values(i.e. min value is 0), we would end up with something like 300 - (600*2) on the left of our distribution and 300 + (600*2) on the right. Does it even make sense to have those negative values below 0? Or in this case, is it imperative to normalize the data? Thanks

  • $\begingroup$ You didn't say this, but am I correct to assume that your data is measuring something that cannot be negative? Can you provide more information on what you're trying to do? $\endgroup$ – Chris Umphlett Aug 20 '18 at 19:23
  • $\begingroup$ Hi Chris, correct. I have a large population of 174k observation circa which represent the deposit amounts from unique users in the last 6 months. That population is largely right skewed and of course cannot accept negative values. I want to apply the ± 2 SD to catch at least 75% of the observations and to get rid of outliers skewing the data. Thanks $\endgroup$ – AlGrasso Aug 20 '18 at 19:30
  • $\begingroup$ for your purposes would you consider any deposit lower than the mean to be an outlier? $\endgroup$ – Chris Umphlett Aug 20 '18 at 19:31
  • $\begingroup$ No, in the specific distribution I have none of the low values below the mean would be considered outliers.Thanks $\endgroup$ – AlGrasso Aug 20 '18 at 19:38
  • 1
    $\begingroup$ In many applications--likely a great many--any rule that automatically eliminates data it declares to be "outliers" will be suspect, difficult to defend, and may lead to procedures that have poor properties. $\endgroup$ – whuber Aug 20 '18 at 19:51

Standard deviation is always positive, so a std of -600 doesn't make sense.

Chebyshev's inequality is just that: an inequality. It doesn't say that to get 75% of the data, you have to go out 2 std. It says you have to go out at most 2 std. In your examples, at least 75% of the data has a value greater than -900. Now, you may know, from sources other than Chebyshev's inequality, that all of the data has a value greater than 0, and hence greater than -900. So in that case, Chebyshev's inequality doesn't give you any more information about the lower bound than what you already had. In that case, Chebyshev's inequality isn't particularly useful, but it is still valid.

  • $\begingroup$ I think the std is 600, OP is just concerned with the value of mean - std*2 being negative. $\endgroup$ – Chris Umphlett Aug 20 '18 at 19:25
  • $\begingroup$ Thanks for your answer, would it be then accurate in your opinion to consider anything from 0 up to +2 SD from the mean a valid threshold to delimit our data, streamlining its central values? $\endgroup$ – AlGrasso Aug 20 '18 at 19:41

Given you said in your comment

none of the low values below the mean would be considered outliers

I wouldn't worry about the lower end (mean - std*2).

I use Chebyshev's inequality in a similar situation-- data that is not normally distributed, cannot be negative, and has a long tail on the high end. While there can be outliers on the low end (where mean is high and std relatively small) it's generally on the high side. We use 3 std for excluding outliers prior to doing some forecasting, so that no more than 11% of observations will be excluded per the inequality. In practice when we first implemented we found that only about 1% of observations were excluded. Chebyshev's was a nice theoretical bound to give us some justification for setting the threshold at 3 std's instead of needing to go higher.

  • $\begingroup$ Thanks Chris, I came to the same conclusion but had no basis to sustain it. However the end result was that 99% circa of the observations were caught with a thick agglomerate close to the mean and mildly above. Thanks $\endgroup$ – AlGrasso Aug 20 '18 at 19:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.