# Chebyshev's theorem with large standard deviations

I was wondering how reliable is a large standard deviation whose negative values are below 0? If we were calculating 2 standard deviations away from the mean to catch at least 75% of the observations over a non-normal distribution and end up with (say) -600 standard deviation over a mean of 300 knowing that the measure we are applying the standard deviation to does not accept negative values(i.e. min value is 0), we would end up with something like 300 - (600*2) on the left of our distribution and 300 + (600*2) on the right. Does it even make sense to have those negative values below 0? Or in this case, is it imperative to normalize the data? Thanks

• You didn't say this, but am I correct to assume that your data is measuring something that cannot be negative? Can you provide more information on what you're trying to do? Aug 20 '18 at 19:23
• Hi Chris, correct. I have a large population of 174k observation circa which represent the deposit amounts from unique users in the last 6 months. That population is largely right skewed and of course cannot accept negative values. I want to apply the ± 2 SD to catch at least 75% of the observations and to get rid of outliers skewing the data. Thanks Aug 20 '18 at 19:30
• for your purposes would you consider any deposit lower than the mean to be an outlier? Aug 20 '18 at 19:31
• No, in the specific distribution I have none of the low values below the mean would be considered outliers.Thanks Aug 20 '18 at 19:38
• In many applications--likely a great many--any rule that automatically eliminates data it declares to be "outliers" will be suspect, difficult to defend, and may lead to procedures that have poor properties.
– whuber
Aug 20 '18 at 19:51

Standard deviation is always positive, so a std of -600 doesn't make sense.

Chebyshev's inequality is just that: an inequality. It doesn't say that to get 75% of the data, you have to go out 2 std. It says you have to go out at most 2 std. In your examples, at least 75% of the data has a value greater than -900. Now, you may know, from sources other than Chebyshev's inequality, that all of the data has a value greater than 0, and hence greater than -900. So in that case, Chebyshev's inequality doesn't give you any more information about the lower bound than what you already had. In that case, Chebyshev's inequality isn't particularly useful, but it is still valid.

• I think the std is 600, OP is just concerned with the value of mean - std*2 being negative. Aug 20 '18 at 19:25
• Thanks for your answer, would it be then accurate in your opinion to consider anything from 0 up to +2 SD from the mean a valid threshold to delimit our data, streamlining its central values? Aug 20 '18 at 19:41

Given you said in your comment

none of the low values below the mean would be considered outliers

I wouldn't worry about the lower end (mean - std*2).

I use Chebyshev's inequality in a similar situation-- data that is not normally distributed, cannot be negative, and has a long tail on the high end. While there can be outliers on the low end (where mean is high and std relatively small) it's generally on the high side. We use 3 std for excluding outliers prior to doing some forecasting, so that no more than 11% of observations will be excluded per the inequality. In practice when we first implemented we found that only about 1% of observations were excluded. Chebyshev's was a nice theoretical bound to give us some justification for setting the threshold at 3 std's instead of needing to go higher.

• Thanks Chris, I came to the same conclusion but had no basis to sustain it. However the end result was that 99% circa of the observations were caught with a thick agglomerate close to the mean and mildly above. Thanks Aug 20 '18 at 19:49