1
$\begingroup$

Suppose I have the following hypothetical data:

  • One thousand times value 15 (i.e., 15 occurs 1000 times)
  • and a single outlier value - 115 (i.e., 115 occurs just once - an outlier)

Thus the mean is: $$((15 \times 1000) + 115)/1001 \Rightarrow (15000 + 115) / 1001 \Rightarrow 15.1$$

The standard deviation is: $3.16$.

Whereas, $Σ|x - x̄| / N$ (i.e., using the absolute deviation instead of squaring) I get: $0.20$.

I think the later value is more appropriate, but I have read in many places that one of the reasons we take the square for calculating the SD is because we want to give more weight to outliers. But I have always thought that outliers are supposed to be ignored!

What is the benefit of using the standard deviation formula (with squared deviations) which amplifies the effect of outliers (as compared to using absolute deviations)?

$\endgroup$

marked as duplicate by Ruben van Bergen, whuber Mar 1 at 14:40

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

  • 6
    $\begingroup$ 1. I did not downvote your question; however, in its current state it's not surprising to me that it would receive downvotes. 2. Since you've accused me of acting improperly, I will leave the remainder of any dealing with this question to others. 3. If you have any substantive complaint to make about our policies or my actions, the proper avenue is in our meta rather than comments. 4. I encourage you to avoid making your comments personal; you risk contravening (network-wide) policies. $\endgroup$ – Glen_b Feb 27 at 22:13
  • 7
    $\begingroup$ Although I can see that your question has something to do with standard deviation and identifying outliers, that's as far as I can get. Are you trying to ask what are some good (or at least standard) ways to detect outliers? About the sensitivity of standard deviation calculations to outliers? Something else? Whatever your aim is, please clarify it by editing your post. No amount of complaining about moderation is going to make it any clearer! $\endgroup$ – whuber Feb 28 at 0:02
  • 6
    $\begingroup$ Based on my best guess about what you are asking, the answer could be learned from this thread: Why square the difference instead of taking the absolute value in standard deviation?, possibly in conjunction with Rigorous definition of an outlier? $\endgroup$ – gung - Reinstate Monica Feb 28 at 3:18
  • 4
    $\begingroup$ "we take square...to give more weightage to outliers" This is not correct. The use of the square in calculating variance is almost never because of giving more weight to outliers (it might, however, be an intuitive way to think of it). The type of cost function, squaring, absolute difference or something else, should depend on the type of distribution that one assumes/expects/uses for the error distribution. Due to how errors are created (sum of little incremental errors), the errors often follow a normal distribution or something resembling. That is the reason for using the square. $\endgroup$ – Sextus Empiricus Feb 28 at 13:33
  • 3
    $\begingroup$ So in your hypothetical case, the use of the square might be very well inappropriate. But it depends a lot on what the data represents and what kind of statistical model would be appropriate to model it. The question is not clear about that. $\endgroup$ – Sextus Empiricus Feb 28 at 13:34
3
$\begingroup$

I have read in many places that one of the reasons we take the square for calculating the SD is because we want to give more weight to outliers.

This is not correct. The use of the square in calculating variance is almost never because of giving more weight to outliers

It might, however, be an intuitive way to think of it. I believe people more often say that the square gives more weight to values further away from the mean. But this is not to be considered outliers.


The type of cost function to apply to the difference, squaring, absolute difference or something else, should depend on the type of distribution that one assumes/expects/uses for the error distribution.

Due to how errors are often created (as a sum of many little incremental errors), the errors often follow a normal distribution or something resembling. That is the more common reason for using the square

So in your hypothetical case, with the thousand values of 15 and one of 115, the use of the square might be very well inappropriate. But... it depends a lot on what the data represents and what kind of statistical model would be appropriate to model it. The question is not clear about that.

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.