I've been studying descriptive statistics and am having a hard time understanding the actual intuition behind standard deviation. I'm trying to get a practical feeling for it and so I'm trying to draw conclusions from it using a distribution of 20 numbers, from 1 to 20. I know the mean is 10.5 and the absolute average deviation is 5, which is pretty intuitive.

Now when taking the standard deviation I get the value 5.77 which still makes some sense if I think about it as the average euclidean deviation from the mean. So I imagine adding orthogonal distances and then averaging them $\frac{\sum(x_i-\bar x)^2}{n}$ and taking the square root of that at the end to get the actual average distance. The formula makes sense from an euclidean perspective. So all that being said, my questions:

1) Why would an euclidean average distance be more accurate than an absolute deviation from the mean? I actually think absolute average deviation is more accurate since it doesn't infer any direction of the values. When taking the euclidean distance, I'm pretty much saying every value is placed at a 90° angle from each other. That does not sound right. So why the Euclidean distance? (I'm aware of this article but If someone could actually explain what efficiency is that would be very helpful: https://www.leeds.ac.uk/educol/documents/00003759.htm)

2) If The advantage of using SD is because of all the math we have developed around normal distribution shapes (68%, 95%, 99,7%...) wouldn't it be better to just rewrite that model with the new average deviation?

3) Ill probably post another question in the future about this, but when calculating standard error, this standard deviation seems to get even worse, since we need corrections for finite populations. Does this make any sense?

  • 1
    $\begingroup$ This question is very similar to Intuition behind standard deviation. See if some of some of the answers there can help you out. $\endgroup$ – user77876 Aug 15 '17 at 15:57
  • $\begingroup$ I have seen that post and pretty much all the others in stats and math stackexchange. None of them really get to the root of the issue. This is why I phrased the question a little differently trying to understand why euclidean mean is better than absolute mean, on the hopes that that might clarify the whole standard deviation formula. Also, I wrote question 2) actually thinking of the first answer in the link you gave. If the reason is because of the normal distribution shape, shouldnt we just re define the math to be more accurate? $\endgroup$ – Danilo Souza Morães Aug 15 '17 at 16:12
  • 1
    $\begingroup$ #1. What you are saying about Euclidean distances and right angles doesn't make sense. There is no 90 degree angle assumption in the Euclidean distance formula. #2. Someday you will realize how simple and beautiful the math that has been developed around normal distributions is and how much more ugly it would be if you replaced it with absolute values everywhere. $\endgroup$ – Aaron Aug 15 '17 at 16:26
  • 3
    $\begingroup$ Look for other reasons. Standard deviations are just versions of the variance. The variance is of enormous importance for three fundamental reasons, none of which is directly related to Normal distributions (or even motivated by mathematical simplicity): (1) variances are additive; (2) the variance emerges as the unique way to measure spread in the Central Limit Theorem; (3) the variance is the surrogate for many second-order differentiable loss functions, which covers a lot of ground in the decision-theoretic development of statistical theory. $\endgroup$ – whuber Aug 15 '17 at 16:28
  • $\begingroup$ Aaron, of course there is a 90 degree angle implied. Thats the calculation of an Euclidean distance. Euclidean distance is actually the distance between 2 orthogonal distances. That's simple Pythogorean theorem. I explained that in the post. Whuber, I understand the benifits of variance, but variance is simply the average Euclidean distance from the mean. Why is it more accurate than absolute average? Saying that it is better for other calculations doesn't really explain why it is more accurate. Newtonian physics is easier to calculate, but isn't accurate under great speeds... $\endgroup$ – Danilo Souza Morães Aug 15 '17 at 16:34