2
$\begingroup$

This is definitely a very basic question but it is something which is in my head for some time now as I struggle to get the real meaning or sense of the standard deviation. How to apply it is not difficult, I guess it is quite self-explained. But how to interprete it?

The standard deviation looks as following

NN

and it is defined as

$ s = \sqrt{ \frac{1}{N-1} \sum_{i=1}^N (x_i - \bar{x} )^2}$

So finally it is meant such that ~68 % of the data falls within one standard deviation, and ~95 % within two standard deviations.

My questions are:

  1. How many standard deviations will cover all data? Infinite?
  2. When you want to detect outliers, it is quite common to define that data within 3-5 standard deviations will be inliers and the rest outliers. So due to the definition, there will always be outliers (because 3-5 sd do not cover all data). What is the sense of that?
  3. For online/streaming data the standard deviation is probably subject to change so it is hard to apply anyways sensefully.
  4. When values are measured it is also common to write it such as Y = 50 +/- 1 std. What does this mean?
  5. Why is the normal distribution finally so important?
|cite|improve this question
$\endgroup$
  • 2
    $\begingroup$ #2 Depends what you mean by "quite common". I don't see this advice in my reading. I quite commonly see advice that as outliers pull up SDs, it's a bit better to use quartiles as providing a rule of thumb. I also see advice that confident identification of outliers depends on metadata and subject-matter knowledge. $\endgroup$ – Nick Cox Nov 28 '19 at 9:40
  • 1
    $\begingroup$ Your 68% and 95% rules apply to the normal distribution and don't have to apply in any other situation. $\endgroup$ – Nick Cox Nov 28 '19 at 9:41
  • $\begingroup$ I missed these points in my answer since I've seen the plot and assumed that we talk about Normal distribution, but there is no mention of it in the text and thus, these clarifications are really important. $\endgroup$ – German Demidov Nov 28 '19 at 10:06
2
$\begingroup$

  1. Yes.

  2. There is no sense, it is just a rule of thumb. If you have 100 data points and something below/above ±3SD - it is a reason to think. If you have 100 millions of data points and have something below/above 3SD (but not e.g. ±5SD) - well, expected.

  3. May be.

  4. I think they use standard error instead of standard deviation and for me - I was always looking at this as at confidence intervals for the estimated parameter.

  5. Normal distribution has great mathematical properties and any person may handle Normally distributed data even if it is multivariate/dependent/outliers/etc etc. For many other distributions - handling of them is not so easy...I also could mention that Normal distribution actually appears in many real world applications, but this is, kind of, obvious [if we don't look at the data too close].

|cite|improve this answer
$\endgroup$
  • 1
    $\begingroup$ At first, thank you! I guess aspect 5) wonders me even more as the above question in general. I would just say there is a reason why there are so many other different distributions as well, for example the Poisson distribution (which goes into the Normal distribution somewhen when I remember right). However, when someone says "Yeah, normal distribution is applicable because we are using real world data" I always think "Ok,.. is it really true? And what otherwise?".. $\endgroup$ – Ben Nov 28 '19 at 9:41
  • 1
    $\begingroup$ "Yeah, normal distribution is applicable because we are using real world data" - this claim at my work sounds more like "Yeah, normal distribution is non applicable because we are using real world data" =) Poisson does not exactly goes to normal, but may be transformed and approximate quite closely using Anscombe transformation. In practice, in my work as a bioinformatician nothing is Normal. I use this as approximation, but knowing that I am making crimes against data doing it. $\endgroup$ – German Demidov Nov 28 '19 at 9:47
  • 1
    $\begingroup$ Oh, well, I think I forgot to mention one amazing thing in my answer - Central Limit Theorem. - en.wikipedia.org/wiki/… - this is the Real Magic $\endgroup$ – German Demidov Nov 28 '19 at 9:49
  • $\begingroup$ so then just a provoking question: Why don't you use another distribution? :) Somewhen I will study this topic more in detail resp. connect it with applied data and will decide on basis of the data I have.. somewhen.. $\endgroup$ – Ben Nov 28 '19 at 11:19
  • $\begingroup$ @Ben - there are hundreds of methods how to deal with the Normal distribution in different contexts. I can play around many situations with this, but e.g. for Negative Binomial I struggle even to correctly fit the parameters in presence of outliers. So, short answer - because I am lazy and Normal (or t-distribution in many cases) works good enough. This is a tradoff between the time you spend, solving the task, and the result you get. $\endgroup$ – German Demidov Nov 28 '19 at 11:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.