0
$\begingroup$

I've been trying to estimate the standard deviation from a sample with N = 8149.

I've seen a few alternatives to perform this and they all use the IQR.

Here's the sample data I have:

+--------------+----------+
| 1st quartile | 1,390.61 |
+--------------+----------+
| Median       | 2,343.00 |
+--------------+----------+
| Mean         | 3,156.03 |
+--------------+----------+
| 3rd quartile | 5,704.91 |
+--------------+----------+

One of the methods I'm using says the std can be estimated using IQR/1.35, but this would result in a std of 3195.778 and a negative lower side of the confidence interval, which given the data doesn't make much sense to me.

Given that I have not only the quartiles but the mean, median and N, is there a better way to estimate the standard deviation of the sample?

Improve this question
$\endgroup$
7
  • 2
    $\begingroup$ The $IQR/1.35$ estimate of standard deviation comes from a normal distribution. Looking at your summary statistics, you have a much longer distance between the third quartile and the median than between the median and the first quartile, indicating a lack of symmetry and, thus, non-normality. Consequently, methods for standard deviation based on normal distributions should not be expected to work. // You also mention a confidence interval (CI). Why shouldn’t the CI be below zero, how are you calculating the CI, and for what value are you calculating the CI? $\endgroup$
    – Dave
    Apr 22 at 1:36
  • 1
    $\begingroup$ These values are consistent with an extremely wide range of possible standard deviations. Without some strong assumptions (e.g. assuming a specific distribution shape) you can't really eliminate those very different possibilities. You can get a lower bound on the standard deviation (looks like something roughly in the ballpark of the mid-1600s), but there isn't really an upper limit. If you add the constraint that all the observations must be positive that will change things a bit, but not enough to change the basic point (an very wide range of possible s.d.'s are consistent with those numbers $\endgroup$
    – Glen_b
    Apr 22 at 1:38
  • 1
    $\begingroup$ Reading the question again, I think the most important part of my earlier comment is asking how you calculated the confidence interval. My suspicion is that you have made the common mistake of mixing up standard deviation and standard error. $\endgroup$
    – Dave
    Apr 22 at 1:41
  • $\begingroup$ There is strong evidence here that the distribution is (materially) non-Normal, throwing into some doubt that factor of 1.35. That factor nevertheless is fairly robust to skewness, but the concern is that the data give no information about the most extreme half of the data (upper quarter and lower quarter), which could lead to extraordinarily large SDs. $\endgroup$
    – whuber ♦
    Apr 22 at 2:09
  • $\begingroup$ @Dave my calculation was a 95% CI, the mean +- 2 * the estimated standard deviation of the sample. Now I've realized I maybe should've used the standard error instead of the std, which would be s/sqrt(n) (considering the estimated std), giving a SE of around 35. If what @ whuber said about the robustness of the std estimation is ok, then would it make sense to consider this calculation? $\endgroup$
    – Caldass_
    Apr 22 at 5:10

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.