2
$\begingroup$

I am analyzing the age of a certain group of people and I want to use percentiles and inter-quartile-range in the data to flag possible outliers. I am getting Q1 - 25th percentile, Q3 - 75th percentile, and inter-quartile-range, IQR = Q3 - Q1. Using the convention in analyzing a boxplot, I am going to use Q1 - 1.5IQR and Q3 + 1.5IQR as the threshold for outliers.

My question is: if the age does not follow normal distribution and is skewed, is using percentiles and IQR to detect outliers still meaningful?

If not, what would be a better approach?

Improve this question
$\endgroup$
8
  • $\begingroup$ You will find useful answers and ideas in a closely related thread (which differs from this one only by being more specific about how the data are skewed): stats.stackexchange.com/questions/13086. $\endgroup$
    – whuber ♦
    Commented Jul 10, 2015 at 18:58
  • $\begingroup$ It's not clear it's necessarily "meaningful" with normal data. What constitutes meaningfulness in this situation? $\endgroup$
    – Glen_b
    Commented Jul 12, 2015 at 11:18
  • $\begingroup$ Another fairly relevant post -- see the discussion here: stats.stackexchange.com/questions/129274/… $\endgroup$
    – Glen_b
    Commented Jul 12, 2015 at 15:52
  • $\begingroup$ @whuber Thanks for the reply. I've read that post before. I wanted to use the "whiskers" from adjusted boxplot but I don't really want the boxplots themselves. But I am not sure how to calculate the coefficients: M, alpha, beta programmatically? $\endgroup$
    – Pakwah
    Commented Jul 13, 2015 at 13:36
  • 1
    $\begingroup$ The problem with using percentiles is that it guarantees you will always find "outliers." Such values would scarcely satisfy the intuitive understanding of an outlier as being unusually different from the rest of the distribution! If you have really big data you could always subsample them randomly (or record a preselected set of quantiles) and compute stats based on the subsample, so asymptotic performance is not going to be a problem in most applications. $\endgroup$
    – whuber ♦
    Commented Jul 13, 2015 at 19:13

1 Answer 1

Reset to default
3
$\begingroup$

The boxplot "rule" (or rather rules, since Tukey had two) was (at least in a sense) 'calibrated' to the normal distribution, but was not intended as a method to explicitly identify outliers, but rather as a way of identifying points for further investigation. See Nick Cox's comment here

With skewed distributions, you'll tend to expect more observations being marked at one end than the other.

In some cases it may not even make much sense to worry about outliers at one end of a distribution.

Ultimately, what it might be meaningful to do will depend on what you're marking the points for. Why identify outliers? What's the impact of an unusually large outlier? What's the impact of an unusually small one?

Improve this answer
$\endgroup$
6
  • $\begingroup$ Use case: What I have at hand is the logging data of clinical users accessing patient information, and I want to see if any user is accessing a patient whose age is outside of the usual age range of patients accessed by this user.We do not put additional significance on whether the outlier is a large or a smaller one. But I do have another problem, which is that usually Q1- 1.5IQR / Q3 + 1.5IQR would cover all the age range, since it's usually pretty small, and the method would not mark any outliers at all. @Glen_b $\endgroup$
    – Pakwah
    Commented Jul 13, 2015 at 13:32
  • $\begingroup$ So why use that rule at all? It doesn't relate to "the usual age range of patients", so why not replace it with something that does? $\endgroup$
    – Glen_b
    Commented Jul 13, 2015 at 14:53
  • $\begingroup$ I think the IQR does represent the age range of patients being accessed by a particular group of users? Also, what are you suggesting as a replacement? $\endgroup$
    – Pakwah
    Commented Jul 13, 2015 at 15:08
  • $\begingroup$ The IQR can't represent the actual age range; it only covers the ages of about half the patients. I don't have patient data, nor a direct knowledge of how you define terms like "usual" - I can't suggest something when I don't have the required information. $\endgroup$
    – Glen_b
    Commented Jul 13, 2015 at 15:37
  • 1
    $\begingroup$ "How much the middle half is spread" doesn't tell us how far away a point must be to be outside "the usual age range of patients" -- unless you know something I do not (which you may; you have familiarity with the data and the requirements of your particular problem). Presumably whatever it is that you know will allow you to say something about how the features of the data tend to relate to whatever constitutes an outlier for your purposes. There's no good basis to judge from the information in your question. $\endgroup$
    – Glen_b
    Commented Jul 13, 2015 at 22:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.