2
$\begingroup$

I am analyzing the age of a certain group of people and I want to use percentiles and inter-quartile-range in the data to flag possible outliers. I am getting Q1 - 25th percentile, Q3 - 75th percentile, and inter-quartile-range, IQR = Q3 - Q1. Using the convention in analyzing a boxplot, I am going to use Q1 - 1.5IQR and Q3 + 1.5IQR as the threshold for outliers.

My question is: if the age does not follow normal distribution and is skewed, is using percentiles and IQR to detect outliers still meaningful?

If not, what would be a better approach?

$\endgroup$
  • $\begingroup$ You will find useful answers and ideas in a closely related thread (which differs from this one only by being more specific about how the data are skewed): stats.stackexchange.com/questions/13086. $\endgroup$ – whuber Jul 10 '15 at 18:58
  • $\begingroup$ It's not clear it's necessarily "meaningful" with normal data. What constitutes meaningfulness in this situation? $\endgroup$ – Glen_b Jul 12 '15 at 11:18
  • $\begingroup$ Another fairly relevant post -- see the discussion here: stats.stackexchange.com/questions/129274/… $\endgroup$ – Glen_b Jul 12 '15 at 15:52
  • $\begingroup$ @whuber Thanks for the reply. I've read that post before. I wanted to use the "whiskers" from adjusted boxplot but I don't really want the boxplots themselves. But I am not sure how to calculate the coefficients: M, alpha, beta programmatically? $\endgroup$ – Pakwah Jul 13 '15 at 13:36
  • 1
    $\begingroup$ The problem with using percentiles is that it guarantees you will always find "outliers." Such values would scarcely satisfy the intuitive understanding of an outlier as being unusually different from the rest of the distribution! If you have really big data you could always subsample them randomly (or record a preselected set of quantiles) and compute stats based on the subsample, so asymptotic performance is not going to be a problem in most applications. $\endgroup$ – whuber Jul 13 '15 at 19:13
2
$\begingroup$

The boxplot "rule" (or rather rules, since Tukey had two) was (at least in a sense) 'calibrated' to the normal distribution, but was not intended as a method to explicitly identify outliers, but rather as a way of identifying points for further investigation. See Nick Cox's comment here

With skewed distributions, you'll tend to expect more observations being marked at one end than the other.

In some cases it may not even make much sense to worry about outliers at one end of a distribution.

Ultimately, what it might be meaningful to do will depend on what you're marking the points for. Why identify outliers? What's the impact of an unusually large outlier? What's the impact of an unusually small one?

$\endgroup$
  • $\begingroup$ Use case: What I have at hand is the logging data of clinical users accessing patient information, and I want to see if any user is accessing a patient whose age is outside of the usual age range of patients accessed by this user.We do not put additional significance on whether the outlier is a large or a smaller one. But I do have another problem, which is that usually Q1- 1.5IQR / Q3 + 1.5IQR would cover all the age range, since it's usually pretty small, and the method would not mark any outliers at all. @Glen_b $\endgroup$ – Pakwah Jul 13 '15 at 13:32
  • $\begingroup$ So why use that rule at all? It doesn't relate to "the usual age range of patients", so why not replace it with something that does? $\endgroup$ – Glen_b Jul 13 '15 at 14:53
  • $\begingroup$ I think the IQR does represent the age range of patients being accessed by a particular group of users? Also, what are you suggesting as a replacement? $\endgroup$ – Pakwah Jul 13 '15 at 15:08
  • $\begingroup$ The IQR can't represent the actual age range; it only covers the ages of about half the patients. I don't have patient data, nor a direct knowledge of how you define terms like "usual" - I can't suggest something when I don't have the required information. $\endgroup$ – Glen_b Jul 13 '15 at 15:37
  • $\begingroup$ I understand that Q1 and Q3 includes 50% of the patients, but I think that it gives a good idea of how the ages of patients accessed by a group of users spread. $\endgroup$ – Pakwah Jul 13 '15 at 20:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.