0
$\begingroup$

I want to use two basic rules of thumb for simple outlier detection/removal of a one dimensional dataset.

I will apply both the 3 sigma rule and 3 * IQR.

I understand that the 3 * IQR rule is more robust for asymmetrical distributions. I don't want to throw away too much data on a skewed dataset due to the 3 sigma rule.

Is there any general rule where I can first determine the skewness or kurtosis of the dataset before deciding whether to apply the 3 sigma rule in addition to the 3 * IQR rule?

Improve this question
$\endgroup$
2
  • $\begingroup$ Although I support e.g. the general stance of @Peter Westfall here, the question seems contradictory in its own terms. First, don't you expect more outliers in skewed distributions? Second, if you have any notion whatsoever of throwing away too much data, then you have an idea of what kind of analysis is good for your data that should give you a better idea of how to proceed. (A rule of thumb on which rule of thumb to use???) $\endgroup$
    – Nick Cox
    Commented Apr 18, 2017 at 11:14
  • $\begingroup$ Your tagging should indicate many, many other threads here that are pertinent. You can't read them all, but some of the highest voted threads should give other and (I personally believe) better guidance. $\endgroup$
    – Nick Cox
    Commented Apr 18, 2017 at 11:20

1 Answer 1

Reset to default
1
$\begingroup$

If you have outliers, they are probably telling you something very interesting. Why throw them out? You will just be losing valuable information.

Sure, if they are mistakes, then delete them. But the mechanism that made them mistakes likely made some non-outlier data values mistakes too, so you should scour the entire data set for errors, not just the extremes.

If you have a system that produces occasional outliers, a better scientific model than the "delete outliers then assume normal distribution" model, is a model that likewise produces outliers; eg, lognormal, Student T, Pareto, etc. Sometimes outliers are a result of heteroscedasticity, so that is another modeling option.

If you model your system in this way, not only will you have a better scientific model for your data-generating process, but the estimation procedure (likelihood, Bayes) will automatically down-weight the outliers.

And if you want to be lazy and not model the scientific process explicitly, you can always use one of the gazillion estimation methods that are robust to outliers. Quantile regression comes to mind.

It is hard to defend "outlier deletion just because they are outliers" as a sound general statistical practice.

Improve this answer
$\endgroup$
1
  • $\begingroup$ I think this is very good general advice, but a little dig at methods you don't prefer as "lazy" is not helpful. For example, there are plenty of positive reasons to want to use quantile regression, including some in which postulating a specific distribution is either impractical or irrelevant. $\endgroup$
    – Nick Cox
    Commented Apr 18, 2017 at 11:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.