Iterative outlier diagnostic

Question

I am working on outlier diagnostics and I have a question about the best way to conduct them. Irrespective of the way used to define an outlier (i.e., statistical indexes, threshold), some of my colleagues argue that we just have to check the presence of outliers in our dataset once, to discard them, and to conduct our main analysis. However, it makes more sense to me to consider outlier diagnostics as an iterative procedure in which we check the presence of outliers, discard them (assuming that they are influential points of course), repeat this two steps until no outlier emerges from the dataset, and perform our main analysis. My understanding of an iterative outlier diagnostic seems consistent with Parrinello et al. (2016)'s method of iterative outlier removal.

For instance, I am using the maximum absolute deviation (MAD, e.g., Leys et al., 2013) and I chose to consider an observation as an outlier if its absolute deviation to the median of the dataset was at least equal to 3*MAD. My initial sample size was N = 36 and I detected three outliers. According to my colleagues, I should have discarded these three observations and conducted my main analysis without checking whether I detected new outliers in my reduced sample (N = 33). Their main argument for doing so is that I would have detected too many outliers at the end of my iterative diagnostic. However, it does not make sense to me to check the presence of outliers only once. So, I checked iteratively until I found no outlier according to my initial threshold (i.e., an absolute deviation to the median of the dataset at least equal to 3*MAD). I noticed that conducting my diagnostic iteratively was almost equivalent to conducting only one diagnostic with a less conservative threshold (i.e., using 2*MAD instead of 3*MAD). Maybe conducting only one diagnostic with a less conservative threshold would be a better idea to save time than using an iterative diagnostic with a more conservative threshold. What do you think about that?

I know that discarding outliers is not necessarily the best way to manage outliers and that using robust method would certainly be a better alternative. However, I am not familiar with robust methods and I still have to analyze my data. In addition, although Fox (1991, p. 40) advised against mindless outlier deletion and argued in favor of robust methods, he also underlined that conducting non-robust analyses with a thoughtful outlier deletion would be almost equivalent to conducting robust analyses in which all outliers are downweighted to 0.

Thanks in advance for your thoughts and advice

References

Fox, J. (1991). Regression Diagnostics: An Introduction. Newbury Park, CA: SAGE Publications.

Leys, C., Ley, C., Klein, O., Bernard, P., & Licata, L. (2013). Detecting outliers: Do not use standard deviation around the mean, use absolute deviation around the median. Journal of Experimental Social Psychology, 49(4), 764‑766. https://doi.org/10.1016/j.jesp.2013.03.013

Parrinello, C. M., Grams, M. E., Sang, Y., Couper, D., Wruck, L. M., Li, D., … Coresh, J. (2016). Iterative Outlier Removal: A Method for Identifying Outliers in Laboratory Recalibration Studies. Clinical Chemistry, 62(7), 966‑972. https://doi.org/10.1373/clinchem.2016.255216

Answer 1

I am going to first voice my disagreement with removing correctly-measured outliers at all, and then answer the substance of your question on the assumption that you are committed to the practice of removing outliers, and want to know the least bad way to do this. I agree with you that your practice of doing outlier tests with each re-fit is the correct approach under this latter practice.

---

Don't remove correctly-measured outliers - adjust your model instead: Identification of an "outlier" in a dataset can indicate that the observation was incorrectly recorded (which is a form of measurement error). Outlier tests can be useful for this, and if you find an observation that has been incorrectly recorded, it is reasonable to remove that point from your analysis, since it does not represent a genuine value. However, it is not generally a good idea to remove "outliers" that are genuine observations, that have been correctly recorded.

Remember that the way a statistical model identifies an "outlier" is by comparing the data points to the posited distributional form in the model, and flagging values that are too far in the tails of this distribution. (Outlier tests show us that one or more values are so far in the tails of the distribution that this distribution is not really plausible.) Identification of an outlier by this method tells us that the data point probably did not come from that distribution/model. Now, logically, we should consider this to be a shortcoming of the model (i.e., it does not have fat enough tails to accurately represent the data). Instead, many analysts seem to treat this as a shortcoming of reality, and instead require that reality conform to their model. Data points that are non-conforming to the model are removed, and --voila!-- all is well. This is what you are doing when you make a practice of removing outliers.

Rather than removing correctly-measured outliers, you are much better off altering your model to allow a distribution with fatter tails. For example, if you have a model that uses a normal distribution for underlying errors, you should consider replacing that with a distribution that has fatter tails, like a generalised error distribution, or a T-distribution. Flagging of "outliers" in outlier tests really just means that the model distribution does not have fat enough tails to accurately represent the data. The solution is to change the model accordingly.

---

If you are removing outliers, it makes sense to do this multiple times: If you decide to remove outliers, on the basis that they are flagged in outlier tests, you are effectively reducing your data to remove any data points that are too far in the tails compared to the distributional form imposed in your model. This means that your model predictions speak only to the bulk of the data that is not in the tails, and your results should be presented accordingly, with appropriate caveats.

The reason to remove outliers, if you decide you must do this, is to stick with your existing model, and apply it only to that subset of the data that it accurately describes. Now, if you are doing this, the purpose is to end up with a model that passes outlier tests so that the model can be regarded as a reasonable representation of the subset of non-outlying data that has not been removed.

A single iteration of outlier-removal changes the data set being used in the analysis, which means you must re-fit the model, which can then lead other data points to now be shown as outliers. In that case, your model still does not pass outlier tests, and so your model does not adequately describe the data it is purporting to describe. For this reason, I agree with your view that you should be running the outlier-detection iteratively, after each re-fit of the model, and you should continue to remove outliers until your model passes the outlier test. At this point, you can say that the model is a reasonable representation of the remaining non-outlying data.

Of course, this might lead you to end up removing a lot of data points. That is bad, since it reduces the scope of what you are describing. However, realistically, this is a product of the fact that you are removing outliers in the first place, and are reducing the scope of the data to fit to your model, rather than altering your model to allow it to fit to the entire data set.

---