# Iterative outlier diagnostic

I am working on outlier diagnostics and I have a question about the best way to conduct them. Irrespective of the way used to define an outlier (i.e., statistical indexes, threshold), some of my colleagues argue that we just have to check the presence of outliers in our dataset once, to discard them, and to conduct our main analysis. However, it makes more sense to me to consider outlier diagnostics as an iterative procedure in which we check the presence of outliers, discard them (assuming that they are influential points of course), repeat this two steps until no outlier emerges from the dataset, and perform our main analysis. My understanding of an iterative outlier diagnostic seems consistent with Parrinello et al. (2016)'s method of iterative outlier removal.

For instance, I am using the maximum absolute deviation (MAD, e.g., Leys et al., 2013) and I chose to consider an observation as an outlier if its absolute deviation to the median of the dataset was at least equal to 3*MAD. My initial sample size was N = 36 and I detected three outliers. According to my colleagues, I should have discarded these three observations and conducted my main analysis without checking whether I detected new outliers in my reduced sample (N = 33). Their main argument for doing so is that I would have detected too many outliers at the end of my iterative diagnostic. However, it does not make sense to me to check the presence of outliers only once. So, I checked iteratively until I found no outlier according to my initial threshold (i.e., an absolute deviation to the median of the dataset at least equal to 3*MAD). I noticed that conducting my diagnostic iteratively was almost equivalent to conducting only one diagnostic with a less conservative threshold (i.e., using 2*MAD instead of 3*MAD). Maybe conducting only one diagnostic with a less conservative threshold would be a better idea to save time than using an iterative diagnostic with a more conservative threshold. What do you think about that?

I know that discarding outliers is not necessarily the best way to manage outliers and that using robust method would certainly be a better alternative. However, I am not familiar with robust methods and I still have to analyze my data. In addition, although Fox (1991, p. 40) advised against mindless outlier deletion and argued in favor of robust methods, he also underlined that conducting non-robust analyses with a thoughtful outlier deletion would be almost equivalent to conducting robust analyses in which all outliers are downweighted to 0.

References

Fox, J. (1991). Regression Diagnostics: An Introduction. Newbury Park, CA: SAGE Publications.

Leys, C., Ley, C., Klein, O., Bernard, P., & Licata, L. (2013). Detecting outliers: Do not use standard deviation around the mean, use absolute deviation around the median. Journal of Experimental Social Psychology, 49(4), 764‑766. https://doi.org/10.1016/j.jesp.2013.03.013

Parrinello, C. M., Grams, M. E., Sang, Y., Couper, D., Wruck, L. M., Li, D., … Coresh, J. (2016). Iterative Outlier Removal: A Method for Identifying Outliers in Laboratory Recalibration Studies. Clinical Chemistry, 62(7), 966‑972. https://doi.org/10.1373/clinchem.2016.255216

• I think this question is really hard to answer generally, because outliers tend to manifest themselves in a problem-dependent way. What may be considered an outlier for one application may not be an outlier for another. Moreover, the criteria for declaring an outlier may change depending on the instance of a particular problem, depending on your tolerance for outliers. It sounds like the definition of an outlier is somewhat arbitrary for your problem, so neither you nor your colleagues are absolutely right. Does your analysis change significantly depending on your outlier rejection criteria? Jun 11, 2018 at 13:57
• Thanks for your comment. Even if an outlier can be defined, in broad terms, as an observation that is far from the rest of the data, I agree that there are some practical issues when operationalizing this concept. I also acknowledge that all outliers have not necessarily a disproportionate influence on the parameter estimates. However, my question is more about the rationale of the outlier diagnostic rather than on how to define an outlier. Jun 12, 2018 at 8:13
• As underlined by Parrinello et al. (2016, see also Courvoisier & Renaud, 2011, p. 85), it makes sense to check the presence of outliers iteratively rather than only once. As the results of my analysis change after outlier deletion, it might be good to present the two analyses (with and without the outliers). However, it seems that this approach is not so good (Courvoisier & Renaud, 2011; Field and Wilcox 2017) and that it would be better to conduct robust analyses rather than classical analysis. Jun 12, 2018 at 8:13
• I would argue that the rationale of the outlier diagnostic, as you put it, should follow the definition of an outlier. In my opinion, if you cannot concretely define an outlier for your application, then it is hard to justify an arbitrary outlier removal method. You are right, robust analysis might be your friend here. But regarding iterative vs. one-and-done outlier removal, it sounds like you have made up your mind as to what you want to do, so I wish you luck in defending your solution to your colleagues! Jun 12, 2018 at 11:40
• Thanks for your comment. However, I am not so obstinate in doing an iterative diagnostic as one might think. Indeed, in my case, the iterative diagnostic would result in a greater number of outliers than the one-and-done diagnostic, which is not so good for my sample size if I decide to exclude all outliers. Thus I have no personal interest of being right compared to my colleagues. Rather, I just want to be able to make an informed decision on the best way to deal with my outliers in the case where I would like to delete them (otherwise the robust methods would certainly be better). Jun 13, 2018 at 9:50

I am going to first voice my disagreement with removing correctly-measured outliers at all, and then answer the substance of your question on the assumption that you are committed to the practice of removing outliers, and want to know the least bad way to do this. I agree with you that your practice of doing outlier tests with each re-fit is the correct approach under this latter practice.

Don't remove correctly-measured outliers - adjust your model instead: Identification of an "outlier" in a dataset can indicate that the observation was incorrectly recorded (which is a form of measurement error). Outlier tests can be useful for this, and if you find an observation that has been incorrectly recorded, it is reasonable to remove that point from your analysis, since it does not represent a genuine value. However, it is not generally a good idea to remove "outliers" that are genuine observations, that have been correctly recorded.

Remember that the way a statistical model identifies an "outlier" is by comparing the data points to the posited distributional form in the model, and flagging values that are too far in the tails of this distribution. (Outlier tests show us that one or more values are so far in the tails of the distribution that this distribution is not really plausible.) Identification of an outlier by this method tells us that the data point probably did not come from that distribution/model. Now, logically, we should consider this to be a shortcoming of the model (i.e., it does not have fat enough tails to accurately represent the data). Instead, many analysts seem to treat this as a shortcoming of reality, and instead require that reality conform to their model. Data points that are non-conforming to the model are removed, and --voila!-- all is well. This is what you are doing when you make a practice of removing outliers.

Rather than removing correctly-measured outliers, you are much better off altering your model to allow a distribution with fatter tails. For example, if you have a model that uses a normal distribution for underlying errors, you should consider replacing that with a distribution that has fatter tails, like a generalised error distribution, or a T-distribution. Flagging of "outliers" in outlier tests really just means that the model distribution does not have fat enough tails to accurately represent the data. The solution is to change the model accordingly.

If you are removing outliers, it makes sense to do this multiple times: If you decide to remove outliers, on the basis that they are flagged in outlier tests, you are effectively reducing your data to remove any data points that are too far in the tails compared to the distributional form imposed in your model. This means that your model predictions speak only to the bulk of the data that is not in the tails, and your results should be presented accordingly, with appropriate caveats.

The reason to remove outliers, if you decide you must do this, is to stick with your existing model, and apply it only to that subset of the data that it accurately describes. Now, if you are doing this, the purpose is to end up with a model that passes outlier tests so that the model can be regarded as a reasonable representation of the subset of non-outlying data that has not been removed.

A single iteration of outlier-removal changes the data set being used in the analysis, which means you must re-fit the model, which can then lead other data points to now be shown as outliers. In that case, your model still does not pass outlier tests, and so your model does not adequately describe the data it is purporting to describe. For this reason, I agree with your view that you should be running the outlier-detection iteratively, after each re-fit of the model, and you should continue to remove outliers until your model passes the outlier test. At this point, you can say that the model is a reasonable representation of the remaining non-outlying data.

Of course, this might lead you to end up removing a lot of data points. That is bad, since it reduces the scope of what you are describing. However, realistically, this is a product of the fact that you are removing outliers in the first place, and are reducing the scope of the data to fit to your model, rather than altering your model to allow it to fit to the entire data set.

• Thanks for your answer Ben. I am working in a field where removing outliers is only one of the bad statistical habits. As you underlined, there are far better alternatives to outlier removal and I am working in this direction. However, I would like to better understand the logic of removal. Your answer makes sense to me and is great. Have you some references in support to the iterative outlier removal? Jun 18, 2018 at 14:00
• I am not aware of any specific references for the argument I have made. References for the outlier tests themselves would be sufficient here, since failure of the test constitutes evidence of an outlier. Hence, if you remove outliers, re-fit your model, and get new outliers that you choose not to remove, then you are still in a position where you are "failing" the outlier test, as described in standard references.
– Ben
Jun 19, 2018 at 0:17