Can I turn the outlier removal problem into a model-selection problem and use cross-validation to solve it?

Take a situation similar to this question: you have a mixed model you want to fit against data. However the data-set contains a number of outliers (due to poor recording, poor explanation of the experiment to the subject and so on). Unfortunately there are many methods to remove outliers in the literature; a priori all equally arbitrary.

What I would like to do is to use cross-validation to help me select the right outlier removal protocol.

To simplify things, imagine that we decided to remove all independent variables who are $z$ standard deviations away from their mean and we have to pick $z$ between $\{2,2.5,3\}$ (this is purely as an example).
Is it possible for me to run outlier removal + regression fit on testing data and use the mean square prediction error on (unfiltered) training folds to choose $z$?

Moreover, since there is no outlier removal on the testing data, should I use a different prediction error form than sum of squares?


One approach is Frank Harrell's in his book Regression Modeling Strategies (2nd edition, 2015) where he discusses cross-validation and outlier detection (p.25 of 1st ed.) in the context of nonparametric, loess smoothers:

After making an initial estimate of the trend line, loess can look for outliers off this trend.It can then delete or downweight those apparent outliers to obtain a more robust estimate.

Harrell's treatment continues with discussion of other approaches such as Friedman's "super smoother," Hastie and Tibshirani's GAMs model and other local regression methods.

Choice of MSE vs MAD is pretty much a function of the analyst's preferences and training.

  • 1
    $\begingroup$ I really don't think the error structure of a mixed model can be managed by a loess smoother. $\endgroup$
    – CarrKnight
    Nov 11 '15 at 7:40
  • $\begingroup$ Interesting comment. I agree that its use would not make sense in the context of fitting some overall function. I think it still makes sense if done at a subset level across the hierarchical structure. I would be interested in your response to that and any elaboration in terms of published references that would support it, if you disagree. $\endgroup$ Nov 11 '15 at 14:57

My approach to a problem like this would be to express my mixed-effects model in the form of a hierarchical Bayesian model [1], and then exploit the flexibility of Bayesian models to introduce a latent 'outlier' variable that could be used to discount (or eliminate) the contribution of 'outlier' data to the model likelihood.

It appears from your OP that you have some substantive prior knowledge of what causes outliers in your data. Such knowledge could be expressed objectively in a Bayesian model of the kind I'm hinting at, thereby dealing nicely with the 'cherry picking' problem Greg Snow described in his answer to the post you linked above.

Also, if any of your priors include knowledge of directionality, then you might have a problem of censoring. But taking advantage of the residual information in a tiny handful of 'outliers' (you haven't indicated how common your outliers seem to be) would probably be more trouble than it was worth.

[1]: Gelman, Andrew, and Jennifer Hill. Data Analysis Using Regression and Multilevel/hierarchical Models. Analytical Methods for Social Research. Cambridge ; New York: Cambridge University Press, 2007.

  • $\begingroup$ not sure I will end up using this, but I like the idea of encoding explicitly what my assumptions on "outlierness" are. $\endgroup$
    – CarrKnight
    Nov 19 '15 at 10:14
  • $\begingroup$ Thx for the props, CarrKnight. The ability to freely and explicitly encode theories is for me the core motivation for using Bayesian methods. But the conceptual integrity and coherence are also important; I struggled mightily with standard frequentist conceptions/presentations of mixed effects models, only finally understanding them through a Bayesian formulation [1]. $\endgroup$ Nov 19 '15 at 14:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.