2
$\begingroup$

I'm working on a model for publishing in a journal. I'm removing outliers from results of my models. Suppose that I have 100 trained neural network, inserting out-of-sample data to these models and obtaining results. In this step I'm removing outliers (based on abs(x-mean(x))>= 2*s.d and use average of remaining results. How can I prove we need removing outliers in my paper? What statistically procedure or graphical presentation we need?

Update. X-axis is every out-of-sample and Y-axis is outputs for every sample. Output range is 0-1 (50 samples from 8000 samples presented in figure). In this figure green filled circles are geomean, red filled circles are averaging after removing outliers using above formula and blue filled circles are arithmetic mean. We have three outputs here.

enter image description here enter image description here enter image description here

$\endgroup$
  • 1
    $\begingroup$ "Outlines"? Do you mean outliers? $\endgroup$ – Peter Flom - Reinstate Monica Sep 11 '15 at 13:45
  • 3
    $\begingroup$ Typically removal of outliers is a no-no unless you have reason to believe that the values are truly incorrect (measured incorrectly or data entry mistake). $\endgroup$ – user30490 Sep 11 '15 at 13:49
  • 1
    $\begingroup$ However, if you want to prove you have an outlier, and you have continuous data, then boxplots are always a good visual aid. $\endgroup$ – user30490 Sep 11 '15 at 13:50
  • 2
    $\begingroup$ 2 sig limits for deviation in observations is pretty aggressive outlier removal- at that rate you're also removing ~5% of legitimate observations, assuming a normal distribution. $\endgroup$ – Thomas Cleberg Sep 11 '15 at 13:50
3
$\begingroup$

I'd say that the answer is that you can't prove that you need to remove outliers and that, in general, you shouldn't remove outliers unless you can show that they are data entry errors or some other form of error.

I don't understand exactly what you are doing, but it clearly involves averaging; why not use some form of averaging that is robust to outliers? (median, trimmed mean, Winsorized mean).

UPDATE for new edit in question:

Given that you want to somehow summarize the data shown in the boxplots, I'm now not only more sure that the arithmetic mean is not appropriate, I'm wondering if any mean is appropriate.

I'd say you have work to do before you summarize; you have to figure out what is going on. Why are some cases so narrowly distributed and others not? Why do the medians vary by a factor of about 15?

It might be that you want the geometric mean. But it might be that you want no mean at all.

$\endgroup$
  • $\begingroup$ Thank you for answer Peter. I updated the question. What do you think? we need outlier detection here? $\endgroup$ – user2991243 Sep 11 '15 at 14:11
  • $\begingroup$ Thank you for update Peter. These outputs for every sample are based one different structures of neural network trained by 10-fold-cross validation (different subsets of data). So in normal preclude we use only one neural network and get one output, but here we are averaging between 80 neural networks to get more robust results. You think the geometric mean is better here? This is a binary classification problem and these are probabilities of attendance in class 1 (Y-axis). 80 probabilities for every sample. $\endgroup$ – user2991243 Sep 11 '15 at 14:27
  • $\begingroup$ Question updated again. Any solution Peter? $\endgroup$ – user2991243 Sep 13 '15 at 7:24
  • 1
    $\begingroup$ I already gave my solution. You can't do what you seek to do. $\endgroup$ – Peter Flom - Reinstate Monica Sep 13 '15 at 12:31
2
$\begingroup$

Simply: You can't.

Also you should never assume that your data is free from gross or systematic errors (GS). Estimators which are based on least squares techniques require data which are free from GS . Only in this case they will yield the most probable parameter values [(in german), p. 115 and following ]. However this is plain theory and applies to GS with arbitrary big magnitude which is unlikely.

In practice least squares $L_2$-Norm estimators can deal with ~ 5% GS unless you're very unlucky. The presence of GS will always smear your parameter but controlled-bad observations will remain visible in residuals. You may want to check this paper how to snoop in your dataset. However: There is no way to see uncontrolled-bad observations in residuals.

Another point is model complexity. The more complex (non-linear) your model is, the more likely for you to suffer from gross and systematic errors. Suffer means that these GS will smear your parameters and won't reveal much in the residuals.

In my field I'm dealing with a large amount of data, gathered through automatic processes where I simply can't ensure that each observation is valid and not an gross error. To avoid silly questions or statements during conferences or journal reviews which claim that my dataset might have been corrupted I use robust estimators wherever it's possible.

Here is an example for line fitting. The first image is an $L_2$-Norm estimator, the second image is an robust $L_1$-Norm estimator. Both use data snooping techniques from the cited paper to remove outlier.

L2-Norm

L1-Norm

Since line fitting is a linear model with low complexity a huge outlier ratio can be found by using plain statistics. However there will be a huge drop once you enter a more complex modelling domains like neural networks.

You can 'tolerate' GS as long as you have enough data. If you don't want to keep them since you fear that they your parameter with an unacceptable magnitude then the best option is to use robust estimators with a high break point and use data snooping techniques as an argument to remove certain observations. Mention this techniques in your paper and compare your parameter before and after elimination to provide more transparent results so no one can blame you.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.