how we can prove we have outliers in the results?

Question

I'm working on a model for publishing in a journal. I'm removing outliers from results of my models. Suppose that I have 100 trained neural network, inserting out-of-sample data to these models and obtaining results. In this step I'm removing outliers (based on abs(x-mean(x))>= 2*s.d and use average of remaining results. How can I prove we need removing outliers in my paper? What statistically procedure or graphical presentation we need?

Update. X-axis is every out-of-sample and Y-axis is outputs for every sample. Output range is 0-1 (50 samples from 8000 samples presented in figure). In this figure green filled circles are geomean, red filled circles are averaging after removing outliers using above formula and blue filled circles are arithmetic mean. We have three outputs here.

Answer 1

I'd say that the answer is that you can't prove that you need to remove outliers and that, in general, you shouldn't remove outliers unless you can show that they are data entry errors or some other form of error.

I don't understand exactly what you are doing, but it clearly involves averaging; why not use some form of averaging that is robust to outliers? (median, trimmed mean, Winsorized mean).

UPDATE for new edit in question:

Given that you want to somehow summarize the data shown in the boxplots, I'm now not only more sure that the arithmetic mean is not appropriate, I'm wondering if any mean is appropriate.

I'd say you have work to do before you summarize; you have to figure out what is going on. Why are some cases so narrowly distributed and others not? Why do the medians vary by a factor of about 15?

It might be that you want the geometric mean. But it might be that you want no mean at all.

Answer 2

Simply: You can't.

Also you should never assume that your data is free from gross or systematic errors (GS). Estimators which are based on least squares techniques require data which are free from GS . Only in this case they will yield the most probable parameter values [(in german), p. 115 and following ]. However this is plain theory and applies to GS with arbitrary big magnitude which is unlikely.

In practice least squares $L_2$-Norm estimators can deal with ~ 5% GS unless you're very unlucky. The presence of GS will always smear your parameter but controlled-bad observations will remain visible in residuals. You may want to check this paper how to snoop in your dataset. However: There is no way to see uncontrolled-bad observations in residuals.

Another point is model complexity. The more complex (non-linear) your model is, the more likely for you to suffer from gross and systematic errors. Suffer means that these GS will smear your parameters and won't reveal much in the residuals.

In my field I'm dealing with a large amount of data, gathered through automatic processes where I simply can't ensure that each observation is valid and not an gross error. To avoid silly questions or statements during conferences or journal reviews which claim that my dataset might have been corrupted I use robust estimators wherever it's possible.

Here is an example for line fitting. The first image is an $L_2$-Norm estimator, the second image is an robust $L_1$-Norm estimator. Both use data snooping techniques from the cited paper to remove outlier.

Since line fitting is a linear model with low complexity a huge outlier ratio can be found by using plain statistics. However there will be a huge drop once you enter a more complex modelling domains like neural networks.

You can 'tolerate' GS as long as you have enough data. If you don't want to keep them since you fear that they your parameter with an unacceptable magnitude then the best option is to use robust estimators with a high break point and use data snooping techniques as an argument to remove certain observations. Mention this techniques in your paper and compare your parameter before and after elimination to provide more transparent results so no one can blame you.