# Reducing standard deviation by removing "noisy" items

I hope I'm using the right terms to describe my problem.

Let's say I have the following list of numbers: 1,1.6,2,2.6,3,3.7,1000000

The last number (1000000) will greatly affect the calculated average and I'd like to somehow "ignore" it when performing the calculation (as its value is very different from the rest of the values) but I don't know what is the right way to approach this issue.

Thanks !

• 1. Your title talks about standard deviation but your body text talks about mean. Can you relate the two more clearly? 2. What may be suitable to do depends on what you're trying to achieve. If you regard the large value as being from a completely different process -- one that you aren't interested in -- it would make sense to talk about the mean of the rest, for example. Commented Jul 14, 2017 at 2:51
• @glen_b regarding your 2nd question - that's exactly what I'm interested in, i.e, getting the mean of the rest, as the largest value clearly doesn't belong to the data set
– Nir
Commented Jul 14, 2017 at 20:14
• Unless the point of the exercise is little more than exploratory, you would need some a priori criterion for this, not some post hoc rationalization of it. Or robust estimators (but choosing them only after you spot an outlier is back with post-hoc choice of procedure issues, with some impact on the properties of any inference) Commented Jul 15, 2017 at 0:05

When one has problems like this, then one can use robust estimators. There are many robust estimation procedures out there. There are procedures like trimming, winsorizing, Huberizaton, using the median, ...

For a gentle introduction to robust estimation, see this article by Erceg-Hurn and Mirosevich http://dx.doi.org/10.1037/0003-066X.63.7.591

A more detailed follow-up is this one by Wilcox and Keselman http://dx.doi.org/10.1037/1082-989X.8.3.254

• Trimming / winsorizing seems exactly what I need. However, I'd like to remove the highest values only and keep the lower ones untouched. Are you familiar with such a robust estimator ? Thanks !
– Nir
Commented Jul 14, 2017 at 20:16
• Huberization is a weighting system; it would weigh down extreme values. Commented Jul 14, 2017 at 20:22
• Do you know whether there's a python library which offers an implementation for huberization ?
– Nir
Commented Jul 14, 2017 at 21:23
• Yes, here - statsmodels.org/dev/rlm.html Commented Jul 14, 2017 at 22:20