1
$\begingroup$

I hope I'm using the right terms to describe my problem.

Let's say I have the following list of numbers: 1,1.6,2,2.6,3,3.7,1000000

The last number (1000000) will greatly affect the calculated average and I'd like to somehow "ignore" it when performing the calculation (as its value is very different from the rest of the values) but I don't know what is the right way to approach this issue.

Thanks !

Improve this question
$\endgroup$
3
  • $\begingroup$ 1. Your title talks about standard deviation but your body text talks about mean. Can you relate the two more clearly? 2. What may be suitable to do depends on what you're trying to achieve. If you regard the large value as being from a completely different process -- one that you aren't interested in -- it would make sense to talk about the mean of the rest, for example. $\endgroup$
    – Glen_b
    Commented Jul 14, 2017 at 2:51
  • $\begingroup$ @glen_b regarding your 2nd question - that's exactly what I'm interested in, i.e, getting the mean of the rest, as the largest value clearly doesn't belong to the data set $\endgroup$
    – Nir
    Commented Jul 14, 2017 at 20:14
  • $\begingroup$ Unless the point of the exercise is little more than exploratory, you would need some a priori criterion for this, not some post hoc rationalization of it. Or robust estimators (but choosing them only after you spot an outlier is back with post-hoc choice of procedure issues, with some impact on the properties of any inference) $\endgroup$
    – Glen_b
    Commented Jul 15, 2017 at 0:05

1 Answer 1

Reset to default
1
$\begingroup$

When one has problems like this, then one can use robust estimators. There are many robust estimation procedures out there. There are procedures like trimming, winsorizing, Huberizaton, using the median, ...

For a gentle introduction to robust estimation, see this article by Erceg-Hurn and Mirosevich http://dx.doi.org/10.1037/0003-066X.63.7.591

A more detailed follow-up is this one by Wilcox and Keselman http://dx.doi.org/10.1037/1082-989X.8.3.254

Improve this answer
$\endgroup$
4
  • $\begingroup$ Trimming / winsorizing seems exactly what I need. However, I'd like to remove the highest values only and keep the lower ones untouched. Are you familiar with such a robust estimator ? Thanks ! $\endgroup$
    – Nir
    Commented Jul 14, 2017 at 20:16
  • $\begingroup$ Huberization is a weighting system; it would weigh down extreme values. $\endgroup$
    – Heteroskedastic Jim
    Commented Jul 14, 2017 at 20:22
  • $\begingroup$ Do you know whether there's a python library which offers an implementation for huberization ? $\endgroup$
    – Nir
    Commented Jul 14, 2017 at 21:23
  • $\begingroup$ Yes, here - statsmodels.org/dev/rlm.html $\endgroup$
    – Heteroskedastic Jim
    Commented Jul 14, 2017 at 22:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.