10
$\begingroup$

I have a data set with all the calls made to an emergency service and the response times of the ambulance department. They admitted that there are some mistakes with the response times as there are cases where they didn't start recording (so the value is 0) or where they didn't stop the clock (so the value can be extremely high).

I want to find out the central tendency and I was wondering if it is better to use the median or the trimmed mean in order to get rid of the outliers?

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ Firstly, I would delete all invalid data (value=0). Then I would visualize the data with a histogram or box plot to see where I stand. Because you cannot just blindly trim the data by 5% if you have 10% of bad data... $\endgroup$
    – alesc
    Commented Feb 26, 2015 at 14:07
  • $\begingroup$ Yeah, or plot the CDF. In R, do this: times = times[times>0]; plot(ecdf(times)) $\endgroup$
    – Paul
    Commented Sep 29, 2015 at 16:00

3 Answers 3

Reset to default
15
$\begingroup$

Consider what a trimmed mean is: In the prototypical case, you first sort your data in increasing order. Then you count up to the trimming percentage from the bottom and discard those values. For example a 10% trimmed mean is common; in that case you count up from the lowest value until you've passed 10% of all the data in your set. The values below that mark are set aside. Likewise, you count down from the highest value until you've passed your trimming percentage, and set all values greater than that aside. You are now left with the middle 80%. You take the mean of that, and that is your 10% trimmed mean. (Note that you can trim unequal proportions from the two tails, or only trim one tail, but these approaches are less common and don't seem as applicable to your situation.)

Now think of what would happen if you calculated a 50% trimmed mean. The bottom half would be set aside, as would the top half. You would be left with only the single value in the middle (ordinally). You would take the mean of that (which is to say, you would just take that value) as your trimmed mean. Note however, that that value is the median. In other words, the median is a trimmed mean (it is a 50% trimmed mean). It is just a very aggressive one. It assumes, in essence, that 99% of your data are contaminated. This gives you the ultimate protection against outliers at the expense of the ultimate loss of power / efficiency.

My guess is a median / 50% trimmed mean is much more aggressive than is necessary for your data, and is too wasteful of the information available to you. If you have any sense of the proportion of outliers that exist, I would use that information to set the trimming percentage and use the appropriate trimmed mean. If you don't have any basis to choose the trimming percentage, you could select one by cross validation, or use a robust regression analysis with only an intercept.

Improve this answer
$\endgroup$
4
  • 1
    $\begingroup$ I agree with the spirit of this, but it could be misread as implying that trimmed means necessarily are based on trimming equal fractions in each tail. That's just common procedure, and the procedure most often discussed for a reference case of approximately symmetric but possibly fat-tailed distributions, but it is in no sense obligatory. There is a literature on trimming in one tail only, which makes sense when all the dubious values might be in the tail. $\endgroup$
    – Nick Cox
    Commented Sep 29, 2015 at 17:40
  • $\begingroup$ @NickCox, good point. I have added a little text to clarify that. Let me know if you think it needs more. $\endgroup$
    – gung - Reinstate Monica
    Commented Sep 29, 2015 at 17:44
  • $\begingroup$ Looks good. Naturally trimming in one tail is just that special case of unequal proportions where one proportion is zero. $\endgroup$
    – Nick Cox
    Commented Sep 29, 2015 at 17:49
  • $\begingroup$ @NickCox, sure, but I thought it might be better to be explicit. $\endgroup$
    – gung - Reinstate Monica
    Commented Sep 29, 2015 at 17:54
-2
$\begingroup$

First of all, remove the invalid data.

Secondly, you do not need to remove the outliers as they are observed values. In some cases, it is useful (like in linear regression) but in your case I don't see the point.

Finally, prefer using the median as it is more precise to find the center of your data. As you said, the mean can be sensitive to outliers (using trimmed mean can be biased).

Improve this answer
$\endgroup$
1
  • 3
    $\begingroup$ Since estimation of location is a particular case of regression, I would be curious to know how it can be useful to remove outliers in the latter but not in the former case. $\endgroup$
    – user603
    Commented Jun 10, 2015 at 20:58
-2
$\begingroup$

In statistics, there are three major errors: bias, variance, and contaminations. Sample mean is a consistent estimator, however, its variance and robustness is not desired in many scenarios, so historically, many attempts have made to reduce the overall errors of mean estimation. For example, trimmed mean, Winsorized mean, Hodges-Lehmann estimator, Huber M-estimator, and median of means.

Comparing the performance of these estimators is still a hot topic in statistics research. Recent advances suggested that the bias bound of Winsorized mean is better than that of the trimmed mean (Mariusz Bieniek (2016) Comparison of the bias of trimmed and Winsorized means, Communications in Statistics - Theory and Methods, 45:22, 6641-6650, DOI: 10.1080/03610926.2014.963620 )

Also, the concentration bound of median of means nears the optimum of sub-Gaussian mean estimator (Luc Devroye. Matthieu Lerasle. Gabor Lugosi. Roberto I. Oliveira. "Sub-Gaussian mean estimators." Ann. Statist. 44 (6) 2695 - 2725, December 2016. https://doi.org/10.1214/16-AOS1440 )

Statistics is undergoing a trend from nonparametrics to semiparametrics, because these research are new, so they are not emphasizing in current textbooks.

In my paper, I defined two new classes of semiparametric distributions, introduced several new mean estimators and further explain why the Winsorized mean is better than the trimmed mean in most cases. Also, the median Hodges-Lehmann mean is proposed as the optimun nonparametric robust mean estimator. If you are interested, you can watch my youtube videos or read my paper https://www.youtube.com/@Iobiomathematics . https://zenodo.org/records/8127703 https://zenodo.org/records/6629988

$\endgroup$
1
  • 1
    $\begingroup$ It's come to our attention that you're reposting the same answer over and over to promote your research. Please refrain from this kind of self-promotion. meta.stackexchange.com/questions/57497/… $\endgroup$
    – Sycorax ♦
    Commented Feb 1 at 17:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.