9
$\begingroup$

I have a data set with all the calls made to an emergency service and the response times of the ambulance department. They admitted that there are some mistakes with the response times as there are cases where they didn't start recording (so the value is 0) or where they didn't stop the clock (so the value can be extremely high).

I want to find out the central tendency and I was wondering if it is better to use the median or the trimmed mean in order to get rid of the outliers?

$\endgroup$
  • 1
    $\begingroup$ Firstly, I would delete all invalid data (value=0). Then I would visualize the data with a histogram or box plot to see where I stand. Because you cannot just blindly trim the data by 5% if you have 10% of bad data... $\endgroup$ – alesc Feb 26 '15 at 14:07
  • $\begingroup$ Yeah, or plot the CDF. In R, do this: times = times[times>0]; plot(ecdf(times)) $\endgroup$ – Paul Sep 29 '15 at 16:00
11
$\begingroup$

Consider what a trimmed mean is: In the prototypical case, you first sort your data in increasing order. Then you count up to the trimming percentage from the bottom and discard those values. For example a 10% trimmed mean is common; in that case you count up from the lowest value until you've passed 10% of all the data in your set. The values below that mark are set aside. Likewise, you count down from the highest value until you've passed your trimming percentage, and set all values greater than that aside. You are now left with the middle 80%. You take the mean of that, and that is your 10% trimmed mean. (Note that you can trim unequal proportions from the two tails, or only trim one tail, but these approaches are less common and don't seem as applicable to your situation.)

Now think of what would happen if you calculated a 50% trimmed mean. The bottom half would be set aside, as would the top half. You would be left with only the single value in the middle (ordinally). You would take the mean of that (which is to say, you would just take that value) as your trimmed mean. Note however, that that value is the median. In other words, the median is a trimmed mean (it is a 50% trimmed mean). It is just a very aggressive one. It assumes, in essence, that 99% of your data are contaminated. This gives you the ultimate protection against outliers at the expense of the ultimate loss of power / efficiency.

My guess is a median / 50% trimmed mean is much more aggressive than is necessary for your data, and is too wasteful of the information available to you. If you have any sense of the proportion of outliers that exist, I would use that information to set the trimming percentage and use the appropriate trimmed mean. If you don't have any basis to choose the trimming percentage, you could select one by cross validation, or use a robust regression analysis with only an intercept.

$\endgroup$
  • 1
    $\begingroup$ I agree with the spirit of this, but it could be misread as implying that trimmed means necessarily are based on trimming equal fractions in each tail. That's just common procedure, and the procedure most often discussed for a reference case of approximately symmetric but possibly fat-tailed distributions, but it is in no sense obligatory. There is a literature on trimming in one tail only, which makes sense when all the dubious values might be in the tail. $\endgroup$ – Nick Cox Sep 29 '15 at 17:40
  • $\begingroup$ @NickCox, good point. I have added a little text to clarify that. Let me know if you think it needs more. $\endgroup$ – gung Sep 29 '15 at 17:44
  • $\begingroup$ Looks good. Naturally trimming in one tail is just that special case of unequal proportions where one proportion is zero. $\endgroup$ – Nick Cox Sep 29 '15 at 17:49
  • $\begingroup$ @NickCox, sure, but I thought it might be better to be explicit. $\endgroup$ – gung Sep 29 '15 at 17:54
-1
$\begingroup$

First of all, remove the invalid data.

Secondly, you do not need to remove the outliers as they are observed values. In some cases, it is useful (like in linear regression) but in your case I don't see the point.

Finally, prefer using the median as it is more precise to find the center of your data. As you said, the mean can be sensitive to outliers (using trimmed mean can be biased).

$\endgroup$
  • 3
    $\begingroup$ Since estimation of location is a particular case of regression, I would be curious to know how it can be useful to remove outliers in the latter but not in the former case. $\endgroup$ – user603 Jun 10 '15 at 20:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.