0
$\begingroup$

My data is discrete and has the following distribution:

P(1) = 0.45, P(2) = 0.5, P(3) = 0.02, P(> 3) = 0.02

I want to remove outliers systematically, given the distribution and the fact that 1 is the lower bound. When I apply Tukey's 1.5 IQR rule and mean +- 2*sd rules, they both result in 3 being included, which intuitively doesn't seem right because 1 and 2 encompass 95% of the data. Are there any methods specifically for discrete data bounded on one side?

$\endgroup$
6
$\begingroup$

Precisely why do you want to remove outliers here?

Just because there are (relative) outliers need not mean that you must remove them from the data.

The 4% of the values with 3 or more might carry much of the interesting or important information.

The Tukey rule you cite was never more than a suggestion for identifying values you should think about, not as an infallible outlier rule (there are no such rules).

If you are worried about the influence of outliers, a log transformation looks an easy approach for you, given the implication that values must be at least 1.

$\endgroup$
2
  • $\begingroup$ I am building a model for the arrival times of an event, and the data described is one of multiple descriptive features. In my sample, I've observed that at every occurrence of the event, the feature described is 1 or 2. Instead of filtering outliers in the feature in isolation as originally described, is there a way to formalize the notion: "Since the events only happen when the feature is 1 or 2, I want to ignore all other values of the feature." I understand that this changes the nature of my question; please let me know if this could be articulated better, I'll re-word the question. $\endgroup$
    – tmakino
    Jun 12 '13 at 19:29
  • 1
    $\begingroup$ If you have a reason to branch on 1 or 2 versus other values, you should do so. However, you have indicated that pattern is just observed in your sample, not guaranteed, so there is more to be said. But I don't think you have given us enough information to allow us to add statistical advice. $\endgroup$
    – Nick Cox
    Jun 12 '13 at 22:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.