1
$\begingroup$

I have a dataset that follows something between a power and exponential law. I'm not happy with the IQR method of detection of outliers because on small sets of data (<50-100), it does not give you an idea of the percentage of outliers that have been droped.
I thought of an iterative method that would drop on each iteration the most outlying number until the the desired percentage is reached. I could run this method only when the IQR method drops more than the desired percentage.
Does it make sense and is there a standard way of dealing with this problem ?

$\endgroup$
  • 1
    $\begingroup$ Removing outliers is extremely complicated and everyone has a different opinion. Why do you believe you have significant outliers? Is there a way to visualize the data to facilitate the identification of outliers? $\endgroup$ – Behacad Apr 3 '13 at 21:02
  • $\begingroup$ This should be an automatized process so there isn't a way to visualize the data beforehand. The data represents reactions to information on social media. I think I won't drop anything if I have less than 20 datapoints, and I will recursively run the IQR method with a larger factor if the number of outliers is larger than 5%. $\endgroup$ – Youcha Apr 3 '13 at 21:13
  • 3
    $\begingroup$ I can't recommend any automated methods of outlier deletion. I would consider robust methods instead. $\endgroup$ – Peter Flom - Reinstate Monica Apr 3 '13 at 22:03
  • 1
    $\begingroup$ your question is ill formulated. What do you mean by the 'IQR method'? $\endgroup$ – user603 Apr 4 '13 at 8:03
  • 1
    $\begingroup$ IQR method consists in throwing datapoints that are not in [median - IQR_factor * IQR ; median + IQR_factor * IQR]. When you use this method on an exponential or power law distribution, you would not be trimming on both sides anyway because you expect the trimming threshold on the left to be smaller than 0. @PeterFlom , what do you mean by robust method? $\endgroup$ – Youcha Apr 4 '13 at 14:47
1
$\begingroup$

You should have a look at this answer. The advices therein also apply to your problem (detecting outliers in univariate settings when the good part of the data is expected to exhibit a skewed distribution).

$\endgroup$
1
$\begingroup$

Some thoughts:

  • I thought of an iterative method that would drop on each iteration the most outlying number until the the desired percentage is reached

    The non-recursive formulation of this is roughly: trim your data to certain quantiles. Whether this is a sensible approach will depend both on your data and the modeling you intend to do.

  • If outliers come as a certain percentage of data points, doesn't that suggest that you have two processes that generate fractions of the data?

  • Personally, I'm no fan of automated "outlier" removal unless it is a filtering according to knowledge about the data (biological/physical/chemical/... data generating processes) - and in that case one could argue that it should not be called outlier removal but excluding data points generated by process XY.
    However, IMHO you can trim your data as you like as long as you treat and report this step as part of your model. For example, for a predictive model that means that "reject" (e.g. NA) is a valid outcome that is produced every so often. Meaning also that you need to report the number (or fraction) of rejected cases during testing of the model.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.