0
$\begingroup$

There is a dataset I'm working on and there are 6 columns with continuous values which are noisy. Here is how these columns look like in term of histogram and boxplot:

lots of outliers in boxplot

As you can see, these columns are crowded with outliers. So I tried to remove these outliers and it dropped 41% of rows. I can take that much lost but the problem here is that even after this much data lost, outliers still exists:

boxplots showing outliers still exist

data is now definitely in better shape but there are outliers in data still.

The code I'm using for IQR method:

columns_with_continuous_values = ['age', 'fnlwgt', 'education_num', 'capital_gain', 'capital_loss', 'hours_per_week']
Q1 = test_df[columns_with_continuous_values].quantile(0.25)
Q3 = test_df[columns_with_continuous_values].quantile(0.75)
IQR = (Q3 - Q1)
test_df = test_df[~((test_df < (Q1 - 1.5 * IQR)) | (test_df > (Q3 + 1.5 * IQR))).any(axis=1)]

Then I tried running that above code 3 more times (4 times total) and after that, all outliers were gone, yet I lost 62% of row: boxplots showing all outliers are gone

I am fine with all these data loses but I have a feeling that I'm doing something wrong since I have to run IQR outlier removal method 4 times.

so here are my questions: Am I doing IQR right? if yes, then why I have to run it 4 times to remove all outliers? Isn't is supposed to eliminate all outliers in one run?

$\endgroup$
9
  • 4
    $\begingroup$ Who or what source recommends this method? It's both dopey and dangerous. Let's imagine a binary variable that is 80% zeros and 20% ones. Then all your ones are outliers on this criterion and should be dropped. $\endgroup$
    – Nick Cox
    May 27, 2020 at 6:45
  • 2
    $\begingroup$ There are many, many threads on outliers here, as a search of your tag will reveal. Look at the most upvoted answers and tell us if any recommends this method, which seems to come from the wilder fringes of data science. $\endgroup$
    – Nick Cox
    May 27, 2020 at 6:47
  • 1
    $\begingroup$ See e.g. this nice answer stats.stackexchange.com/questions/468423/… by @BruceET $\endgroup$
    – Nick Cox
    May 27, 2020 at 6:51
  • $\begingroup$ @NickCox I've came across a few source that recommend this method. One is: towardsdatascience.com/… You are right about binary variables. But is it still dangerous for continuous floating point variables? these 6 columns that I am inspecting are all continuous. $\endgroup$
    – ali
    May 27, 2020 at 6:53
  • $\begingroup$ It's dangerous for any variable. In many fields most variables are skewed and throwing out data points using this criterion loses genuine values that are interesting, informative and important. The only good rules for removing data points are being impossible values or irrelevant to the real problem. $\endgroup$
    – Nick Cox
    May 27, 2020 at 6:57

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.