have new outliers after capping

Question

I'm trying to cap outliers in a column my pandas DataFrame. Here's the boxplot for a column of my original data. boxplot for a column of my original data So, using code from this stackoverflow answer, I tried capping outliers.

Here's how capped column looks like, with new outliers after upper bound. only with outliers at upper bound capped
my problem now is, i can't keep capping outliers, yes!
what do i do now.

Answer 1

Some distributions are inherently right-skewed, and almost all samples from them have boxplot outliers. In some cases, repeated 'capping' can seem to be never-ending.

In the following example, a simulated sample in R of size $n=1000$ from a standard exponential distribution (mean and rate both 1) gets capped three times, using the upper 'fence' of the boxplot outlier rule as the criterion. And there is still more capping to be done.

set.seed(1234); n = 1000
x1 = rexp(n)
 cap1 = quantile(x1,.75)+1.5*IQR(x1)
x2 = x1[x1 <= cap1]
 cap2 = quantile(x2,.75)+1.5*IQR(x2)
x3 = x2[x2 <= cap2]
 cap3 = quantile(x3,.75)+1.5*IQR(x3)
x4 = x3[x3 <= cap3]
boxplot(x1, x2, x3, x4, col="skyblue2")
 abline(h = c(cap1,cap2,cap3), col=c("red","green2","blue"))

You have to be clear about your reason for capping. If you have a good reason, maybe you have a good 'cap' to go with it. Then you could cap just once, based on a reasonable standard---not based on what data you happen to get.

In my exponential example above, about 95% of the population has values of 3 or below. Maybe just choose 3 as the cap.

Addendum: Almost all exponential samples of size only $n=100$ have at least one outlier and the average number of outliers among such samples is more than 4.8.

set.seed(2019)
nr.out = replicate(10^6, 
           length(boxplot.stats(rexp(100))$out))
mean(nr.out); mean(nr.out > 0)
[1] 4.850963
[1] 0.990244
round(table(nr.out)/10^6, 4)
nr.out
     0      1      2      3      4      5      6      7      8      9 
0.0098 0.0423 0.0954 0.1470 0.1744 0.1687 0.1388 0.0985 0.0619 0.0340 
    10     11     12     13     14     15     16     17     18 
0.0170 0.0077 0.0030 0.0011 0.0004 0.0001 0.0000 0.0000 0.0000 

The distribution of the number of outliers per sample of 100 is roughly Poisson with mean 4.85.