# have new outliers after capping

I'm trying to cap outliers in a column my pandas DataFrame. Here's the boxplot for a column of my original data. boxplot for a column of my original data So, using code from this stackoverflow answer, I tried capping outliers.

Here's how capped column looks like, with new outliers after upper bound. only with outliers at upper bound capped
my problem now is, i can't keep capping outliers, yes!
what do i do now.

• What is this a price of, if it is something you can disclose? Commented Aug 5, 2019 at 2:20
• price of hotels. link to data: kaggle.com/stevezhenghp/airbnb-price-prediction Commented Aug 5, 2019 at 10:21
• There is no reason to believe those are outliers as far as I can tell. You should be careful with capping because you would be surrendering information about the parameters. Commented Aug 5, 2019 at 15:13

Some distributions are inherently right-skewed, and almost all samples from them have boxplot outliers. In some cases, repeated 'capping' can seem to be never-ending.

In the following example, a simulated sample in R of size $$n=1000$$ from a standard exponential distribution (mean and rate both 1) gets capped three times, using the upper 'fence' of the boxplot outlier rule as the criterion. And there is still more capping to be done.

set.seed(1234); n = 1000
x1 = rexp(n)
cap1 = quantile(x1,.75)+1.5*IQR(x1)
x2 = x1[x1 <= cap1]
cap2 = quantile(x2,.75)+1.5*IQR(x2)
x3 = x2[x2 <= cap2]
cap3 = quantile(x3,.75)+1.5*IQR(x3)
x4 = x3[x3 <= cap3]
boxplot(x1, x2, x3, x4, col="skyblue2")
abline(h = c(cap1,cap2,cap3), col=c("red","green2","blue"))


You have to be clear about your reason for capping. If you have a good reason, maybe you have a good 'cap' to go with it. Then you could cap just once, based on a reasonable standard---not based on what data you happen to get.

In my exponential example above, about 95% of the population has values of 3 or below. Maybe just choose 3 as the cap.

Addendum: Almost all exponential samples of size only $$n=100$$ have at least one outlier and the average number of outliers among such samples is more than 4.8.

set.seed(2019)
nr.out = replicate(10^6,
length(boxplot.stats(rexp(100))\$out))
mean(nr.out); mean(nr.out > 0)
[1] 4.850963
[1] 0.990244
round(table(nr.out)/10^6, 4)
nr.out
0      1      2      3      4      5      6      7      8      9
0.0098 0.0423 0.0954 0.1470 0.1744 0.1687 0.1388 0.0985 0.0619 0.0340
10     11     12     13     14     15     16     17     18
0.0170 0.0077 0.0030 0.0011 0.0004 0.0001 0.0000 0.0000 0.0000


The distribution of the number of outliers per sample of 100 is roughly Poisson with mean 4.85.