# How to detect outliers in skewed data?

I have a dataset I need to use to predict the probability of conversion based on the number of days an individual has spent using my app. I got a list of historical users and the number of session days and calculated how many of those converted. The data is left-skewed and towards the right of the dataset I have very few users. My SQL is

select
session_days,
sum(converted) converted, -- if a user converted it is 1 else 0
count(distinct user_id) users,
sum(converted) / count(distinct user_id) prob_convert
from
my_table


When I plot session_days and prob_convert I get this chart I believe the highlighted values are outliers but apart from filtering out rows > an arbitrary number of users or session_days, what is the best statistical calculation I can use to filter out these values? I tried z-score but since the data does not have a standard distribution it will not work. Also tried IQR but again the same problem. 1.5 IQR will reach above the max value and only need to filter out the right side.

PS: Skew = -1.0787; Kurt = 0.7383

EDIT: In Python I've originally filtered out the rows with a number of users below Q1 and fit a polynomial line. To identify the best fit I've used the BIC function as per below: $$BIC_{k} = n*log(SS_{\epsilon}) + k * log(n)$$

X, y = df[['session_days']], df[['prob_convert']]
n = X.shape
BIC = []
for k in range(1,100):
poly_reg = PolynomialFeatures(degree=k)
X_poly = poly_reg.fit_transform(X)
pol_reg = LinearRegression()
pol_reg.fit(X_poly, y)
df_tmp = pd.concat([pd.DataFrame(pol_reg.predict(X_poly)).rename(columns={0:"pred"}), y], axis=1)
df_tmp['res2'] = (df_tmp.prob_convert-df_tmp.pred)**2
sse = df_tmp.res2.sum()
BIC.append({"deg": k, "BIC": n * np.log(sse) + k * np.log(n)})
df_tmp = pd.DataFrame(BIC)
best_deg = df_tmp[df_tmp.BIC == df_tmp.BIC.min()].deg.values

fig = px.line(df_tmp, x='deg', y='BIC', title='Polynomial Line BIC Scores by Degrees Used')
fig.update_layout(
xaxis_title='Degree',
yaxis_title='BIC Score'
)
x=best_deg,
y=df_tmp.BIC.min(),
text=f'Degree: {best_deg}',
xref="x",
yref="y"))
fig.show()


Which returned a 6th order as best fit After fitting the line matches what I'd expect to see: But I don't believe removing outliers arbitrarily is the correct approach. After reading the comments I've looked at keeping the outliers and how to add sample_weight which would be in the pol_reg.fit(X_poly, y) line.

I've tried giving a sample_weight of np.array(range(n, 0, -1)) (less weight as session_days increases) but I don't get good results: • Why do you want to filter them out? They don't seem problematic. Dec 21, 2021 at 13:34

• Another method to give less weight to outliers is the option family="symmetric" in the R function loess. It is not well documented, but it iteratively reduces the weight of outliers according to fits from the previous iteration. Dec 21, 2021 at 13:55