How to detect outliers in skewed data?

Question

I have a dataset I need to use to predict the probability of conversion based on the number of days an individual has spent using my app. I got a list of historical users and the number of session days and calculated how many of those converted. The data is left-skewed and towards the right of the dataset I have very few users. My SQL is

select
   session_days,
   sum(converted) converted, -- if a user converted it is 1 else 0
   count(distinct user_id) users,
   sum(converted) / count(distinct user_id) prob_convert
from
   my_table

When I plot session_days and prob_convert I get this chart

I believe the highlighted values are outliers but apart from filtering out rows > an arbitrary number of users or session_days, what is the best statistical calculation I can use to filter out these values? I tried z-score but since the data does not have a standard distribution it will not work. Also tried IQR but again the same problem. 1.5 IQR will reach above the max value and only need to filter out the right side.

PS: Skew = -1.0787; Kurt = 0.7383

EDIT: In Python I've originally filtered out the rows with a number of users below Q1 and fit a polynomial line. To identify the best fit I've used the BIC function as per below: $$ BIC_{k} = n*log(SS_{\epsilon}) + k * log(n) $$

X, y = df[['session_days']], df[['prob_convert']]
n = X.shape[0]
BIC = []
for k in range(1,100):
    poly_reg = PolynomialFeatures(degree=k)
    X_poly = poly_reg.fit_transform(X)
    pol_reg = LinearRegression()
    pol_reg.fit(X_poly, y)
    df_tmp = pd.concat([pd.DataFrame(pol_reg.predict(X_poly)).rename(columns={0:"pred"}), y], axis=1)
    df_tmp['res2'] = (df_tmp.prob_convert-df_tmp.pred)**2
    sse = df_tmp.res2.sum()
    BIC.append({"deg": k, "BIC": n * np.log(sse) + k * np.log(n)})
df_tmp = pd.DataFrame(BIC)
best_deg = df_tmp[df_tmp.BIC == df_tmp.BIC.min()].deg.values[0]

fig = px.line(df_tmp, x='deg', y='BIC', title='Polynomial Line BIC Scores by Degrees Used')
fig.update_layout(
    xaxis_title='Degree',
    yaxis_title='BIC Score'
)
fig.add_annotation(dict(font=dict(color='rgba(0,0,200,0.8)',size=12),
                                        x=best_deg,
                                        y=df_tmp.BIC.min(),
                                        text=f'Degree: {best_deg}',
                                        xref="x",
                                        yref="y"))
fig.show()

Which returned a 6th order as best fit

After fitting the line matches what I'd expect to see:

But I don't believe removing outliers arbitrarily is the correct approach. After reading the comments I've looked at keeping the outliers and how to add sample_weight which would be in the pol_reg.fit(X_poly, y) line.

I've tried giving a sample_weight of np.array(range(n, 0, -1)) (less weight as session_days increases) but I don't get good results:

Answer 1

If your end goal is to fit the curve, you can use robust fitting methods to give less weight to the outliers that have high residuals. In this way, these outliers will have less impact on the curve.