I have a dataset I need to use to predict the probability of conversion based on the number of days an individual has spent using my app. I got a list of historical users and the number of session days and calculated how many of those converted. The data is left-skewed and towards the right of the dataset I have very few users. My SQL is

   sum(converted) converted, -- if a user converted it is 1 else 0
   count(distinct user_id) users,
   sum(converted) / count(distinct user_id) prob_convert

When I plot session_days and prob_convert I get this chart

enter image description here

I believe the highlighted values are outliers but apart from filtering out rows > an arbitrary number of users or session_days, what is the best statistical calculation I can use to filter out these values? I tried z-score but since the data does not have a standard distribution it will not work. Also tried IQR but again the same problem. 1.5 IQR will reach above the max value and only need to filter out the right side.

PS: Skew = -1.0787; Kurt = 0.7383

EDIT: In Python I've originally filtered out the rows with a number of users below Q1 and fit a polynomial line. To identify the best fit I've used the BIC function as per below: $$ BIC_{k} = n*log(SS_{\epsilon}) + k * log(n) $$

X, y = df[['session_days']], df[['prob_convert']]
n = X.shape[0]
BIC = []
for k in range(1,100):
    poly_reg = PolynomialFeatures(degree=k)
    X_poly = poly_reg.fit_transform(X)
    pol_reg = LinearRegression()
    pol_reg.fit(X_poly, y)
    df_tmp = pd.concat([pd.DataFrame(pol_reg.predict(X_poly)).rename(columns={0:"pred"}), y], axis=1)
    df_tmp['res2'] = (df_tmp.prob_convert-df_tmp.pred)**2
    sse = df_tmp.res2.sum()
    BIC.append({"deg": k, "BIC": n * np.log(sse) + k * np.log(n)})
df_tmp = pd.DataFrame(BIC)
best_deg = df_tmp[df_tmp.BIC == df_tmp.BIC.min()].deg.values[0]

fig = px.line(df_tmp, x='deg', y='BIC', title='Polynomial Line BIC Scores by Degrees Used')
    yaxis_title='BIC Score'
                                        text=f'Degree: {best_deg}',

Which returned a 6th order as best fit

enter image description here

After fitting the line matches what I'd expect to see:

enter image description here

But I don't believe removing outliers arbitrarily is the correct approach. After reading the comments I've looked at keeping the outliers and how to add sample_weight which would be in the pol_reg.fit(X_poly, y) line.

I've tried giving a sample_weight of np.array(range(n, 0, -1)) (less weight as session_days increases) but I don't get good results:

enter image description here

  • $\begingroup$ Why do you want to filter them out? They don't seem problematic. $\endgroup$
    – Nick Cox
    Dec 21, 2021 at 13:34

1 Answer 1


If your end goal is to fit the curve, you can use robust fitting methods to give less weight to the outliers that have high residuals. In this way, these outliers will have less impact on the curve.

  • $\begingroup$ Another method to give less weight to outliers is the option family="symmetric" in the R function loess. It is not well documented, but it iteratively reduces the weight of outliers according to fits from the previous iteration. $\endgroup$
    – cdalitz
    Dec 21, 2021 at 13:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.