# Dealing with outlier data causing a non-linearity for logistic regression model

I'm building a logistic regression model where yes is the target, one of the attributes spnr_avg_spend_mod is shown below. It is a continuous variable but has been binned up into 10 bins using pd.qcut.

As shown in the image below, apart from the first data point there is a linear decrease in the proportion of yes with increasing spend (increasing spnr_avg_spend_mod).

I am looking for ideas/methods on how to deal to this when modelling as it is a nice feature apart from that point.

In : df
Out:
spnr_avg_spend_mod       yes
0                   0  0.474293
1                   1  0.531138
2                   2  0.533260
3                   3  0.503260
4                   4  0.503418
5                   5  0.482936
6                   6  0.479729
7                   7  0.460062
8                   8  0.450755
9                   9  0.421202

In : plt.scatter(df.spnr_avg_spend_mod, df.yes)
Out: <matplotlib.collections.PathCollection at 0x7f80962873c8> • What does it mean to "deal with" the value of yes at spnr_avg_spend_mod=0? One model would just draw a straight line between neighboring points. This would hit every point exactly, but the function would not be very smooth. Is this a valid way to "deal with" this problem? Why or why not? In other words, what problem are you trying to solve?
– Sycorax
Jan 24, 2020 at 16:43
• You may be interested in this question. stats.stackexchange.com/questions/63978/…
– Sycorax
Jan 24, 2020 at 17:08

Binning a continuous IV is almost always a mistake. See Frank Harrell's book Regression Modeling Strategies where he lists 11 problems with this and sums up "Nothing could be more disastrous". Leave the IV continuous and then you can try using a spline of it as a predictor.

Since there isn't a textbook solution to what you're asking, I'll offer a few considerations:

1. How do you know that data point is an outlier? Do you know for certain that your trend must be linear? There are numerous phenomena that are non-linear. Your logistic regression will pick that up if it's true.
2. Have you tried using a different number of bins? Do you know for a fact that this feature should be cut into deciles? For example, if you had 3 or 4 bins, there's a good chance that the resulting bins would be linear. Similarly, if you had 20 bins, then perhaps only the first bin or two would be lower than expected, which helps identify the cause of a potential outlier.
3. Related to 2., in my experience data on the edges (e.g., first and last bin) can behave differently than the middle because of their proximity to the borders. That is, the first bin may include 0's or negatives which may have unexpected results on your output variable. Have you done the necessary data cleaning required?
• It isn't an outlier, If I go to 20 bins there is a similar trend. I imagine that if i was to go to 3 bins it might be linear but would be loosing out on a bit too much info. I would assume that since logititic regression is a linear model, it works best with linear features? Perhaps another solution would be to linearise it, i.e. fit a 3rd order polynomial to it. May 1, 2018 at 8:41
• If you don't think this is a data issue, then yes, transforming using a 3rd order polynomial seems reasonable. Just beware of potential overfitting. May 3, 2018 at 0:33
• Please see my answer with 3rd order polynomial compared to an equation search result. I would place it here, but cannot put images in a comment. Sep 26, 2018 at 15:33

This is my comparison of a third order polynomial and a different equation from an equation search, placed here as I cannot display images in the comments. The third order polynomial does not visually appear to fit the shape of the data as well, especially at the higher end.

First the third order polynomial: import math
def Polynomial_Cubic_model(x_in): # from zunzun.com
temp = 0.0

# coefficients
a = 4.8789645034964935E-01
b = 3.1491002913752966E-02
c = -8.4664254079254170E-03
d = 4.7255536130533238E-04

temp += a + b * x_in + c * math.pow(x_in, 2.0) + d * math.pow(x_in, 3.0)
return temp


and here the equation search result: import math
def Simple_SimpleEquation_18_model(x_in): # from zunzun.com
temp = 0.0

# coefficients
a = 4.7730833313413012E-01
b = -6.6062232805066859E-02
c = 1.5999550517794348E-01

temp = a*math.exp(b*x_in+c*math.pow(x_in,0.5))
return temp