0
$\begingroup$

I am running a logit regression and the "field1" is an existing list of 1's and 0's (which I am converting into a numpy list before passing as a parameter to Logit). I am trying to come up with a predicted set of 1's and O's using the Logit regression.

I read an excel file, import it into a data frame.

I assume that the y.astype(float) parameter - which is the existing 1's and 0's are needed to be passed into the Logit method (Which is the field1).

The input file contains dummy variables for a few fields. Before passing parameters to Logit, I have had to convert these parameters into float, using the following post:

https://stackoverflow.com/questions/33833832/building-multi-regression-model-throws-error-pandas-data-cast-to-numpy-dtype-o

Code:

I get the error at the end of this question - any ideas appreciated!! What am I doing wrong? My guess is that I don't need to include the Actual set of 1's and 0's because the 0's are causing the overflow?

import pandas as pd
import statsmodels.api as sm

def runLogit():
    df = pd.read_excel('InputFile.xlsx', sheetname='InputToCode')
    field1 = df['field1']
    field2 = df['field2']
    field3 = df['field3']
    field4 = df['field4']
    field5 = df['field5']
    field6 = df['field6']
    field7 = df['field7']
    field8 = df['field8']
    field9 = df['field9']
    field10 = df['field10']
    field11 = df['field11']
    field12 = df['field12']
    field13 = df['field13']

    df = pd.DataFrame({
        'field1': field1,
        'field2': field2,
        'field3': field3,
        'field4': field4,
        'field5': field5,
        'field6': field6,
        'field7': field7,
        'field8': field8,
        'field9': field9,
        'field10': field10,
        'field11': field11,
        'field12': field12,
        'field13': field13
    })

    """
    Field1 is an Actual list of 1's and 0's in the 
    input data set (which we are trying to predict 
    through the Logit).
    """
    y = df['field1'].values
    print(len(y))
    print(df.shape)

    logit_model = sm.Logit(y.astype(float), df.astype(float))
    result = logit_model.fit()
    print(result.summary())

# Init call
runLogit()

The error I am getting:

Warning: Maximum number of iterations has been exceeded.
Current function value: inf
Iterations: 35
C:\Users\xxxxx\AppData\Local\Continuum\anaconda3\lib\site-packages\statsmodels\discrete\discrete_model.py:1214: RuntimeWarning: overflow encountered in exp
return 1/(1+np.exp(-X))
C:\Users\xxxxx\AppData\Local\Continuum\anaconda3\lib\site-packages\statsmodels\discrete\discrete_model.py:1264: RuntimeWarning: divide by zero encountered in log
return np.sum(np.log(self.cdf(q*np.dot(X,params))))
Improve this question
$\endgroup$
4
  • $\begingroup$ what would be the best way to proceed? $\endgroup$
    – Utpal Mattoo
    Mar 19, 2018 at 18:11
  • $\begingroup$ here's the main stats.se thread on how to deal with perfect separation in logistic regression which is my first guess for the cause of what's happening to you: stats.stackexchange.com/questions/11109 $\endgroup$
    – jld
    Mar 19, 2018 at 18:17
  • $\begingroup$ So removing a few predictors improved the result a little bit: Here is what I get now: Warning: Maximum number of iterations has been exceeded. Current function value: 0.296861 Iterations: 35 - the best (highest) value of the Current Function Value that I get is about 0.64. I can modify the question with the current update? there were inf-act some predictors which when included give us a "perfect separation" problem. I removed those features from the list of predictors. $\endgroup$
    – Utpal Mattoo
    Mar 19, 2018 at 19:46
  • $\begingroup$ yeah you are definitely welcome to update the question as you know more about your problem. $\endgroup$
    – jld
    Mar 19, 2018 at 21:18

0

Reset to default