0
$\begingroup$

I am running a logit regression and the "field1" is an existing list of 1's and 0's (which I am converting into a numpy list before passing as a parameter to Logit). I am trying to come up with a predicted set of 1's and O's using the Logit regression.

I read an excel file, import it into a data frame.

I assume that the y.astype(float) parameter - which is the existing 1's and 0's are needed to be passed into the Logit method (Which is the field1).

The input file contains dummy variables for a few fields. Before passing parameters to Logit, I have had to convert these parameters into float, using the following post:

https://stackoverflow.com/questions/33833832/building-multi-regression-model-throws-error-pandas-data-cast-to-numpy-dtype-o

Code:

I get the error at the end of this question - any ideas appreciated!! What am I doing wrong? My guess is that I don't need to include the Actual set of 1's and 0's because the 0's are causing the overflow?

 import pandas as pd
    import statsmodels.api as sm                                                                                                                                                                        
                                            def runLogit():                                                                                                                  

                                                   df = pd.read_excel('InputFile.xlsx', sheetname='InputToCode')                                                               

                                                   field1    = df['field1']                                                                            
                                                   field2    = df['field2']                                                                                          
                                                   field3    = df['field3']                                                                                         
                                                   field4    = df['field4']                                                                                      
                                                   field5      = df['field5']                                                                                           
                                                   field6     = df['field6']                                                                                          
                                                   field7    = df['field7']                                                                                       
                                                   field8    = df['field8']                                                                                        
                                                   field9    = df['field9']                                                                       

                                                   field10    = df['field10']                                                                           
                                                   field11e  = df['field11']                                                                           
                                                   field12  = df['field12']                                                                             
                                                   field13 = df['field13']                                                                                 

                                                   df = pd.DataFrame(                                                                                                        
                                                                      {                                                                                                      
                                                                       'field1': field1,                                                           
                                                                       'field2': field2,                                                                      
                                                                       'field3': field3,                                                                     
                                                                       'field4': field4,                                                             
                                                                       'field5': field5,                                                                       
                                                                       'field6': field6,                                                                      
                                                                       'field7': feild7,                                                                 
                                                                       'field8': field8,                                                                 
                                                                       'field9': field9,                                                       
                                                                       'field10': field10,                                                            
                                                                       'field11': field11,                                                             
                                                                       'field12': field12,                                                                
                                                                       'field13':   field13                                                                        
                                                                       }                                                                                                     
                                                                    )                                                                                                        

                                          # Field1 is an Actual list of 1's and 0's in the input data set (which we are trying to predict through the Logit)                                                                                                                              
                                                   y = df['field1'].values                                                                                      

                                                   print (len(y))                                                                                                            
                                                   print (df.shape)                                                                                                          
                                                   logit_model = sm.Logit(y.astype(float), df.astype(float))                                                                 
                                                   result = logit_model.fit()                                                                                                
                                                   print (result.summary())                                                                                                  




                                            #==============================================================================                                                  
                                            # Initial call                                                                                                                   
                                            #==============================================================================                                                  

                                            runLogit()  


        The error I am getting is:

        Warning: Maximum number of iterations has been exceeded.
             Current function value: inf
             Iterations: 35
    C:\Users\xxxxx\AppData\Local\Continuum\anaconda3\lib\site-packages\statsmodels\discrete\discrete_model.py:1214: RuntimeWarning: overflow encountered in exp
      return 1/(1+np.exp(-X))
    C:\Users\xxxxx\AppData\Local\Continuum\anaconda3\lib\site-packages\statsmodels\discrete\discrete_model.py:1264: RuntimeWarning: divide by zero encountered in log
      return np.sum(np.log(self.cdf(q*np.dot(X,params))))
$\endgroup$
  • $\begingroup$ what would be the best way to proceed? $\endgroup$ – User1 Mar 19 '18 at 18:11
  • $\begingroup$ here's the main stats.se thread on how to deal with perfect separation in logistic regression which is my first guess for the cause of what's happening to you: stats.stackexchange.com/questions/11109 $\endgroup$ – jld Mar 19 '18 at 18:17
  • $\begingroup$ So removing a few predictors improved the result a little bit: Here is what I get now: Warning: Maximum number of iterations has been exceeded. Current function value: 0.296861 Iterations: 35 - the best (highest) value of the Current Function Value that I get is about 0.64. I can modify the question with the current update? there were inf-act some predictors which when included give us a "perfect separation" problem. I removed those features from the list of predictors. $\endgroup$ – User1 Mar 19 '18 at 19:46
  • $\begingroup$ yeah you are definitely welcome to update the question as you know more about your problem. $\endgroup$ – jld Mar 19 '18 at 21:18