31
$\begingroup$

I am trying to understand why the output from logistic regression of these two libraries gives different results.

I am using the dataset from UCLA idre tutorial, predicting admit based on gre, gpa and rank. rank is treated as categorical variable, so it is first converted to dummy variable with rank_1 dropped. An intercept column is also added.

df = pd.read_csv("https://stats.idre.ucla.edu/stat/data/binary.csv")
y, X = dmatrices('admit ~ gre + gpa + C(rank)', df, return_type = 'dataframe')
X.head()
>  Intercept  C(rank)[T.2]  C(rank)[T.3]  C(rank)[T.4]  gre   gpa
0          1             0             1             0  380  3.61
1          1             0             1             0  660  3.67
2          1             0             0             0  800  4.00
3          1             0             0             1  640  3.19
4          1             0             0             1  520  2.93

# Output from scikit-learn
model = LogisticRegression(fit_intercept = False)
mdl = model.fit(X, y)
model.coef_
> array([[-1.35417783, -0.71628751, -1.26038726, -1.49762706,  0.00169198,
     0.13992661]]) 
# corresponding to predictors [Intercept, rank_2, rank_3, rank_4, gre, gpa]

# Output from statsmodels
logit = sm.Logit(y, X)
logit.fit().params
> Optimization terminated successfully.
     Current function value: 0.573147
     Iterations 6
Intercept      -3.989979
C(rank)[T.2]   -0.675443
C(rank)[T.3]   -1.340204
C(rank)[T.4]   -1.551464
gre             0.002264
gpa             0.804038
dtype: float64

The output from statsmodels is the same as shown on the idre website, but I am not sure why scikit-learn produces a different set of coefficients. Does it minimize some different loss function? Is there any documentation that states the implementation?

$\endgroup$
28
$\begingroup$

Your clue to figuring this out should be that the parameter estimates from the scikit-learn estimation are uniformly smaller in magnitude than the statsmodels counterpart. This might lead you to believe that scikit-learn applies some kind of parameter regularization. You can confirm this by reading the scikit-learn documentation.

There is no way to switch off regularization in scikit-learn, but you can make it ineffective by setting the tuning parameter C to a large number. Here is how that works in your case:

# module imports
from patsy import dmatrices
import pandas as pd
from sklearn.linear_model import LogisticRegression
import statsmodels.discrete.discrete_model as sm

# read in the data & create matrices
df = pd.read_csv("http://www.ats.ucla.edu/stat/data/binary.csv")
y, X = dmatrices('admit ~ gre + gpa + C(rank)', df, return_type = 'dataframe')

# sklearn output
model = LogisticRegression(fit_intercept = False, C = 1e9)
mdl = model.fit(X, y)
model.coef_

# sm
logit = sm.Logit(y, X)
logit.fit().params
$\endgroup$
  • $\begingroup$ Thank you very much for the explanation! With this regularized result, I was trying to duplicate the result using the glmnet package in R, but could not get the same coefficient. glmnet has a slightly different cost function comparing to sklearn, but even if I set alpha=0 in glmnet (meaning only use l2-penalty) and set 1/(N*lambda)=C, I still do not get the same result? $\endgroup$ – hurrikale Mar 26 '16 at 13:04
  • $\begingroup$ My intuition is that if I divide both terms of the cost function in glmnet by lambda and set the new constant in font of the log-likelihood, which is 1/(N*lambda) equal to that in sklearn, the two cost functions become identical, or am I missing something? $\endgroup$ – hurrikale Mar 26 '16 at 13:06
  • $\begingroup$ @hurrikale Ask a new question and link it here, and I will take a look. $\endgroup$ – tchakravarty Mar 26 '16 at 13:11
  • $\begingroup$ Thanks! I posted the question here. $\endgroup$ – hurrikale Mar 26 '16 at 13:55
  • $\begingroup$ I think the best way to switch off the regularization in scikit-learn is by setting penalty='none'. $\endgroup$ – Nzbuu Jul 17 at 15:30
3
$\begingroup$

Another difference is that you've set fit_intercept=False, which effectively is a different model. You can see that Statsmodel includes the intercept. Not having an intercept surely changes the expected weights on the features. Try the following and see how it compares:

model = LogisticRegression(C=1e9)

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.