1
$\begingroup$

I'm creating a classifier using linear regression to classify images of hand-drawn digits from the MNIST dataset. I realize that linear regression is not the appropriate approach, but this is for a school assignment meant to illustrate why logistic regression exists.

Anyway, my issue is that when I fit the model using Python's OLS Statsmodels, the model summary returns coefficients with value and standard error = 0, and thus null p-values and test statistics, etc. associated with the coefficients.

My assumption is that this is incorrect, and I'm wondering if statsmodels is trying to 'drop' the coefficients in this model in the presence of multicollinearity, similar to how R returns null values for coefficients with this issue. If so, I was going to drop these coefficients with value = 0 before using the model to predict digits.

Basically I'm wondering if anyone has seen this behavior before and what it might suggest about my data.

Improve this question
$\endgroup$
3
  • $\begingroup$ statsmodels is using pinv which can in some cases result in this. see for example stackoverflow.com/questions/40935624/… stats.stackexchange.com/questions/116825/… $\endgroup$
    – Josef
    Commented Nov 22, 2022 at 21:07
  • $\begingroup$ @Josef thank you!! this was incredibly helpful. my next question -- do you know if this same behavior exists for sklearn? I do not get the same error, and the regression runs in what appears to be successful, but i'm wondering if there's some underlying issue still present that sklearn just doesn't notify us of? or is it because sklearn logistic regression includes a default regularization penalty? $\endgroup$
    – jmoore00
    Commented Nov 23, 2022 at 0:08
  • 1
    $\begingroup$ logistic regression, Logit is different from OLS, the former uses a scipy optimizer, while the latter just uses linear algebra, AFAIK, linear regression in sklearn also uses SVD based estimation. Logit in statsmodels usually fails with singular hessian if x is not of full rank, sklearn still works because of the default penalization $\endgroup$
    – Josef
    Commented Nov 23, 2022 at 7:09

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.