I keep getting warnings such as

RuntimeWarning: invalid value encountered in greaterreturn (a < x) & (x < b)

and my model summary is full of nans and very large standard errors. The model performance is near identical with what I get when I train with sklearn so it works fine for predictions. But why am I seeing so many weird numbers? I've seen answers about perfect separation causing similar issues - but that is not the case here? I've seen with real data but I get the same issues with generated data as well.

Code to reproduce

import statsmodels.api as sm
import pandas as pd
from sklearn import datasets
from numpy import random

data = datasets.make_classification(n_features = 70, n_informative = 50, n_redundant = 20,n_samples= 10000, random_state = 3)
X = pd.DataFrame(data[0] )
y = data[1]

X['rand_feat1'] = random.randint(100, size=(X.shape[0]))
X['rand_feat2'] = random.randint(100, size=(X.shape[0]))/100

logit_model=sm.Logit(y, X)
sm_result=logit_model.fit_regularized(maxiter = 10000)



enter image description here

  • $\begingroup$ The answer in the link for closing, namely perfect separation, is not an answer to this question. The question is related to multicollinearity problems and penalized estimation. Logit.fit_regularized adds nans in standard errors also intentionally for L1 penalized parameters close to zero because standard inference doesn't apply.. $\endgroup$ – Josef Aug 15 '20 at 17:33

A typical example of (near) singular feature matrix. Some of your features are (near) duplicates of one another and they blow up the $(X'X)^{-1}$ matrix. Fortunately, some implementations of regression have their own way to dealing with it and you can see some result. So the coefficients up there wouldn't have much meaning. e.g. if you have identical variable $X_1$ and $X_2$, different values of $\beta_1$ and $\beta_2$ would result in the same prediction as long as $\beta_1+\beta_2$ is unchanged. That means you can still can enjoy the predictive performance of model, although your coefficients are messed up.

I recommend using LASSO or ridge to regularize to get 'interpretable' coefficients.


Not the answer you're looking for? Browse other questions tagged or ask your own question.