LogisticRegression with GridSearchCV not converging

Question

I'm trying to find the best parameters for a logistoic regression but I find that the "best estimator" doesn't converge.

Is there a way to specify that the estimator needs to converge to take it into account?

Here is my code.

# NO PCA
cv = GroupKFold(n_splits=10)
pipe = Pipeline([('scale', StandardScaler()),
    ('mnl', LogisticRegression(fit_intercept=True, multi_class="multinomial"))])

param_grid = [{'mnl__solver': ['newton-cg', 'lbfgs','sag', 'saga'],
               'mnl__C':[0.5,1,1.5,2,2.5],
               'mnl__class_weight':[None,'balanced'],
              'mnl__max_iter':[1000,2000,3000],
              'mnl__penalty':['l1','l2']}]

grid = GridSearchCV(estimator = pipe, param_grid=param_grid, scoring=scoring, n_jobs=-1, refit='neg_log_loss', cv=cv, verbose=2, return_train_score=True)

grid.fit(X, y, groups=data.groups)


# WITH PCA
pipe = Pipeline([(
    ('scale', StandardScaler()),
    ('pca', PCA())
    ('mnl', mnl)])

param_grid = [{'pca__n_components':[None,15,30,45,65]
            'mnl__solver': ['newton-cg', 'lbfgs','sag', 'saga'],
              'mnl__max_iter':[1000,2000,3000],
             'mnl__C':[0.5,1,1.5,2,2.5],
              'mnl__class_weight':[None,'balanced'],
              'mnl__penalty':['l1','l2']}]

grid = GridSearchCV(estimator = pipe, param_grid=param_grid, scoring='neg_log_loss', n_jobs=-1, refit=True, cv=cv, verbose=2)

grid.fit(X, y, groups=data.groups)

On the first case, the best estimator found is with an l2-lbfgs solver, with 1000 iterations, and it converges. The second one, the best estimator found is with saga solver and l1 penalty, 3000 iterations. I feel it has to do with the solver... but anyways, is there a straightforward way to state that it has to converge to accept it as best?

Answer 1

I wouldn't advise making that constraint. Failure of the solver to converge just means it hasn't reached the global optimum* to within the specified tolerance. If you're getting better cross-validation scores, then you should probably be "close enough" to that optimum to not worry about it. Increase max_iter for the refit, if you'd like.
*(In logistic regression the loss is convex, so there's just one global optimum, barring collinear features or perfect separation.)

In a similar spirit, I wouldn't search over solvers (except maybe as a convenient way to deal with different solvers being capable of using different regularization penalties), or maximum number of iterations. After fixing the regularization type and strength, there's unique optimal coefficients (again, barring degenerate cases), and running different solvers should produce the same results unless (1) the solver goes off the rails somehow, or (2) difference in precision causes some difference. And the number of iterations should just be set high enough to reach convergence (note that it's "maximum" number of iterations; if a solver gets within its tolerance before that, it won't keep chugging).

Answer 2

This is more than likely a problem with your data not being appropriate for this approach, rather than your code being incorrect. Especially since you arent constructing your own silver or anything I would really check out your data more thoroughly.