0
$\begingroup$

I've run into the following problem which is kinda puzzling me.

I've two GridSearch classes configured, one with the scoring set to roc_auc and the other using the default accuracy. Yet when evaluating the results I find that the model selecting on accuracy peforms better than the one selecting on roc_auc.

pipe_params = {
    'pre_processing': [StandardScaler(), MinMaxScaler(), None],
    'pca': [PCA(n_components=0.95), PCA(n_components=0.85), None],
}

logit_params = {
    'logit__C': [0.01, 0.1, 1, 10, 100],
    'logit__penalty': ['l1', 'l2']
}

merged = {
    **logit_params, **pipe_params
}


pipe = Pipeline([
    ('pre_processing', None),
    ('pca', None),
    ('logit', LogisticRegression())
])


logit = GridSearchCV(
    pipe, param_grid=merged,
    n_jobs=-1, scoring='roc_auc', cv=10
).fit(X_train, y_train)

Which, when evaluated, produces the following results:

ACCURACY: 0.842372573916198
ROC_AUC : 0.842372573916198


F REPORT:
             precision    recall  f1-score   support

          0       0.85      0.90      0.87       149
          1       0.77      0.68      0.72        74

avg / total       0.82      0.83      0.82       223


CONFUSION MATRIX:
[[134  15]
 [ 24  50]]

However, when I set the scoring to the default:

logit = GridSearchCV( pipe, param_grid=merged, n_jobs=-1, cv=10 ).fit(X_train, y_train)

The results show that it actually performs better / gets a higher roc_auc score.

ACCURACY: 0.8295964125560538

ROC_AUC: 0.8451841102847815

F REPORT:
             precision    recall  f1-score   support

          0       0.86      0.89      0.88       149
          1       0.76      0.70      0.73        74

avg / total       0.83      0.83      0.83       223

CONFUSION MATRIXES:

[[133  16]
 [ 22  52]]

What am I missing here, shouldn't the scoring parameter lead to better results for the selected value?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.