0
$\begingroup$

I'm running a logistic regression on a balanced data set and wanted to validate my model using the ROC-AUC metric.

features_train = df_train.loc[:, ["num_following", "ratio_numlen_username", "num_posts"]]
target_train = df_train.iloc[:, 0]
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
features_standardized = scaler.fit_transform(features_train)
model = LogisticRegression()
model.fit(features_standardized, target_train)

But for some reason, my CV scores are way better than my training score:

roc_auc = roc_auc_score(target_train, model.predict(features_standardized))
print('AUC: %.3f' % roc_auc)

OUT: AUC: 0.823

cv_results = cross_val_score(estimator=model,
                            X=features_standardized,
                            y=target_train,
                            cv=2,
                            scoring="roc_auc")
print(cv_results)
print(round(cv_results.mean(), 2))

OUT: [0.92332176 0.93171296]
0.93

The CV and training scores are the same though, when using other metrics like accuracy, F1, precision and recall

f1_score(target_train, model.predict(features_standardized))
OUT: 0.8118081180811808

cv_results = cross_val_score(estimator=model,
                            X=features_standardized,
                            y=target_train,
                            cv=2,
                            scoring="f1")
print(cv_results)
print(round(cv_results.mean(), 2))

OUT: [0.76377953 0.83392226]
0.8

Any idea why?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.