# How to identify a case of overfitting using stratified k fold cross validation?

I am working on a problem with a very small amount of data of 211 examples. The problem is a binary classification problem with 2 sets of classes. The data is highly imbalanced with 84% being the majority class and 16% being the minority class.

I am using logistic regression for classification and using f1 score as an evaluation metric. I did stratified k fold cross validation with 5 folds. I divided my data into train and holdout. I am using train in cv and holdout to see how good my model is generalising on unseen data.

f1_score_list = []
f1_score_holdout = []
f1_score_train = []
folds = []
model = LogisticRegression(random_state=RANDOM_STATE)
skf = StratifiedKFold(n_splits=5, random_state=RANDOM_STATE, shuffle=True)
for i, (train_index, test_index) in enumerate(skf.split(X, y)):
X_train, X_valid = X[train_index], X[test_index]
y_train, y_valid = y[train_index], y[test_index]
model.fit(X_train, y_train)
train_pred = model.predict(X_train)
print('Logistic Regression, training set, fold ', i, ': ', f1_score(y_train, train_pred))

pred = model.predict(X_valid)
#Measure of the fit of your model.
print('Logistic Regressaion, validation set, fold ', i, ': ', f1_score(y_valid, pred))

# DATA WHICH MODEL HAS NOT SEEN
pred_holdout = model.predict(X_holdout)
print('Logsitic Regression, holdout set, fold ', i, ': ', f1_score(y_holdout, pred_holdout))
print('Prediction length on validation set, Logistic Regression, fold ', i, ': ', len(pred))

folds.append(i)
f1_score_list.append(f1_score(y_valid, pred))
f1_score_holdout.append(f1_score(y_holdout, pred_holdout))
f1_score_train.append(f1_score(y_train, train_pred))

print ('train f1_score', np.mean(f1_score_train))
print ('cross-val f1_score', np.mean(f1_score_list))
print ('hold out score', np.mean(f1_score_holdout))
plt.plot(folds, f1_score_list, label = 'validation score')
plt.plot(folds, f1_score_holdout, label='holdout score')
plt.plot(folds, f1_score_train, label='training score')
plt.legend()
plt.show()


The output is listed below.

Logistic Regression, training set, fold  0 :  1.0
Logistic Regression, validation set, fold  0 :  0.7692307692307693
Logsitic Regression, holdout set, fold  0 :  0.9166666666666666
Prediction length on validation set, Logistic Regression, fold  0 :  30
Logistic Regression, training set, fold  1 :  1.0
Logistic Regression, validation set, fold  1 :  0.8333333333333333
Logsitic Regression, holdout set, fold  1 :  0.9166666666666666
Prediction length on validation set, Logistic Regression, fold  1 :  30
Logistic Regression, training set, fold  2 :  1.0
Logistic Regression, validation set, fold  2 :  0.9090909090909091
Logsitic Regression, holdout set, fold  2 :  0.9565217391304348
Prediction length on validation set, Logistic Regression, fold  2 :  30
Logistic Regression, training set, fold  3 :  1.0
Logistic Regression, validation set, fold  3 :  1.0
Logsitic Regression, holdout set, fold  3 :  1.0
Prediction length on validation set, Logistic Regression, fold  3 :  29
Logistic Regression, training set, fold  4 :  1.0
Logistic Regression, validation set, fold  4 :  0.888888888888889
Logsitic Regression, holdout set, fold  4 :  0.9166666666666666
Prediction length on validation set, Logistic Regression, fold  4 :  28
train f1_score 1.0
cross-val f1_score 0.8801087801087801
hold out score 0.941304347826087


Based on these results how can I identify whether model is overfitting or not. Any suggestions in this regard would be helpful.

You can check how your model is performing when predicting the training samples on each step of the k-fold cross validation. If the metric is higher on the training samples than on validation or holdout samples, it could be a symptom that the model is overfitting.

• from the training samples, you mean I should make predictions on the same data I am fitting to the model and then calculating training score ?
– Muss
Aug 9, 2019 at 21:30
• yes. measure f1. it should be lower than the metrics for vallidation and holdout
– Victor Ruiz
Aug 9, 2019 at 21:35
– Muss
Aug 9, 2019 at 21:43
• Your graphic shows that the model is slightly overfitting (But f1 score in holdout score is close to 100%, so its not a problem)
– Victor Ruiz
Aug 9, 2019 at 22:13
• yes I tested on new data and most of the predictions are correct, I will try with different parameters to see it generalises well. thanks for the suggestion :)
– Muss
Aug 9, 2019 at 22:25