How to identify a case of overfitting using stratified k fold cross validation?

Question

I am working on a problem with a very small amount of data of 211 examples. The problem is a binary classification problem with 2 sets of classes. The data is highly imbalanced with 84% being the majority class and 16% being the minority class.

I am using logistic regression for classification and using f1 score as an evaluation metric. I did stratified k fold cross validation with 5 folds. I divided my data into train and holdout. I am using train in cv and holdout to see how good my model is generalising on unseen data.

f1_score_list = []
f1_score_holdout = []
f1_score_train = []
folds = []
model = LogisticRegression(random_state=RANDOM_STATE)
skf = StratifiedKFold(n_splits=5, random_state=RANDOM_STATE, shuffle=True)
for i, (train_index, test_index) in enumerate(skf.split(X, y)):
    X_train, X_valid = X[train_index], X[test_index]
    y_train, y_valid = y[train_index], y[test_index]
    model.fit(X_train, y_train)
    train_pred = model.predict(X_train)
    print('Logistic Regression, training set, fold ', i, ': ', f1_score(y_train, train_pred))

    pred = model.predict(X_valid) 
    #Measure of the fit of your model.
    print('Logistic Regressaion, validation set, fold ', i, ': ', f1_score(y_valid, pred))

    # DATA WHICH MODEL HAS NOT SEEN
    pred_holdout = model.predict(X_holdout)
    print('Logsitic Regression, holdout set, fold ', i, ': ', f1_score(y_holdout, pred_holdout))
    print('Prediction length on validation set, Logistic Regression, fold ', i, ': ', len(pred))

    folds.append(i)
    f1_score_list.append(f1_score(y_valid, pred))
    f1_score_holdout.append(f1_score(y_holdout, pred_holdout))
    f1_score_train.append(f1_score(y_train, train_pred))

print ('train f1_score', np.mean(f1_score_train))
print ('cross-val f1_score', np.mean(f1_score_list))
print ('hold out score', np.mean(f1_score_holdout))
plt.plot(folds, f1_score_list, label = 'validation score')
plt.plot(folds, f1_score_holdout, label='holdout score')
plt.plot(folds, f1_score_train, label='training score')
plt.legend()
plt.show()

The output is listed below.

Logistic Regression, training set, fold  0 :  1.0
Logistic Regression, validation set, fold  0 :  0.7692307692307693
Logsitic Regression, holdout set, fold  0 :  0.9166666666666666
Prediction length on validation set, Logistic Regression, fold  0 :  30
Logistic Regression, training set, fold  1 :  1.0
Logistic Regression, validation set, fold  1 :  0.8333333333333333
Logsitic Regression, holdout set, fold  1 :  0.9166666666666666
Prediction length on validation set, Logistic Regression, fold  1 :  30
Logistic Regression, training set, fold  2 :  1.0
Logistic Regression, validation set, fold  2 :  0.9090909090909091
Logsitic Regression, holdout set, fold  2 :  0.9565217391304348
Prediction length on validation set, Logistic Regression, fold  2 :  30
Logistic Regression, training set, fold  3 :  1.0
Logistic Regression, validation set, fold  3 :  1.0
Logsitic Regression, holdout set, fold  3 :  1.0
Prediction length on validation set, Logistic Regression, fold  3 :  29
Logistic Regression, training set, fold  4 :  1.0
Logistic Regression, validation set, fold  4 :  0.888888888888889
Logsitic Regression, holdout set, fold  4 :  0.9166666666666666
Prediction length on validation set, Logistic Regression, fold  4 :  28
train f1_score 1.0
cross-val f1_score 0.8801087801087801
hold out score 0.941304347826087

Based on these results how can I identify whether model is overfitting or not. Any suggestions in this regard would be helpful.

Answer 1

You can check how your model is performing when predicting the training samples on each step of the k-fold cross validation. If the metric is higher on the training samples than on validation or holdout samples, it could be a symptom that the model is overfitting.