1
$\begingroup$

enter image description here

I am working on a problem with a very small amount of data of 211 examples. The problem is a binary classification problem with 2 sets of classes. The data is highly imbalanced with 84% being the majority class and 16% being the minority class.

I am using logistic regression for classification and using f1 score as an evaluation metric. I did stratified k fold cross validation with 5 folds. I divided my data into train and holdout. I am using train in cv and holdout to see how good my model is generalising on unseen data.

f1_score_list = []
f1_score_holdout = []
f1_score_train = []
folds = []
model = LogisticRegression(random_state=RANDOM_STATE)
skf = StratifiedKFold(n_splits=5, random_state=RANDOM_STATE, shuffle=True)
for i, (train_index, test_index) in enumerate(skf.split(X, y)):
    X_train, X_valid = X[train_index], X[test_index]
    y_train, y_valid = y[train_index], y[test_index]
    model.fit(X_train, y_train)
    train_pred = model.predict(X_train)
    print('Logistic Regression, training set, fold ', i, ': ', f1_score(y_train, train_pred))

    pred = model.predict(X_valid) 
    #Measure of the fit of your model.
    print('Logistic Regressaion, validation set, fold ', i, ': ', f1_score(y_valid, pred))

    # DATA WHICH MODEL HAS NOT SEEN
    pred_holdout = model.predict(X_holdout)
    print('Logsitic Regression, holdout set, fold ', i, ': ', f1_score(y_holdout, pred_holdout))
    print('Prediction length on validation set, Logistic Regression, fold ', i, ': ', len(pred))

    folds.append(i)
    f1_score_list.append(f1_score(y_valid, pred))
    f1_score_holdout.append(f1_score(y_holdout, pred_holdout))
    f1_score_train.append(f1_score(y_train, train_pred))

print ('train f1_score', np.mean(f1_score_train))
print ('cross-val f1_score', np.mean(f1_score_list))
print ('hold out score', np.mean(f1_score_holdout))
plt.plot(folds, f1_score_list, label = 'validation score')
plt.plot(folds, f1_score_holdout, label='holdout score')
plt.plot(folds, f1_score_train, label='training score')
plt.legend()
plt.show()

The output is listed below.

Logistic Regression, training set, fold  0 :  1.0
Logistic Regression, validation set, fold  0 :  0.7692307692307693
Logsitic Regression, holdout set, fold  0 :  0.9166666666666666
Prediction length on validation set, Logistic Regression, fold  0 :  30
Logistic Regression, training set, fold  1 :  1.0
Logistic Regression, validation set, fold  1 :  0.8333333333333333
Logsitic Regression, holdout set, fold  1 :  0.9166666666666666
Prediction length on validation set, Logistic Regression, fold  1 :  30
Logistic Regression, training set, fold  2 :  1.0
Logistic Regression, validation set, fold  2 :  0.9090909090909091
Logsitic Regression, holdout set, fold  2 :  0.9565217391304348
Prediction length on validation set, Logistic Regression, fold  2 :  30
Logistic Regression, training set, fold  3 :  1.0
Logistic Regression, validation set, fold  3 :  1.0
Logsitic Regression, holdout set, fold  3 :  1.0
Prediction length on validation set, Logistic Regression, fold  3 :  29
Logistic Regression, training set, fold  4 :  1.0
Logistic Regression, validation set, fold  4 :  0.888888888888889
Logsitic Regression, holdout set, fold  4 :  0.9166666666666666
Prediction length on validation set, Logistic Regression, fold  4 :  28
train f1_score 1.0
cross-val f1_score 0.8801087801087801
hold out score 0.941304347826087

Based on these results how can I identify whether model is overfitting or not. Any suggestions in this regard would be helpful.

$\endgroup$
0

1 Answer 1

1
$\begingroup$

You can check how your model is performing when predicting the training samples on each step of the k-fold cross validation. If the metric is higher on the training samples than on validation or holdout samples, it could be a symptom that the model is overfitting.

$\endgroup$
5
  • $\begingroup$ from the training samples, you mean I should make predictions on the same data I am fitting to the model and then calculating training score ? $\endgroup$
    – Muss
    Aug 9, 2019 at 21:30
  • $\begingroup$ yes. measure f1. it should be lower than the metrics for vallidation and holdout $\endgroup$
    – Victor Ruiz
    Aug 9, 2019 at 21:35
  • $\begingroup$ I have edit my question to add more information of training f1 score. $\endgroup$
    – Muss
    Aug 9, 2019 at 21:43
  • $\begingroup$ Your graphic shows that the model is slightly overfitting (But f1 score in holdout score is close to 100%, so its not a problem) $\endgroup$
    – Victor Ruiz
    Aug 9, 2019 at 22:13
  • $\begingroup$ yes I tested on new data and most of the predictions are correct, I will try with different parameters to see it generalises well. thanks for the suggestion :) $\endgroup$
    – Muss
    Aug 9, 2019 at 22:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.