I am working on a problem with a very small amount of data of 211 examples. The problem is a binary classification problem with 2 sets of classes. The data is highly imbalanced with 84% being the majority class and 16% being the minority class.
I am using logistic regression for classification and using f1 score as an evaluation metric. I did stratified k fold cross validation with 5 folds. I divided my data into train and holdout. I am using train in cv and holdout to see how good my model is generalising on unseen data.
f1_score_list = []
f1_score_holdout = []
f1_score_train = []
folds = []
model = LogisticRegression(random_state=RANDOM_STATE)
skf = StratifiedKFold(n_splits=5, random_state=RANDOM_STATE, shuffle=True)
for i, (train_index, test_index) in enumerate(skf.split(X, y)):
X_train, X_valid = X[train_index], X[test_index]
y_train, y_valid = y[train_index], y[test_index]
model.fit(X_train, y_train)
train_pred = model.predict(X_train)
print('Logistic Regression, training set, fold ', i, ': ', f1_score(y_train, train_pred))
pred = model.predict(X_valid)
#Measure of the fit of your model.
print('Logistic Regressaion, validation set, fold ', i, ': ', f1_score(y_valid, pred))
# DATA WHICH MODEL HAS NOT SEEN
pred_holdout = model.predict(X_holdout)
print('Logsitic Regression, holdout set, fold ', i, ': ', f1_score(y_holdout, pred_holdout))
print('Prediction length on validation set, Logistic Regression, fold ', i, ': ', len(pred))
folds.append(i)
f1_score_list.append(f1_score(y_valid, pred))
f1_score_holdout.append(f1_score(y_holdout, pred_holdout))
f1_score_train.append(f1_score(y_train, train_pred))
print ('train f1_score', np.mean(f1_score_train))
print ('cross-val f1_score', np.mean(f1_score_list))
print ('hold out score', np.mean(f1_score_holdout))
plt.plot(folds, f1_score_list, label = 'validation score')
plt.plot(folds, f1_score_holdout, label='holdout score')
plt.plot(folds, f1_score_train, label='training score')
plt.legend()
plt.show()
The output is listed below.
Logistic Regression, training set, fold 0 : 1.0
Logistic Regression, validation set, fold 0 : 0.7692307692307693
Logsitic Regression, holdout set, fold 0 : 0.9166666666666666
Prediction length on validation set, Logistic Regression, fold 0 : 30
Logistic Regression, training set, fold 1 : 1.0
Logistic Regression, validation set, fold 1 : 0.8333333333333333
Logsitic Regression, holdout set, fold 1 : 0.9166666666666666
Prediction length on validation set, Logistic Regression, fold 1 : 30
Logistic Regression, training set, fold 2 : 1.0
Logistic Regression, validation set, fold 2 : 0.9090909090909091
Logsitic Regression, holdout set, fold 2 : 0.9565217391304348
Prediction length on validation set, Logistic Regression, fold 2 : 30
Logistic Regression, training set, fold 3 : 1.0
Logistic Regression, validation set, fold 3 : 1.0
Logsitic Regression, holdout set, fold 3 : 1.0
Prediction length on validation set, Logistic Regression, fold 3 : 29
Logistic Regression, training set, fold 4 : 1.0
Logistic Regression, validation set, fold 4 : 0.888888888888889
Logsitic Regression, holdout set, fold 4 : 0.9166666666666666
Prediction length on validation set, Logistic Regression, fold 4 : 28
train f1_score 1.0
cross-val f1_score 0.8801087801087801
hold out score 0.941304347826087
Based on these results how can I identify whether model is overfitting or not. Any suggestions in this regard would be helpful.