I have a dataset with 1289 observations and around 2000 features. I split my dataset into a 70/30 training and test set. I use GridSearchCV from scikit-learn to perform 5 fold cross validation on the training data. The plot shows the train and validation scores from the cross validation. Am I overfitting the model? Because in some cases I see that the training accuracy goes up to 1. However, When I test the model on the test set I still get a F1 score of around 0.95.
-
$\begingroup$ How is it possible for your Training Score to Drop while increasing your number of parameters? Shouldn't it increase monotonically? What kind of model are you using there? Edit: i just recognised, that your working in high-dimensional space and your p>>n, so to answer your question you realy have to add further information on your algorithm $\endgroup$– platypusMay 22, 2020 at 8:32
-
$\begingroup$ The model i'm using is SVM. I'm using a param grid of {'C': [1, 10, 100, 1000], 'kernel': ['linear']}, {'C': [1, 10, 100, 1000], 'gamma': [0.001, 0.0001], 'kernel': ['rbf']} and the y axis reflects the number of combinations of the hyperparameters(12 in total) and not the number of parameters. Sorry for the confusion $\endgroup$– Ajay SundaresanMay 22, 2020 at 8:58
-
$\begingroup$ @platypus I converted you answer to comment, since it doesn't seem to answer the question. $\endgroup$– Tim ♦May 22, 2020 at 9:00
-
$\begingroup$ This question seems very similar, but more general: stats.stackexchange.com/questions/294661/… $\endgroup$– rinspyMay 22, 2020 at 9:17
2 Answers
There's nothing "bad" about having 100% accuracy on training sample. In fact, it is common practice in deep learning to start with building a model that is able overfitt a small subset of training set before proceeding further. We are talking about overfitting when there's a discrepancy between training performance of the model, and the performance when the model is applied. This is commonly measured using cross validation, where we validate the model on different data then was used for training. There's no rule how big the difference needs to be to talk about overfitting, since this highly depends on the nature of the problem you are trying to solve. In practice, what you would do, is you'd aim at the test set performance that acceptable for you, while the discrepancy between train and test performance would suggest that there could be some possible areas of improvement.
What is also worth pointing out, as @gunes mentioned in the comment, that accuracy is not the best metric and there are many issues with it.
-
2$\begingroup$ (+1) In addition, this can be a biased set and prediction accuracy can be a wrong metric to plot altogether. $\endgroup$– gunesMay 22, 2020 at 8:09
-
$\begingroup$ Oops. Sorry I specified the wrong metric. I was actually using the F1 scores on the test set and not the accuracy. Edited the question to reflect this $\endgroup$ May 22, 2020 at 8:32
I believe the amount of features is too large. The model which you train might generalize badly on other "test" sets. It could be worth trying to reduce amount of features with PCA and try again.