Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
I have a dataset with 1289 observations and around 2000 features. I split my dataset into a 70/30 training and test set. I use GridSearchCV from scikit-learn to perform 5 fold cross validation on the training data. The plot shows the train and validation scores from the cross validation. Am I overfitting the model? Because in some cases I see that the training accuracy goes up to 1. However, When I test the model on the test set I still get a F1 score of around 0.95.
There's nothing "bad" about having 100% accuracy on training sample. In fact, it is common practice in deep learning to start with building a model that is able overfitt a small subset of training set before proceeding further. We are talking about overfitting when there's a discrepancy between training performance of the model, and the performance when the model is applied. This is commonly measured using cross validation, where we validate the model on different data then was used for training. There's no rule how big the difference needs to be to talk about overfitting, since this highly depends on the nature of the problem you are trying to solve. In practice, what you would do, is you'd aim at the test set performance that acceptable for you, while the discrepancy between train and test performance would suggest that there could be some possible areas of improvement.
What is also worth pointing out, as @gunes mentioned in the comment, that accuracy is not the best metric and there are many issues with it.
I believe the amount of features is too large. The model which you train might generalize badly on other "test" sets. It could be worth trying to reduce amount of features with PCA and try again.
