I am running 10-folds 10 repeats cross validation over my data. I am using XGBoost Classifier with hyper parameter tuning. The learning curve looks as follows:
However, both the training and validation accuracy are increasing, am I overfitting ?
$\begingroup$
$\endgroup$
4
-
$\begingroup$ Not necessarily, though at some point you want to see the plateau. But how exactly are you doing the cross validation? $\endgroup$– NULLCommented Nov 7, 2019 at 14:37
-
$\begingroup$ @NULL Basically my data has the size of 620. I split the data into 80% training and 20% training. I do cross validation with hyper parameter tuning on the 80% training data using stratified kfold cross validation to ensure equal class distribution (Specifically, 10 folds with 10 repeats link). After that, I choose the hyper parameters that gave up the best accuracy in cv. $\endgroup$– Perl Del ReyCommented Nov 7, 2019 at 14:46
-
$\begingroup$ I'm not really following. You're training and testing accuracy is improving as you train the model on more samples, this is what you would expect. $\endgroup$– Matthew DruryCommented Nov 7, 2019 at 15:20
-
$\begingroup$ @MatthewDrury I thought I was overfitting, so this should be the normal behavior of any "good fit" model ? I am neither underfitting nor overfitting, right ? $\endgroup$– Perl Del ReyCommented Nov 7, 2019 at 15:40
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
I'm assuming you are performing nested cross validation often called m-by-n fold validation where m is say 5 and n is 10.
There are two important points here that I want you to note before answering your question:
Your validation set is the one used for parameter tuning (inner loop in nested CV). Your test set (actual unseen data) is the one that is not used/seen in anyway during training/tuning your model (here the outer loop of CV). So you test your best model m times and get a m-component vector of performance (each value is from one round of test) and then say average it to get the final performance.
You need to make sure you have enough samples in your validation and test set. If you don't have enough samples in your validation, you will get optimistic performance results and that would affect your model selection (tuning). If you have small number of test samples, that's bad, because then you are conclusion is not that reliable.
Okay, now that you know these, the figure looks fine because all it is showing is that the more samples the model see, the better its performance is on unseen data. I'm assuming validation here is the actual unseen data (test set).
But there is a catch, on x-axis you are showing the training set size, what is your validation set size? And again is this the test set you call validation or the parameter tuning set (the actual validation set)? That's very important. Is it getting smaller and smaller? I assume so, then yeah, that's not ideal, particularly if you have an unbalanced dataset.
So be very careful with your performance evaluation metric. Read about different methods available and choose one that makes most sense, I'm afraid accuracy is often not the best choice.
