I want to plot a learning curve to see how the error rate of my model varies as the number of training data increases.
To get the training error, it's simple, I just train and evaluate my model on an increasing portion of the dataset.
However, to get the cross validation error, I don't know the correct way of combining the k-fold cross validation technique while gradually incrementing the size of the training dataset.
What is the correct approach to use for plotting cross validation learning curve and using k-fold cross validation?
I know it would be easier if my test set was fixed like when using the holdout method instead of k-fold. In that case I would have 30% of my dataset assigned for testing. I would gradually increment the training set from the remaining 70% and test my model on the 30% holdout set.
But this method is criticised in many textbooks I read. So, I'd rather use the k-fold cross validation method instead.