plot learning curves with k fold cross validation

I want to plot a learning curve to see how the error rate of my model varies as the number of training data increases.

To get the training error, it's simple, I just train and evaluate my model on an increasing portion of the dataset.

However, to get the cross validation error, I don't know the correct way of combining the k-fold cross validation technique while gradually incrementing the size of the training dataset.

What is the correct approach to use for plotting cross validation learning curve and using k-fold cross validation?

I know it would be easier if my test set was fixed like when using the holdout method instead of k-fold. In that case I would have 30% of my dataset assigned for testing. I would gradually increment the training set from the remaining 70% and test my model on the 30% holdout set.

But this method is criticised in many textbooks I read. So, I'd rather use the k-fold cross validation method instead.

• You you want to change the size of the partition itself used with cross validation, or do you want to change the $K$ (for number of partitions in cross validation) for this too? – geekoverdose Aug 3 '16 at 8:47
• k fold cross validation would have to be undertaken for an increasing size of the training set (increasing the number of folds seems easier). suppose I I am using 5 folds. In the first run I would designate one fold to be used for cross validation and the other 4 for training. instead of doing the training with all 4 folds I should do it with an increasing number of folds(1,2,3,4). By going through all possible cross validation folds, I would get 4*5 performance measures as opposed to 5. Is this the right way of doing it ? – Imlerith Aug 3 '16 at 18:25
• What criticism of hold-out are exactly are you refering to? – cbeleites unhappy with SX Aug 4 '16 at 14:30
• And what exactly do you want to do with the learning curve once you have it. The best procedure to calculate it does depend on whether you research on "how to measure learning curves" or "how many more samples do we need to add to this data set" or "how many cases does this type of classifier need for that type of data". – cbeleites unhappy with SX Aug 4 '16 at 14:37
• @cbeleites I want to plot learning curves to locate the exact problem responsible for the average classification performance of my model. I don't want to improvise. I want to make an informed decision for the best strategy to take in order to improve the model, hence, the need to plot learning curves – Imlerith Aug 4 '16 at 17:23

First of all, while I'd usually agree that hold-out is not making efficient use of the available samples and the typical set-up is prone to the same mistakes as cross validation, repeated set validation / repeated hold-out is a resampling technique that I think is well suitable for your learning curve calculation. This way, you can reflect what is going on inside the data set you have covering the variation due to different splits (but not fully the variation you'd have to expect with new data set of size $n$). This way you get the fine-grained control over training set size of hold out together with resampling properties close to k-fold.