2
$\begingroup$

I want to plot a learning curve to see how the error rate of my model varies as the number of training data increases.

To get the training error, it's simple, I just train and evaluate my model on an increasing portion of the dataset.

However, to get the cross validation error, I don't know the correct way of combining the k-fold cross validation technique while gradually incrementing the size of the training dataset.

What is the correct approach to use for plotting cross validation learning curve and using k-fold cross validation?

I know it would be easier if my test set was fixed like when using the holdout method instead of k-fold. In that case I would have 30% of my dataset assigned for testing. I would gradually increment the training set from the remaining 70% and test my model on the 30% holdout set.

But this method is criticised in many textbooks I read. So, I'd rather use the k-fold cross validation method instead.

Improve this question
$\endgroup$
6
  • $\begingroup$ You you want to change the size of the partition itself used with cross validation, or do you want to change the $K$ (for number of partitions in cross validation) for this too? $\endgroup$
    – geekoverdose
    Aug 3, 2016 at 8:47
  • $\begingroup$ k fold cross validation would have to be undertaken for an increasing size of the training set (increasing the number of folds seems easier). suppose I I am using 5 folds. In the first run I would designate one fold to be used for cross validation and the other 4 for training. instead of doing the training with all 4 folds I should do it with an increasing number of folds(1,2,3,4). By going through all possible cross validation folds, I would get 4*5 performance measures as opposed to 5. Is this the right way of doing it ? $\endgroup$
    – Imlerith
    Aug 3, 2016 at 18:25
  • $\begingroup$ What criticism of hold-out are exactly are you refering to? $\endgroup$
    – cbeleites unhappy with SX
    Aug 4, 2016 at 14:30
  • $\begingroup$ And what exactly do you want to do with the learning curve once you have it. The best procedure to calculate it does depend on whether you research on "how to measure learning curves" or "how many more samples do we need to add to this data set" or "how many cases does this type of classifier need for that type of data". $\endgroup$
    – cbeleites unhappy with SX
    Aug 4, 2016 at 14:37
  • $\begingroup$ @cbeleites I want to plot learning curves to locate the exact problem responsible for the average classification performance of my model. I don't want to improvise. I want to make an informed decision for the best strategy to take in order to improve the model, hence, the need to plot learning curves $\endgroup$
    – Imlerith
    Aug 4, 2016 at 17:23

1 Answer 1

Reset to default
1
$\begingroup$

First of all, while I'd usually agree that hold-out is not making efficient use of the available samples and the typical set-up is prone to the same mistakes as cross validation, repeated set validation / repeated hold-out is a resampling technique that I think is well suitable for your learning curve calculation. This way, you can reflect what is going on inside the data set you have covering the variation due to different splits (but not fully the variation you'd have to expect with new data set of size $n$). This way you get the fine-grained control over training set size of hold out together with resampling properties close to k-fold.

However, here's a caveat for the informed decision: in case you are talking about small sample size classification, the usual figures of merit (sensitivity, specificity, overall accuracy etc.) are subject to very high testing variance. This testing variance is limited by the number of actual independent cases you have in the denominator of the calculation and can easily be so large that you cannot sensibly use such measured learning curves (keep in mind, "use" typically means extrapolation).

See our paper for details: Beleites, C. and Neugebauer, U. and Bocklitz, T. and Krafft, C. and Popp, J.: Sample size planning for classification models. Anal Chim Acta, 2013, 760, 25-33. DOI: 10.1016/j.aca.2012.11.007
accepted manuscript on arXiv: 1211.1323

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.