I'm trying to find a full example of how to plot learning curves.

I watched Andrew Ng's ML class on Coursera and he mentions using learning curves to diagnose variance-bias issues.

My notes show the top line as the test set or cv error slopping downward as the number of training examples increases. This makes sense to me.

My problem is on the lower line of the plot which is training error. I'm not sure what this means and why the error would increase with more training examples. Should the training error be multiplied by -1? Or is there something else I need to do?

Any working examples would be appreciated as would an R package that helps with this.

  • $\begingroup$ I think there may be some confusion as to the labels for the lines in question. Could you post an example of the chart(s) you're looking at? $\endgroup$ – babelproofreader Oct 19 '12 at 21:10
  • $\begingroup$ @babelproofreader: cs229.stanford.edu/materials/ML-advice.pdf Here is a paper with 2 examples on pgs. 7-8. $\endgroup$ – screechOwl Oct 19 '12 at 21:53

The training error refers to the error found when testing an algorithm on the data it was trained with. The training error curve slopes up because with very few training samples in relation to the number of features the model can over fit the training data and create a near perfect fit. As the number of training examples increases the model can no longer perfectly fit the data.

Suppose you are classifying email as spam or not spam and you have only 4 features. Lets say the features are if it contains the words buy, deal, offer, or try. There are 2^4 = 16 possible combinations of feature vectors. Now if you have 10 training examples it is feasible they could all have a unique combination of feature values. So when a model is trained on this data it is possible to exactly fit the training examples and the training error will be 0. Now if you use 100 training examples instead this is no longer possible. Some of the training examples will have the same feature vector and if they have different classifications the training error will increase.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.