4
$\begingroup$

I am a newbie in the Machine Learning world, I completed the course (very good by the way) of Andrew Ng on Coursera. This question is very software-independent. I would like to know, when you draw a learning curve, do you represent the training error and CV error (using the metric that we want like rmse or $R^2$ for linear regression) as a function of the training set size? Or do you represent instead training error and test error as a function of the training set size? I have seen lot of people plotting the learning curve for the test error, whereas in the course of Andrew Ng I have seen the learning curve for the CV error.

I attach as an example some curve that I got few months ago using Python.

Thanks a lot for the clarification, best regards

example of learning curve

$\endgroup$
1
$\begingroup$

It represents training error and testing error. No cross validation involved (usually we have one big fixed testing data set, and changing the size of training samples to produce the curve).

My answers here gives you more details:

How to know if a learning curve from SVM model suffers from bias or variance?

$\endgroup$
  • $\begingroup$ But if we are in the search for the best model can't we plot learning curves comparing training and CV error from many models to check which looks more robust? $\endgroup$ – Felipe Cruz Jul 13 '17 at 14:44
  • 3
    $\begingroup$ Learning curves are not meant to help you choose between competing models, they are meant to tell you if more training data would be useful (I think, I don't find them very useful in practice). $\endgroup$ – Matthew Drury Jul 13 '17 at 15:39
  • $\begingroup$ @FelipeCruz check Matthew's comments. $\endgroup$ – hxd1011 Jul 13 '17 at 16:06
  • 2
    $\begingroup$ @MatthewDrury I feel it is a useful tool in practice. I have seen to many people tuning parameters or collect more data without guidance. $\endgroup$ – hxd1011 Jul 13 '17 at 16:09
  • $\begingroup$ I suppose i've never been in the situation that I could just collect more data, so maybe that is on me. On the other hand, they are not much use in tuning parameters, much more useful is the model complexity vs performance plots. $\endgroup$ – Matthew Drury Jul 13 '17 at 16:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.