I am trying to plot learning curves to determine if my model is undercutting or overfitting, but cannot get learning curves like enter image description here or enter image description here Not sure what I did is wrong. I have several stupid simple questions:

(1) Does that mean, for each chosen number of training dataset, we train a model? Suppose m (total training data size) = 100000, n (test data size) = 20000. We build model for training dataset size = 0.1m, 0.2m, 0.3m,.... until m, and then for each model, we evaluate training errors using training dataset with 0.1m, 0.2m,... and test errors (always using n=2000 test dataset)?

(2) What does the error mean here? For regression problems, is it the total errors or mean errors?

(3) For classification model, I saw many people use accuracy as error to make plot. Can we use the cross entropy instead?

  • $\begingroup$ Can you please show the training curves you get. $\endgroup$ – usεr11852 Dec 8 '18 at 23:33
  1. First, divide your data into training data and test data. Then you construct these kinds of learning curves by training models on data of different sizes, such as 10 samples, 100 samples, 1000 samples. (For most models, you won't use all of your training data.) For each model, estimate the out-of-sample error on your entire test set.

  2. Mean error probably makes the most sense, since each experiment has a different size of training data.

  3. It's up to you to decide which metric is most interesting for your purposes.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.