how to make learning curve plot

Question

I am trying to plot learning curves to determine if my model is undercutting or overfitting, but cannot get learning curves like or Not sure what I did is wrong. I have several stupid simple questions:

(1) Does that mean, for each chosen number of training dataset, we train a model? Suppose m (total training data size) = 100000, n (test data size) = 20000. We build model for training dataset size = 0.1m, 0.2m, 0.3m,.... until m, and then for each model, we evaluate training errors using training dataset with 0.1m, 0.2m,... and test errors (always using n=2000 test dataset)?

(2) What does the error mean here? For regression problems, is it the total errors or mean errors?

(3) For classification model, I saw many people use accuracy as error to make plot. Can we use the cross entropy instead?

Answer 1

1. First, divide your data into training data and test data. Then you construct these kinds of learning curves by training models on data of different sizes, such as 10 samples, 100 samples, 1000 samples. (For most models, you won't use all of your training data.) For each model, estimate the out-of-sample error on your entire test set.

2. Mean error probably makes the most sense, since each experiment has a different size of training data.

3. It's up to you to decide which metric is most interesting for your purposes.