Learning Curves: Should the training set size be increased incrementally or Random Selection? I am trying to write a bespoke learning curve function. I was wondering how is it usually implemented. When the size of the training set is increased - Is it normally increased by adding new samples to the already existing set incrementally? Or is the training size size is selected randomly?
To give an example: Suppose the train-set size ratios are [0.2, 0.3, 0.4 ...], then when we go from 0.2 to 0.3, do we add an extra 0.1 on top of what existed before(0.2) incrementally? Or do we just get another random sample from the full set?
 A: I think you should get another fully random sample. This can help if, for example, the first iteration of the sample is biased, one way or another.
A: In my experiments, I always create increasing training set sizes by adding samples to the already existing set. So, essentially my training set at each step is $T_1 \subset T_2 \subset \ldots T_n$. However, I ensure I use the same seed throughout and then run my experiments with different seeds. So, my final training sets look like,
$$T_{11} \subset T_{12} \subset \ldots T_{1n}\\
T_{21} \subset T_{22} \subset \ldots T_{2n}\\
\vdots \\
T_{m1} \subset T_{m2} \subset \ldots T_{mn}$$
Then I report the results averaged over these m experiments.
There is a reason why increased training set sizes are created by adding samples. Most algorithms have stochastic behavior when it comes to generalization. If you want to show increasing performance or decreasing loss  curves with respect to training set sizes, it become difficult to portray this behaviour in the presence of variability of performance with random training sets. That is if the curves are not as expected you don't know what went wrong - whether it was the training sample or your model not performing as expected. However, by keeping these increasing superset of training samples, we can say something about what went wrong if the curves still are not as we expected them to be.
