Learning curve vs training (loss) curve?

Question

In machine learning, there are two commonly used plots to identify overfitting.

One is the learning curve, which plots the training + test error (y-axis) over the training set size (x-axis).

The other is the training (loss/error) curve, which plots the training + test error (y-axis) over the number of iterations/epochs of one model (x-axis).

Why do we need both curves? Specifically, what does a learning curve tell us over a training curve? (If we just want to detect if a model overfits, the training curve seems much more efficient to plot.)

Answer 1

The learning curve gives you an idea of how the model benefits from being incrementally fed more and more data observations, therefore focusing on inputs external to the model, thereby quantifying the marginal benefit of each new data point.

The training curve gives you an idea of how the model benefits from having its bias-variance trade-off managed while cycling its algorithm back from start to finish repeatedly, therefore, focusing on processes or parameter calibration inputs internal to the model, likely while leaving the number of data observations unchanged.

Answer 2

Short Answer

Overfitting can be estimated from the difference between training- and test- errors. The two curves give you different additional information.

Long Answer

Let's look at the problem more formally. Assume you want to estimate some relationship between $x$ and $y$. The best estimator satisfies:
$$h^* = \underset {h \in H} {\text{argmin }} R(h)= \underset {h \in H} {\text{argmin }} \int l(h(x),y) dp(x,y)$$
where $H$ denotes our hypothesis space, $p$ denote the joint probability distribution, and $l$ denotes a loss function. $R(h)$ here is called the risk functional. In other words, we are looking for a function $h^*$ that minimises the expected error over all possible $x$ values.

However, in practice we don't have access to all values of $x$. Instead we have a finite amount (say $m$) of data pairs $(x,y)$. So, instead of minimizing the true risk, we seek: $$\hat{h} = \underset {h \in H} {\text{argmin }} \hat{R}(h)=\sum_{i=1}^{m} \frac{1}{m} l(h(x_i), y_i)$$ I.e. we seek to minimise the average loss over available data, and "hope" that this will correspond to a small loss over all data as well. This is known as the ERM principle. Here, $\hat{R}(h)$ is called the empirical risk functional.

Here, you can clearly see that problems arise from restricting ourselves to a finite amount of data; often $h^* \neq \hat{h}$. In practice, to estimate the error caused by the finiteness of the amount of data one studies the difference between test error and training error. A big difference between them corresponds to overfitting. That's it! You don't need further analysis to diagnose overfitting.

With this fixed, let's see what info each curve gives us. In general, these two curves give us information on how to solve an overfitting problem.

Learning curve

Notice that $\hat{R}(h) \to R(h)$ as the size of dataset goes to infinity. Hence, getting more data reduces overfitting. So, a learning curve tells us how much overfitting is decreasing as we add more and more data.

Training curve

Some models like neural networks do stochastic optimisation, i.e. they find the best hypothesis function iteratively. So, a training curve tells us how much overfitting is changing as we approach the hypothesis that miminises the empirical loss.