0
$\begingroup$

In machine learning, there are two commonly used plots to identify overfitting.

One is the learning curve, which plots the training + test error (y-axis) over the training set size (x-axis).

The other is the training (loss/error) curve, which plots the training + test error (y-axis) over the number of iterations/epochs of one model (x-axis).

Why do we need both curves? Specifically, what does a learning curve tell us over a training curve? (If we just want to detect if a model overfits, the training curve seems much more efficient to plot.)

$\endgroup$
0
$\begingroup$

The learning curve gives you an idea of how the model benefits from being incrementally fed more and more data observations, therefore focusing on inputs external to the model, thereby quantifying the marginal benefit of each new data point.

The training curve gives you an idea of how the model benefits from having its bias-variance trade-off managed while cycling its algorithm back from start to finish repeatedly, therefore, focusing on processes or parameter calibration inputs internal to the model, likely while leaving the number of data observations unchanged.

$\endgroup$
11
  • $\begingroup$ This definitionally makes sense, but I still don't fully understand why we have both: what can we learn from a learning curve that we can't learn from a training curve? (I've only seen a learning curve in the context of overfitting, but a training curve seems much more efficient for detecting that.) $\endgroup$ – kennysong Jul 10 '20 at 3:55
  • $\begingroup$ "I've only seen a learning curve in the context of overfitting." Well, it may not just be useful for that. Imagine that you need to re-train new models with new data, but this future data will costly to collect. The learning curve could help you understand the value of each new datapoint in order to weigh against the cost of collecting. It could also give you an idea how "how accurate" future models of this kind will be as a function of sample size. $\endgroup$ – eithompson Jul 16 '20 at 0:59
  • $\begingroup$ In fact, I don't even see how the learning curve would help with preventing overfitting. Usually, shouldn't the test curve (green in the Wiki article) be monotonic-increasing? Unlike the training (loss/error) curve, I wouldn't expect it to ever reverse direction (random noise aside). It's hard to imagine artificially restricting the amount of data you are feeding a model so as to increase accuracy. It really just seems more like a tool to better understand a model, in terms of how much data it needs in order to function well. $\endgroup$ – eithompson Jul 16 '20 at 1:03
  • $\begingroup$ @eithompson "The learning curve could [also] help you understand the value of each new datapoint". Thanks, that's exactly the answer I was looking for. If you write that as an answer, and link to a reference about extrapolating the learning curve, I'll accept. $\endgroup$ – kennysong Jul 16 '20 at 3:46
  • 1
    $\begingroup$ @develarist This gets into the philosophical territory of test/training set separation. If you where truly "minimizing some ratio of training and test error" then your test set would have effectively become part of your training set which defeats the point of the separation. I suppose you could hold out a validation set to do this with, but I don't think merely minimizing the ratio would be ideal. See the graph in Kenny's link #2: the ratio is pretty tight very early on (when error is high). I guess you could pick the point that minimizes test error with a penalty on [test err]/[train err] $\endgroup$ – eithompson Jul 16 '20 at 15:38
1
+50
$\begingroup$

Short Answer

Overfitting can be estimated from the difference between training- and test- errors. The two curves give you different additional information.

Long Answer

Let's look at the problem more formally. Assume you want to estimate some relationship between $x$ and $y$. The best estimator satisfies:
$$h^* = \underset {h \in H} {\text{argmin }} R(h)= \underset {h \in H} {\text{argmin }} \int l(h(x),y) dp(x,y)$$
where $H$ denotes our hypothesis space, $p$ denote the joint probability distribution, and $l$ denotes a loss function. $R(h)$ here is called the risk functional. In other words, we are looking for a function $h^*$ that minimises the expected error over all possible $x$ values.

However, in practice we don't have access to all values of $x$. Instead we have a finite amount (say $m$) of data pairs $(x,y)$. So, instead of minimizing the true risk, we seek: $$\hat{h} = \underset {h \in H} {\text{argmin }} \hat{R}(h)=\sum_{i=1}^{m} \frac{1}{m} l(h(x_i), y_i)$$ I.e. we seek to minimise the average loss over available data, and "hope" that this will correspond to a small loss over all data as well. This is known as the ERM principle. Here, $\hat{R}(h)$ is called the empirical risk functional.

Here, you can clearly see that problems arise from restricting ourselves to a finite amount of data; often $h^* \neq \hat{h}$. In practice, to estimate the error caused by the finiteness of the amount of data one studies the difference between test error and training error. A big difference between them corresponds to overfitting. That's it! You don't need further analysis to diagnose overfitting.

With this fixed, let's see what info each curve gives us. In general, these two curves give us information on how to solve an overfitting problem.

Learning curve

Notice that $\hat{R}(h) \to R(h)$ as the size of dataset goes to infinity. Hence, getting more data reduces overfitting. So, a learning curve tells us how much overfitting is decreasing as we add more and more data.

Training curve

Some models like neural networks do stochastic optimisation, i.e. they find the best hypothesis function iteratively. So, a training curve tells us how much overfitting is changing as we approach the hypothesis that miminises the empirical loss.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.