difference between overtraining and overfitting

Question

After reading several tutorials, journal articles, and websites, I am confused about the difference between overtraining and overfitting. Could some gurus enlighten?

-# Others remarks:-#-

In J. Chem. Inf. Comput. Sci., 1995, 35 (5), pp 826–833 (http://pubs.acs.org/doi/abs/10.1021/ci00027a006), the authors said:

• ANN = artificial neural networks with one hidden layer (ANN)

"Since a neural network with a sufficient number of neurons in the hidden layer can exactly implement an arbitrary training set, it can learn both investigated dependencies and a noise that will lower the predictive ability of the network. Two conditions influence the problem:

*size of ANN *time of ANN training

The overfitting problem refers to exceeding some optimal ANN size, while overtraining refers to the time of ANN training that may finally result in worse predictive ability of a network. "

Moreover, this mailing list also has some discussions:

https://groups.google.com/forum/#!topic/comp.ai.neural-nets/V_ryR8fgw2E

-# my understanding:-#-

So my understanding is that over-fitting occurs when a model is too complex, where anything can be correlated with anything if there are enough variables/ factors.

Cross validation is to avoid over-fitting. In a regression model, Using q2 (cross-validated r2) can judge if a model is over-fitting ( J. Am. Chem. Soc. 1988,110, 5959- 5967)?

0.4 < q2 < 1 => good predictive power

0 < q2 < 0.4 => poor model

Also, my understanding is that when a model is over-fitting, it is likely it doesn't have strong predictive power and therefore it is a over-trained model? If yes, good validation results (good q2), but poor prediction results (poor r2 in a regression model) suggests over-training? Could some gurus enlighten? Thanks.

Answer 1

I think the authors are simply using the word "overfitting" to refer to overparametrizing.

In the case of neural networks, overfitting is a consequence of overtraining an overparametrized (i.e. overly complex) model. As far as I can tell, there is no difference between an overtrained and an overfitted model, insofar as the prefix "over" already implies that a line has been crossed. Moreover, "fit" and "training" are basically the same thing. A good indication of this is the fact that Wikipedia redirects you to overfitting when you search for overtraining.

Overfitting can occur when the model is too complex. However, a very complex but well regularized model can still do well, as the function space it searches within is restricted or penalized in certain regions. In fact, some regard deep neural networks as overparametrized but carefully trained and regularized models.

Cross validation can help you get a more or less reliable estimate of how your model is likely to do in the wild. More than avoiding overfitting, it prevents you from relying on the overly optimistic prospect brought by the error obtained by an overfitted model on the training set. It can also help you find a set of hyperparameters that will prevent your model from overfitting, by trying to get good performance on the validation splits.

Answer 2

I think the best thing to do is discard terms like "over-fitting" and "over-training" entirely and just consider bias and variance. Keep in mind that when people say a model is "overfit," they typically mean that the model's variance is higher than the value of variance corresponding to minimum out-of-sample loss. From there, think about how each of the hyperparameters of your model affect both bias and variance. As an example, consider a single-hidden-layer feedforward neural network with MSE loss (and bear in mind that $\hat{MSE}=\hat{\text{var}}+\hat{\text{bias}^2}$). We'll assume that the training dataset is stationary, and that all observations are drawn from the same multivariate distribution.

1) as $N_{iter}$ (number of training iterations) increases, $\hat{\text{var}}$ increases, and $\hat{\text{bias}^2}$ decreases. I believe this is what "over-training" refers to.

2) As $N_{obs}$ (number of observations in the training set) increases, $\hat{\text{var}}$ decreases, and $\hat{\text{bias}^2}$ is unaffected. Clearly, as long as our assumptions hold, larger $N_{obs}$ is always better. In practice, non-stationarity due to temporal changes in the relationships between the predictors and response can create situations where some finite value of $N_{obs}$ is optimal.

3) as $P$ (the number of parameters, or connection weights, in the network) increases , $\hat{\text{var}}$ increases, and $\hat{\text{bias}^2}$ is unaffected.

4) as $\lambda$ (the L2 regularization penalty) increases, $\hat{\text{var}}$ decreases, and $\hat{\text{bias}^2}$ increases.

My preferred approach -- although expensive -- is to determine the topology of the network first (and thus fix $P$, or at least parameterize it in terms of the number of predictors), and then adjust $\lambda$ (or whatever parameter governs your chosen form of regularization) such that the model converges in a way that generalizes well out-of-sample even for arbitrarily large $N_{iter}$.

In the second link I provided above there is a plot of bias, variance, and MSE in the context of ridge regression. It's worth noting that the minimum value of $\hat{MSE}$ does not necessarily coincide with the minimum of $\hat{\text{var}}$. Therefore, targeting the minimum $\hat{\text{var}}$ (and it's $\hat{\text{var}}$ that's generally associated with "overfitting") doesn't necessarily yield you the optimal model in terms of $\hat{MSE}$ when $\hat{\text{bias}^2}>0$. Note also that increasing regularization hyperparameters like $\lambda$, while having the benefit of decreasing $\hat{\text{var}}$, typically have the (generally undesirable) side effect of increasing $\hat{\text{bias}^2}$. Hence the bias-variance tradeoff.

On a somewhat separate note, you can decrease $\hat{\text{var}}$ without increasing $\hat{\text{bias}^2}$ by ensembling (bagging) multiple neural networks trained on the same data with different initializations. But there's no free lunch to be found because the computational cost scales linearly with the number of networks you train.

Answer 3

I'm very much of the opinion that these terms should be separated. Although many people use over-fitting as a catch-all term, it doesn't really capture what is happening in a machine learning setting. That is because many machine learning implementations aren't actually FITTING anything (e.g. as in linear regression or GAM where you're trying to find an equation that best fits the data).

There are also no 'goodness of fit' tests performed in most of the machine learning world (although there are metrics used to measure model accuracy).

An overly complicated model in machine learning is over-LEARNED or over-TRAINED however in that the method may have learned exactly all the relationships in the dataset perfectly, but it isn't generalized enough for inference or prediction.