Skip to main content
91 votes
Accepted

How is it possible that validation loss is increasing while validation accuracy is increasing as well

Other answers explain well how accuracy and loss are not necessarily exactly (inversely) correlated, as loss measures a difference between raw output (float) and a class (0 or 1 in the case of binary ...
Soltius's user avatar
  • 1,356
86 votes
Accepted

What should I do when my neural network doesn't generalize well?

First of all, let's mention what does "my neural network doesn't generalize well" mean and what's the difference with saying "my neural network doesn't perform well". When training ...
Djib2011's user avatar
  • 6,023
70 votes
Accepted

Why do we worry about overfitting even if "all models are wrong"?

The quote by Box is along the lines of "All models are wrong, but some are useful." If we have bad overfitting, our model will not be useful in making predictions on new data.
Dave's user avatar
  • 64.4k
67 votes
Accepted

Is overfitting "better" than underfitting?

Overfitting is likely to be worse than underfitting. The reason is that there is no real upper limit to the degradation of generalisation performance that can result from over-fitting, whereas there ...
Dikran Marsupial's user avatar
54 votes
Accepted

Dealing with singular fit in mixed models

When you obtain a singular fit, this is often indicating that the model is overfitted – that is, the random effects structure is too complex to be supported by the data, which naturally leads to the ...
Robert Long's user avatar
47 votes

How is it possible that validation loss is increasing while validation accuracy is increasing as well

Accuracy evaluated by comparing the highest softmax output and the correctly labeled class. It is not dependent on how high the softmax output is. To make it clearer, here are some numbers: Suppose ...
ANKUR SATYA's user avatar
42 votes

Why do we worry about overfitting even if "all models are wrong"?

Why do we worry about overfitting even if “all models are wrong”? Your question appears to be a variation of the Nirvana fallacy, implicitly suggesting that if there is no perfect model, then every ...
Ben's user avatar
  • 127k
41 votes

Is overfitting "better" than underfitting?

Roughly, overfitting is fitting the model to noise, while underfitting is not fitting a model to the signal. In your prediction with overfitting you'll reproduce the noise, the underfitting will show ...
Aksakal's user avatar
  • 61.7k
38 votes
Accepted

Is ridge regression useless in high dimensions ($n \ll p$)? How can OLS fail to overfit?

A natural regularization happens because of the presence of many small components in the theoretical PCA of $x$. These small components are implicitly used to fit the noise using small coefficients. ...
Benoit Sanchez's user avatar
37 votes
Accepted

Boosting: why is the learning rate called a regularization parameter?

Suppose you are trying to minimize the objective function via number of iterations. And current value is $100.0$. In given data set, there are no "irreducible errors" and you can minimize the loss to $...
Haitao Du's user avatar
  • 37.1k
36 votes
Accepted

Is an overfitted model necessarily useless?

I think the argument is correct. If 70% is acceptable in the particular application, then the model is useful even though it is overfitted (more generally, regardless of whether it is overfitted or ...
Richard Hardy's user avatar
34 votes

Overfitting, but why is the training deviance dropping?

This is exactly what it means to overfit! In many scenarios, you can make the training performance arbitrarily great, perhaps going as far as playing connect-the-dots. This is analogous to your ...
Dave's user avatar
  • 64.4k
33 votes
Accepted

Is it true that Bayesian methods don't overfit?

No, it is not true. Bayesian methods will certainly overfit the data. There are a couple of things that make Bayesian methods more robust against overfitting and you can make them more fragile as ...
Dave Harris's user avatar
  • 7,760
33 votes

Is an overfitted model necessarily useless?

In my past project with Credit Card Fraud detection, we intentionally want to over fit the data / hard coded to remember fraud cases. (Note, overfitting one class is not exactly the general ...
Haitao Du's user avatar
  • 37.1k
32 votes

Overfitting and Underfitting

I'll try to answer in the simplest way. Each of those problems has its own main origin: Overfitting: Data is noisy, meaning that there are some deviations from reality (because of measurement errors, ...
Luis Da Silva's user avatar
29 votes

Dealing with singular fit in mixed models

This is a very interesting thread, with interesting answers and comments! Since this hasn't been brought up yet, I wanted to point out that we have very little data for each subject (as I understand ...
Isabella Ghement's user avatar
28 votes

how to avoid overfitting in XGBoost model

XGBoost (and other gradient boosting machine routines too) has a number of parameters that can be tuned to avoid over-fitting. I will mention some of the most obvious ones. For example we can change: ...
usεr11852's user avatar
  • 44.8k
26 votes
Accepted

Is using both training and test sets for hyperparameter tuning overfitting?

The idea behind holdout and cross validation is to estimate the generalization performance of a learning algorithm--that is, the expected performance on unknown/unseen data drawn from the same ...
user20160's user avatar
  • 32.8k
26 votes

Can overfitting and underfitting occur simultaneously?

Your reasoning makes sense to me. Here is an extremely simple example. Suppose that $X$ consists of only two columns $x_1$ and $x_2$, and the true DGP is $$ y=\beta_1x_1+\beta_2x_2+\epsilon $$ with ...
Stephan Kolassa's user avatar
26 votes

How can we explain the "bad reputation" of higher-order polynomials?

High degree polynomials do not overfit the data This is a common misconception which is nonetheless found in many textbooks. In general, in order to specify a statistical model, it is necessary to ...
Simon Segert's user avatar
  • 2,054
26 votes

Why is controlling for too many variables considered harmful?

There is no such thing as a "sweet spot" for the number of variables to control for in order to get an unbiased estimate of the causal effect. Since we are talking about confounding, we must ...
ColorStatistics's user avatar
25 votes
Accepted

Out of Bag Error makes CV unnecessary in Random Forests?

training error (as in predict(model, data=train)) is typically useless. Unless you do (non-standard) pruning of the trees, it cannot be much above 0 by design of ...
cbeleites unhappy with SX's user avatar
25 votes

Is ridge regression useless in high dimensions ($n \ll p$)? How can OLS fail to overfit?

Thanks everybody for the great ongoing discussion. The crux of the matter seems to be that minimum-norm OLS is effectively performing shrinkage that is similar to the ridge regression. This seems to ...
amoeba's user avatar
  • 106k
25 votes
Accepted

Why is logistic regression particularly prone to overfitting in high dimensions?

The existing answers aren't wrong, but I think the explanation could be a little more intuitive. There are three key ideas here. 1. Asymptotic Predictions In logistic regression we use a linear model ...
Eoin's user avatar
  • 9,277
24 votes

What should I do when my neural network doesn't generalize well?

There is plenty of empirical evidence that deep enough neural networks can memorize random labels on huge datasets (Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, Oriol Vinyals, "...
DeltaIV's user avatar
  • 18.2k
24 votes
Accepted

What does interpolating the training set actually mean?

Your question already got two nice answers, but I feel that some more context is needed. First, we are talking here about overparametrized models and the double descent phenomenon. By overparametrized ...
Tim's user avatar
  • 140k
23 votes
Accepted

Mathematical/Algorithmic definition for overfitting

Yes there is a (slightly more) rigorous definition: Given a model with a set of parameters, the model can be said to be overfitting the data if after a certain number of training steps, the training ...
Skander H.'s user avatar
23 votes

Why do we worry about overfitting even if "all models are wrong"?

The full quote is "All models are wrong, but some are useful". We care about overfitting, because we still want our models to be useful. If you are familiar with the Bias-variance tradeoff, the "all ...
Cliff AB's user avatar
  • 21.4k
22 votes
Accepted

Can K-fold cross validation cause overfitting?

K-fold cross validation is a standard technique to detect overfitting. It cannot "cause" overfitting in the sense of causality. However, there is no guarantee that k-fold cross-validation removes ...
Has QUIT--Anony-Mousse's user avatar
22 votes

Overfitting, but why is the training deviance dropping?

Instead of looking at the deviance plot for training and test data we could also take a look at some plots of actual fits. Below is an example of fitting with a polynomial. From left to right the ...
Sextus Empiricus's user avatar

Only top scored, non community-wiki answers of a minimum length are eligible