Skip to main content
106 votes
Accepted

Bias and variance in leave-one-out vs K-fold cross validation

why would models learned with leave-one-out CV have higher variance? [TL:DR] A summary of recent posts and debates (July 2018) This topic has been widely discussed both on this site, and in the ...
Xavier Bourret Sicotte's user avatar
71 votes
Accepted

When is a biased estimator preferable to unbiased one?

Yes. Often it is the case that we are interested in minimizing the mean squared error, which can be decomposed into variance + bias squared. This is an extremely fundamental idea in machine learning, ...
jld's user avatar
  • 20.4k
67 votes
Accepted

Is overfitting "better" than underfitting?

Overfitting is likely to be worse than underfitting. The reason is that there is no real upper limit to the degradation of generalisation performance that can result from over-fitting, whereas there ...
Dikran Marsupial's user avatar
41 votes

Bias and variance in leave-one-out vs K-fold cross validation

[...] my intuition tells me that in leave-one-out CV one should see relatively lower variance between models than in the $K$-fold CV, since we are only shifting one data point across folds and ...
captain_ahab's user avatar
  • 1,512
41 votes

Is overfitting "better" than underfitting?

Roughly, overfitting is fitting the model to noise, while underfitting is not fitting a model to the signal. In your prediction with overfitting you'll reproduce the noise, the underfitting will show ...
Aksakal's user avatar
  • 61.5k
40 votes

Understanding bias-variance tradeoff derivation

A few more steps of the Bias - Variance decomposition Indeed, the full derivation is rarely given in textbooks as it involves a lot of uninspiring algebra. Here is a more complete derivation using ...
Xavier Bourret Sicotte's user avatar
29 votes
Accepted

Understanding bias-variance tradeoff derivation

You are not wrong, but you made an error in one step since $E[(f(x)-f_k(x))^2] \ne Var(f_k(x))$. Instead, $E[(f(x)-f_k(x))^2]$ is $\text{MSE}(f_k(x)) = Var(f_k(x)) + \text{Bias}^2(f_k(x))$. \begin{...
Greenparker's user avatar
  • 15.6k
25 votes

Intuitive explanation of the bias-variance tradeoff?

First, lets understand the meaning of bias and variance: Imagine the center of the red bulls' eye region is the true mean value of our target random variable which we are trying to predict. Every ...
Thamme Gowda's user avatar
  • 1,211
24 votes

Why is best subset selection not favored in comparison to lasso?

In principle, if the best subset can be found (i.e. for large enough datasets or if the solution is extremely sparse or if there are nonnegativity constraints which can help to identify the best ...
Tom Wenseleers's user avatar
23 votes
Accepted

Why is best subset selection not favored in comparison to lasso?

In subset selection, the nonzero parameters will only be unbiased if you have chosen a superset of the correct model, i.e., if you have removed only predictors whose true coefficient values are zero. ...
Stephan Kolassa's user avatar
21 votes
Accepted

Can I (justifiably) train a second model only on the observations that a previous model predicted poorly?

As noticed in the comments, you’ve re-discovered boosting. Nothing wrong with this approach, but usually it’s easier and safer to use a method already implemented and battle-tested by someone else ...
Tim's user avatar
  • 139k
19 votes

Intuitive explanation of the bias-variance tradeoff?

The basic idea is that too simple a model will underfit (high bias) while too complex a model will overfit (high variance) and that bias and variance trade off as model complexity is varied. (Neal, ...
Tim's user avatar
  • 139k
18 votes

Bias and variance in leave-one-out vs K-fold cross validation

Although this question is rather old, I would like to add an additional answer because I think it is worth clarifying this a bit more. My question is partly motivated by this thread: Optimal ...
D1X's user avatar
  • 763
18 votes

How can we explain the fact that "Bagging reduces the variance while retaining the bias" mathematically?

Quite surprising that the experts couldn't help you out, the chapter on random forests in "The Elements of Statistical Learning" explains it very well. Basically, given n i.d.d. random ...
Scholar's user avatar
  • 1,045
18 votes
Accepted

What is meant by Low Bias and High Variance of the Model?

The key point is that parameter estimates are random variables. If you sample from a population many times and fit a model each time, then you get different parameter estimates. So it makes sense to ...
Stephan Kolassa's user avatar
17 votes

Is overfitting "better" than underfitting?

The question of what is good and what is bad depends on the problem, the question and the circumstances. Some thoughts: The most impressive proof of how underfitting is not generally rejected so much ...
Bernhard's user avatar
  • 8,467
15 votes

Bias-variance decomposition and independence of $X$ and $\epsilon$

Here is a derivation of the bias-variance decomposition, in which I make use of the independence of $X$ and $\epsilon$. True model Suppose that a target variable $Y$ and a feature variable $X$ are ...
Peter McHale's user avatar
14 votes

Bias and variance in leave-one-out vs K-fold cross validation

The issues are indeed subtle. But it is definitely not true that LOOCV has larger variance in general. A recent paper discusses some key aspects and addresses several seemingly widespread ...
Zack's user avatar
  • 141
13 votes
Accepted

Is it possible to decompose fitted residuals into bias and variance, after fitting a linear model?

You generally can't decompose error (residuals) into bias and variance components. The simple reason is that you generally don't know the true function. Recall that $bias(\hat f(x)) = E[\hat f(x) - f(...
einar's user avatar
  • 4,272
13 votes
Accepted

Modern machine learning and the bias-variance trade-off

The main point about Belkin's Double Descent is that, at the interpolation threshold, i.e. the least model capacity where you fit training data exactly, the number of solutions is very constrained. ...
Firebug's user avatar
  • 19.3k
12 votes

Bias / variance tradeoff math

First, nobody says that squared bias and variance behave just like $e^{\pm x}$, in case you are wondering. The point simply is that one increases and the other decreases. It'd similar to supply and ...
Stephan Kolassa's user avatar
12 votes

Different usage of the term "Bias" in stats/machine learning

I will give you a run-down of the terminology used in statistics, which I think is sensible terminology. Cases (1) and (2) do refer to bias in the usual statistical sense, (4) refers to something ...
Ben's user avatar
  • 126k
12 votes

Expected loss function from bias variance trade off (integral help)

This is a functional derivative. Employing the notation in my explanation at https://stats.stackexchange.com/a/236159/919, let $\mathcal L$ be the functional, let $h$ be any (integrable) function, and ...
whuber's user avatar
  • 325k
11 votes
Accepted

Do multiple deep descents exist?

I found two recent works which seem relevant -- Triple descent and the two kinds of overfitting: Where & why do they appear? The claim is that there are two (sample-wise) peaks: one when number of ...
shimao's user avatar
  • 26.2k
10 votes
Accepted

When is it better to use Multiple Linear Regression instead of Polynomial Regression?

First of all note that polynomial regression is a special case of multiple linear regression. Let's consider three models: Model1 $Y = \beta_1 x_1 + \beta_2 x_2 + \beta_3 x_3 + \epsilon$ and ...
Ferdi's user avatar
  • 5,179
10 votes

Statistical Learning. Contradictions?

There’s no contradiction. The fact that something is easy to interpret has nothing to do with how accurate is it. The most interpretable model you could imagine is to predict constant, independently ...
Tim's user avatar
  • 139k
10 votes

Can I (justifiably) train a second model only on the observations that a previous model predicted poorly?

As was mentioned in the comments this idea of iteratively learning from previous model errors is at the core of boosting methodologies like Adaboost or gradient boosting. As you theorize the idea is ...
Tylerr's user avatar
  • 1,552
10 votes

Why do we say that the model has a high variance when variance is actually the measure of spread of the data and not some property of the model?

First off: Bias and variance of a model are measures of how bad your model is, while over- and underfitting are possible reasons for why your model is bad. Definition of bias and variance Bias and ...
frank's user avatar
  • 10.8k
10 votes
Accepted

Understanding importance sampling in Monte Carlo integration

Importance sampling fails to produce reliable estimators when the random ratio \begin{equation} \frac{p(X)}{q(X)}\,f(X)\qquad X\sim q(\cdot) \end{equation} has infinite variance, i.e., when the ...
Xi'an's user avatar
  • 106k
10 votes

Expected loss function from bias variance trade off (integral help)

Starting from the objective $$\min_{\hat f}\mathbb E_D\left[\iint (y-\hat f(x))^2 p(x,y)\,\text dx\text dy \right]$$ the derivation of the optimal function$$\hat f: x\longmapsto \hat f(x)$$follows ...
Xi'an's user avatar
  • 106k

Only top scored, non community-wiki answers of a minimum length are eligible