60k views

Bias and variance in leave-one-out vs K-fold cross validation

How do different cross-validation methods compare in terms of model variance and bias? My question is partly motivated by this thread: Optimal number of folds in $K$-fold cross-validation: is leave-...
142k views

Choice of K in K-fold cross-validation

I've been using the $K$-fold cross-validation a few times now to evaluate performance of some learning algorithms, but I've always been puzzled as to how I should choose the value of $K$. I've often ...
13k views

Optimal number of folds in $K$-fold cross-validation: is leave-one-out CV always the best choice?

Computing power considerations aside, are there any reasons to believe that increasing the number of folds in cross-validation leads to better model selection/validation (i.e. that the higher the ...
51k views

10-fold Cross-validation vs leave-one-out cross-validation

I'm doing nested cross-validation. I have read that leave-one-out cross-validation can be biased (don't remember why). Is it better to use 10-fold cross-validation or leave-one-out cross-validation ...
46k views

Why is leave-one-out cross-validation (LOOCV) variance about the mean estimate for error high? [duplicate]

In leave-one-out cross-validation (LOOCV), each of the training sets looks very similar to the others, differing in only one observation. When you want to estimate the test error, you take the average ...
6k views

What causes lasso to be unstable for feature selection?

In compressed sensing, there is a theorem guarantee that $$\text{argmin} \Vert c \Vert_1\\ \text{subject to } y = Xc$$ has a unique sparse solution $c$ (See appendix for more details). Is there a ...
961 views

Why highly correlated means higher variance?

I am reading the book Introduction to Statistical Learning and on page 183, the book states that Since the mean of many highly correlated quantities has higher variance than does the mean of many ...
3k views

k-fold crossvalidation: how does MSE go with k? [duplicate]

I'm trying to get an intuition about choosing the right "k" for K-Fold validation. Is the following right? Average of the OOS MSEs should generally decrease as k increases. Because, a bigger "k" ...
1k views

Why does k-fold cross validation generate an MSE estimator that has higher bias, but lower variance then leave-one-out cross-validation?

Looks like the rationale behind the accepted answer of this post is incorrect. Under leave one out cross validation(LOOCV), the variance of its MSE estimator is var [\frac{\Sigma_i x_i}{n}] = \...
2k views

Cross Validation and Confidence Interval of the True Error

I'm interested in the relation between Cross Validation and the True Error Estimation of a Classifier (Chapter 5 - Machine Learning - Mitchell). Suppose we have 150 examples, I decide to use a 100 ...
1k views

How will one determine a classifier to be of high bias or high variance?

The bias and variance of a classifier determines the degree to which it can underfit and overfit the data respectively. How could one determine a classifier to be characterized as high bias or high ...
894 views

Leave-one-out cross validation: Relatively unbiased estimate of generalization performance?

I have read that leave-one-out cross-validation provides a relatively “unbiased estimate of the true generalization performance” (e.g. here) and that this is an advantageous property of the leave-one-...
In a linear regression problem, $y = (y_1, \cdots, y_{80})$ is the response, $X = (x_1, \cdots, x_{80})$ is a $4500 \times 80$ matrix of predictors. $k = (k_1, \cdots, k_{4500})$ is the vector of ...