Linked Questions

117
votes
8answers
60k views

Bias and variance in leave-one-out vs K-fold cross validation

How do different cross-validation methods compare in terms of model variance and bias? My question is partly motivated by this thread: Optimal number of folds in $K$-fold cross-validation: is leave-...
172
votes
4answers
142k views

Choice of K in K-fold cross-validation

I've been using the $K$-fold cross-validation a few times now to evaluate performance of some learning algorithms, but I've always been puzzled as to how I should choose the value of $K$. I've often ...
59
votes
2answers
13k views

Optimal number of folds in $K$-fold cross-validation: is leave-one-out CV always the best choice?

Computing power considerations aside, are there any reasons to believe that increasing the number of folds in cross-validation leads to better model selection/validation (i.e. that the higher the ...
34
votes
2answers
51k views

10-fold Cross-validation vs leave-one-out cross-validation

I'm doing nested cross-validation. I have read that leave-one-out cross-validation can be biased (don't remember why). Is it better to use 10-fold cross-validation or leave-one-out cross-validation ...
16
votes
5answers
46k views

Why is leave-one-out cross-validation (LOOCV) variance about the mean estimate for error high? [duplicate]

In leave-one-out cross-validation (LOOCV), each of the training sets looks very similar to the others, differing in only one observation. When you want to estimate the test error, you take the average ...
14
votes
4answers
6k views

What causes lasso to be unstable for feature selection?

In compressed sensing, there is a theorem guarantee that $$\text{argmin} \Vert c \Vert_1\\ \text{subject to } y = Xc $$ has a unique sparse solution $c$ (See appendix for more details). Is there a ...
7
votes
4answers
961 views

Why highly correlated means higher variance?

I am reading the book Introduction to Statistical Learning and on page 183, the book states that Since the mean of many highly correlated quantities has higher variance than does the mean of many ...
4
votes
1answer
3k views

k-fold crossvalidation: how does MSE go with k? [duplicate]

I'm trying to get an intuition about choosing the right "k" for K-Fold validation. Is the following right? Average of the OOS MSEs should generally decrease as k increases. Because, a bigger "k" ...
7
votes
2answers
1k views

Why does k-fold cross validation generate an MSE estimator that has higher bias, but lower variance then leave-one-out cross-validation?

Looks like the rationale behind the accepted answer of this post is incorrect. Under leave one out cross validation(LOOCV), the variance of its MSE estimator is $$var [\frac{\Sigma_i x_i}{n}] = \...
4
votes
2answers
2k views

Cross Validation and Confidence Interval of the True Error

I'm interested in the relation between Cross Validation and the True Error Estimation of a Classifier (Chapter 5 - Machine Learning - Mitchell). Suppose we have 150 examples, I decide to use a 100 ...
8
votes
1answer
1k views

How will one determine a classifier to be of high bias or high variance?

The bias and variance of a classifier determines the degree to which it can underfit and overfit the data respectively. How could one determine a classifier to be characterized as high bias or high ...
8
votes
1answer
894 views

Leave-one-out cross validation: Relatively unbiased estimate of generalization performance?

I have read that leave-one-out cross-validation provides a relatively “unbiased estimate of the true generalization performance” (e.g. here) and that this is an advantageous property of the leave-one-...
1
vote
1answer
694 views

Unstable Prediction Probabilities

We have a data set of a company that needs to predict their employee resignation status. So We developed four classification models "Bagging","Boosting", "RandomForest" and "Logistic". The task we ...
2
votes
1answer
272 views

Feature filtering with LASSO and cross validation

In a linear regression problem, $y = (y_1, \cdots, y_{80})$ is the response, $X = (x_1, \cdots, x_{80})$ is a $4500 \times 80$ matrix of predictors. $k = (k_1, \cdots, k_{4500})$ is the vector of ...