Linked Questions

1 vote
0 answers

Why is the outer loop of nested cross validation needed? [duplicate]

I have read several threads and articles about the need to use nested cross validation when, for example, choosing between different machine learning algorithms. Here are two describing the use of ...
Sean's user avatar
  • 634
0 votes
1 answer

Cross validation and hyperparameters tuning [duplicate]

I have few questions concerning the selection of hyperparameters for predictions in cross validation. If I understand well, during the CV, you just create folds (inner and outer for a nested CV), and ...
Nicolas's user avatar
  • 13
25 votes
5 answers

Cross validation and parameter tuning

Can anyone tell me what exactly a cross-validation analysis gives as result? Is it just the average accuracy or does it give any model with parameters tuned? Because, I heard somewhere that cross-...
Sana Sudheer's user avatar
37 votes
4 answers

Internal vs external cross-validation and model selection

My understanding is that with cross validation and model selection we try to address two things: P1. Estimate the expected loss on the population when training with our sample P2. Measure and report ...
Amelio Vazquez-Reina's user avatar
25 votes
4 answers

How bad is hyperparameter tuning outside cross-validation?

I know that performing hyperparameter tuning outside of cross-validation can lead to biased-high estimates of external validity, because the dataset that you use to measure performance is the same one ...
Ben Kuhn's user avatar
  • 5,748
35 votes
3 answers

How to build the final model and tune probability threshold after nested cross-validation?

Firstly, apologies for posting a question that has already been discussed at length here, here, here, here, here, and for reheating an old topic. I know @DikranMarsupial has written about this topic ...
Dr. Andi Lowe's user avatar
30 votes
3 answers

How to get hyper parameters in nested cross validation?

I have read the following posts for nested cross validation and still am not 100% sure what I am to do with model selection with nested cross validation: Nested cross validation for model selection ...
Heavy Breathing's user avatar
12 votes
2 answers

Nested cross-validation - how is it different from model selection via kfold CV on the training set?

I often see people talking about 5x2 cross-validation as a special case of nested cross validation. I assume the first number (here: 5) refers to the number of folds in the inner loop and the second ...
user avatar
10 votes
3 answers

Implementation of nested cross-validation

I'm trying to figure out if my understanding of nested cross-validation is correct, therefore I wrote this toy example to see if I'm right: ...
abudis's user avatar
  • 295
25 votes
1 answer

How is the confusion matrix reported from K-fold cross-validation?

Suppose I do K-fold cross-validation with K=10 folds. There will be one confusion matrix for each fold. When reporting the results, should I calculate what is the average confusion matrix, or just sum ...
der's user avatar
  • 251
17 votes
1 answer

How to obtain optimal hyperparameters after nested cross validation?

In general, if we have a large dataset, we can split it into (1) training, (2) validation, and (3) test. We use validation to identify the best hyperparameters in cross validation (e.g., C in SVM) and ...
RockTheStar's user avatar
  • 13.1k
5 votes
3 answers

How does cross validation works for feature selection (using stepwise regression)?

I have used the MATLAB regression learner application to do some stepwise regression with a 10-fold cross validation for feature selection. But now I want to code it myself and I'm confused about the ...
Azarang's user avatar
  • 59
3 votes
5 answers

10-fold cross validation, why having a validation set?

I have my data stratified in 10 folders. So far I was using 9 of them to train the model, and the remaining one for testing it. A sudden thought just crossed my mind saying "you might be cheating". ...
sdiabr's user avatar
  • 987
16 votes
1 answer

How to do cross-validation with cv.glmnet (LASSO regression in R)?

I'm wondering how to approach properly training and testing a LASSO model using glmnet in R? Specifically, I'm wondering how to do so if a lack of an external test data set necessitates I use cross-...
theforestecologist's user avatar
15 votes
2 answers

Nested cross validation vs repeated k-fold

I know there are many topics(1,2,3), papers(1,2,3) and websites(1) that discuss this topic at length. However for the past two days I am reading all I can find about the subject and seems I hit a ...
Pedro Dreyer's user avatar

15 30 50 per page
2 3 4 5