Linked Questions
59 questions linked to/from Nested cross validation for model selection
1
vote
0
answers
95
views
Why is the outer loop of nested cross validation needed? [duplicate]
I have read several threads and articles about the need to use nested cross validation when, for example, choosing between different machine learning algorithms. Here are two describing the use of ...
- 614
22
votes
5
answers
30k
views
Cross validation and parameter tuning
Can anyone tell me what exactly a cross-validation analysis gives as result?
Is it just the average accuracy or does it give any model with parameters tuned?
Because, I heard somewhere that cross-...
- 221
33
votes
4
answers
15k
views
Internal vs external cross-validation and model selection
My understanding is that with cross validation and model selection we try to address two things:
P1. Estimate the expected loss on the population when training with our sample
P2. Measure and report ...
- 17.9k
23
votes
4
answers
8k
views
How bad is hyperparameter tuning outside cross-validation?
I know that performing hyperparameter tuning outside of cross-validation can lead to biased-high estimates of external validity, because the dataset that you use to measure performance is the same one ...
- 5,423
34
votes
3
answers
6k
views
How to build the final model and tune probability threshold after nested cross-validation?
Firstly, apologies for posting a question that has already been discussed at length here, here, here, here, here, and for reheating an old topic. I know @DikranMarsupial has written about this topic ...
- 441
24
votes
3
answers
9k
views
How to get hyper parameters in nested cross validation?
I have read the following posts for nested cross validation and still am not 100% sure what I am to do with model selection with nested cross validation:
Nested cross validation for model selection
...
- 461
10
votes
3
answers
11k
views
Implementation of nested cross-validation
I'm trying to figure out if my understanding of nested cross-validation is correct, therefore I wrote this toy example to see if I'm right:
...
- 285
11
votes
2
answers
6k
views
Nested cross-validation - how is it different from model selection via kfold CV on the training set?
I often see people talking about 5x2 cross-validation as a special case of nested cross validation.
I assume the first number (here: 5) refers to the number of folds in the inner loop and the second ...
user39663
23
votes
1
answer
20k
views
How is the confusion matrix reported from K-fold cross-validation?
Suppose I do K-fold cross-validation with K=10 folds. There will be one confusion matrix for each fold. When reporting the results, should I calculate what is the average confusion matrix, or just sum ...
- 231
3
votes
5
answers
3k
views
10-fold cross validation, why having a validation set?
I have my data stratified in 10 folders. So far I was using 9 of them to train the model, and the remaining one for testing it. A sudden thought just crossed my mind saying "you might be cheating". ...
- 917
14
votes
1
answer
7k
views
How to obtain optimal hyperparameters after nested cross validation?
In general, if we have a large dataset, we can split it into (1) training, (2) validation, and (3) test. We use validation to identify the best hyperparameters in cross validation (e.g., C in SVM) and ...
- 11.6k
14
votes
1
answer
20k
views
How to do cross-validation with cv.glmnet (LASSO regression in R)?
I'm wondering how to approach properly training and testing a LASSO model using glmnet in R?
Specifically, I'm wondering how to do so if a lack of an external test data set necessitates I use cross-...
- 2,007
7
votes
1
answer
4k
views
CV for model parameter tuning AND then model evaluation
I have a basic question on using cross-validation for model parameter tuning (model training) and model evaluation (testing) similar to this Model Tuning and Model Evaluation in Machine Learning
I ...
- 73
5
votes
2
answers
5k
views
Is cross validation needed?
Suppose we have training data set and a test data set. The outcome variable is binary. Is it usually necessary to split the training data set so that there is a cross validation data set? Or can you ...
- 159
7
votes
2
answers
6k
views
How to use cross validation for model comparison
I have been reading the questions related to nested cross validation and model selection and also gone through some tutorials. But I still do not understand how to solve the following problem:
Suppose ...
- 505