Linked Questions
63 questions linked to/from Nested cross validation for model selection
1
vote
0
answers
103
views
Why is the outer loop of nested cross validation needed? [duplicate]
I have read several threads and articles about the need to use nested cross validation when, for example, choosing between different machine learning algorithms. Here are two describing the use of ...
- 624
0
votes
1
answer
38
views
Cross validation and hyperparameters tuning [duplicate]
I have few questions concerning the selection of hyperparameters for predictions in cross validation.
If I understand well, during the CV, you just create folds (inner and outer for a nested CV), and ...
- 13
24
votes
5
answers
35k
views
Cross validation and parameter tuning
Can anyone tell me what exactly a cross-validation analysis gives as result?
Is it just the average accuracy or does it give any model with parameters tuned?
Because, I heard somewhere that cross-...
- 241
37
votes
4
answers
17k
views
Internal vs external cross-validation and model selection
My understanding is that with cross validation and model selection we try to address two things:
P1. Estimate the expected loss on the population when training with our sample
P2. Measure and report ...
25
votes
4
answers
8k
views
How bad is hyperparameter tuning outside cross-validation?
I know that performing hyperparameter tuning outside of cross-validation can lead to biased-high estimates of external validity, because the dataset that you use to measure performance is the same one ...
- 5,698
35
votes
3
answers
7k
views
How to build the final model and tune probability threshold after nested cross-validation?
Firstly, apologies for posting a question that has already been discussed at length here, here, here, here, here, and for reheating an old topic. I know @DikranMarsupial has written about this topic ...
30
votes
3
answers
11k
views
How to get hyper parameters in nested cross validation?
I have read the following posts for nested cross validation and still am not 100% sure what I am to do with model selection with nested cross validation:
Nested cross validation for model selection
...
- 541
10
votes
3
answers
11k
views
Implementation of nested cross-validation
I'm trying to figure out if my understanding of nested cross-validation is correct, therefore I wrote this toy example to see if I'm right:
...
- 295
12
votes
2
answers
7k
views
Nested cross-validation - how is it different from model selection via kfold CV on the training set?
I often see people talking about 5x2 cross-validation as a special case of nested cross validation.
I assume the first number (here: 5) refers to the number of folds in the inner loop and the second ...
user39663
25
votes
1
answer
21k
views
How is the confusion matrix reported from K-fold cross-validation?
Suppose I do K-fold cross-validation with K=10 folds. There will be one confusion matrix for each fold. When reporting the results, should I calculate what is the average confusion matrix, or just sum ...
- 251
15
votes
1
answer
10k
views
How to obtain optimal hyperparameters after nested cross validation?
In general, if we have a large dataset, we can split it into (1) training, (2) validation, and (3) test. We use validation to identify the best hyperparameters in cross validation (e.g., C in SVM) and ...
- 12.7k
3
votes
5
answers
4k
views
10-fold cross validation, why having a validation set?
I have my data stratified in 10 folders. So far I was using 9 of them to train the model, and the remaining one for testing it. A sudden thought just crossed my mind saying "you might be cheating". ...
- 987
5
votes
3
answers
647
views
How does cross validation works for feature selection (using stepwise regression)?
I have used the MATLAB regression learner application to do some stepwise regression with a 10-fold cross validation for feature selection. But now I want to code it myself and I'm confused about the ...
- 59
16
votes
1
answer
22k
views
How to do cross-validation with cv.glmnet (LASSO regression in R)?
I'm wondering how to approach properly training and testing a LASSO model using glmnet in R?
Specifically, I'm wondering how to do so if a lack of an external test data set necessitates I use cross-...
- 2,207
9
votes
2
answers
9k
views
How to use cross validation for model comparison
I have been reading the questions related to nested cross validation and model selection and also gone through some tutorials. But I still do not understand how to solve the following problem:
Suppose ...
- 525