Linked Questions

1
vote
0answers
42 views

Why is the outer loop of nested cross validation needed? [duplicate]

I have read several threads and articles about the need to use nested cross validation when, for example, choosing between different machine learning algorithms. Here are two describing the use of ...
19
votes
5answers
22k views

Cross validation and parameter tuning

Can anyone tell me what exactly a cross-validation analysis gives as result? Is it just the average accuracy or does it give any model with parameters tuned? Because, I heard somewhere that cross-...
30
votes
4answers
12k views

Internal vs external cross-validation and model selection

My understanding is that with cross validation and model selection we try to address two things: P1. Estimate the expected loss on the population when training with our sample P2. Measure and report ...
21
votes
4answers
6k views

How bad is hyperparameter tuning outside cross-validation?

I know that performing hyperparameter tuning outside of cross-validation can lead to biased-high estimates of external validity, because the dataset that you use to measure performance is the same one ...
25
votes
3answers
3k views

How to build the final model and tune probability threshold after nested cross-validation?

Firstly, apologies for posting a question that has already been discussed at length here, here, here, here, here, and for reheating an old topic. I know @DikranMarsupial has written about this topic ...
10
votes
2answers
10k views

Implementation of nested cross-validation

I'm trying to figure out if my understanding of nested cross-validation is correct, therefore I wrote this toy example to see if I'm right: ...
20
votes
1answer
16k views

How is the confusion matrix reported from K-fold cross-validation?

Suppose I do K-fold cross-validation with K=10 folds. There will be one confusion matrix for each fold. When reporting the results, should I calculate what is the average confusion matrix, or just sum ...
10
votes
2answers
5k views

Nested cross-validation - how is it different from model selection via kfold CV on the training set?

I often see people talking about 5x2 cross-validation as a special case of nested cross validation. I assume the first number (here: 5) refers to the number of folds in the inner loop and the second ...
18
votes
3answers
7k views

How to get hyper parameters in nested cross validation?

I have read the following posts for nested cross validation and still am not 100% sure what I am to do with model selection with nested cross validation: Nested cross validation for model selection ...
2
votes
5answers
2k views

10-fold cross validation, why having a validation set?

I have my data stratified in 10 folders. So far I was using 9 of them to train the model, and the remaining one for testing it. A sudden thought just crossed my mind saying "you might be cheating". ...
10
votes
1answer
17k views

How to do cross-validation with cv.glmnet (LASSO regression in R)?

I'm wondering how to approach properly training and testing a LASSO model using glmnet in R? Specifically, I'm wondering how to do so if a lack of an external test data set necessitates I use cross-...
7
votes
1answer
3k views

CV for model parameter tuning AND then model evaluation

I have a basic question on using cross-validation for model parameter tuning (model training) and model evaluation (testing) similar to this Model Tuning and Model Evaluation in Machine Learning I ...
9
votes
1answer
3k views

How to obtain optimal hyperparameters after nested cross validation?

In general, if we have a large dataset, we can split it into (1) training, (2) validation, and (3) test. We use validation to identify the best hyperparameters in cross validation (e.g., C in SVM) and ...
4
votes
2answers
3k views

Is cross validation needed?

Suppose we have training data set and a test data set. The outcome variable is binary. Is it usually necessary to split the training data set so that there is a cross validation data set? Or can you ...
3
votes
5answers
838 views

Cross-validation: Which classifier to use in the end? [duplicate]

This might sound like a very simple question, but I haven't been able to find an answer to it, yet: Assuming I am working on a binary classification task and I am using k-fold cross-validation to ...

15 30 50 per page