Linked Questions

6
votes
1answer
915 views

Nested cross validation vs repeated k-fold

I know there are many topics(1,2,3), papers(1,2,3) and websites(1) that discuss this topic at length. However for the past two days I am reading all I can find about the subject and seems I hit a ...
1
vote
0answers
71 views

K-Fold Cross Validation, the right way

I searched for this in many forums, but didn't find any answers that would answer my question (or I didn't understand correctly). So, I will post here. I have a dataset A I have a machine learning ...
1
vote
1answer
40 views

Refit a Neural Network after a (nested) cross validation procedure

During the last years I've been running cross validation procedures (and sometimes nested CV) to have an estimate of my model while also doing hyperparameter search. The usual procedure that people ...
1
vote
1answer
40 views

Nested Cross Validation - Which Models Should We Evaluate in the Outer Loop?

Lets assume for example that I am attempting to predict a binary outcome using p predictors in which n>p with methods including a LASSO Regression, a Logistic Regression and SVM with an RBF kernel. ...
3
votes
1answer
470 views

Evaluation of final model in feature selection with nested cross-validation

I am doing feature selection with wrapper method on microarray datasets. I have read several papers and answers here about cross-validation (CV) evaluation on feature selection. Especially the answers ...
1
vote
2answers
107 views

outer folds errors in nested cross-validation

I have a time series data that I wish to be able to obtain the general performance of it. For that, I use nested cross-validation with time series flavor as described in this amazing blog. As you ...
1
vote
2answers
108 views

hyper parameter tuning AFTER Nested cross-validation

I have read very well the awesome answers and suggestions by @cbeleites and @Dikran Marsupial here for nested CV but I am still confused about something: Basically now I understand that nested CV is ...
1
vote
1answer
153 views

Why my validation accuracy and AUC are higher than my training accuracy and AUC?

I have a binary classification problem and I use LightGBM classifier to build my model based on 5 features. I divided my dataset (94 observations) into two parts: Training dataset: 60 observations ...
1
vote
2answers
700 views

K nearest neighbors with nested cross validation

I'm working on a binary classification problem on this dataset, using the k-nn algorithm. For the performance evaluation and the parameter tuning (i.e. the choosing of k) I'm using the nested cross ...
1
vote
1answer
40 views

Testing set accuracy by using cross validation using xgboost with caret

I am working on an xgboost model using caret. I'm using cross validation, but don't know if I'm understanding it correctly. As I understand, it creates multiple training and test sets. Does this mean ...
1
vote
0answers
84 views

Test set vs nested cross-validation

It seems that similar arguments can be made for using nested cross-validation instead of a simple hold-out test set, as the arguments for using cross-validation instead of a single validation set. The ...
1
vote
1answer
753 views

parameter tuning using nested cross validation

Parameter tuning in SVM has been performed using a nested cross-validation(CV) approach with 45 folds(outer loop) and 13 folds(inner loop). In this process, the outer loop will have 45 prediction ...
5
votes
2answers
597 views

Which gamma regression model to use for extrapolation?

I'm looking for a regression model which would satify these requirements: My target variable follows the exponential distribution, so to my understanding I should use gamma loss function. I have ...
1
vote
0answers
139 views

Why not use (nested) cross-validation to update weights when building final model?

I have been trying to find an answer to this question for some time. I understand that cross-validation is primarily used for model selection, i.e. to tune parameters/hyperparameters, but I don’t ...
17
votes
1answer
2k views

How to build the final model and tune probability threshold after nested cross-validation?

Firstly, apologies for posting a question that has already been discussed at length here, here, here, here, here, and for reheating an old topic. I know @DikranMarsupial has written about this topic ...

15 30 50 per page