49 questions linked to/from Nested cross validation for model selection
912 views

### Nested cross validation vs repeated k-fold

I know there are many topics(1,2,3), papers(1,2,3) and websites(1) that discuss this topic at length. However for the past two days I am reading all I can find about the subject and seems I hit a ...
71 views

### K-Fold Cross Validation, the right way

I searched for this in many forums, but didn't find any answers that would answer my question (or I didn't understand correctly). So, I will post here. I have a dataset A I have a machine learning ...
39 views

### Refit a Neural Network after a (nested) cross validation procedure

During the last years I've been running cross validation procedures (and sometimes nested CV) to have an estimate of my model while also doing hyperparameter search. The usual procedure that people ...
39 views

### Nested Cross Validation - Which Models Should We Evaluate in the Outer Loop?

Lets assume for example that I am attempting to predict a binary outcome using p predictors in which n>p with methods including a LASSO Regression, a Logistic Regression and SVM with an RBF kernel. ...
468 views

### Evaluation of final model in feature selection with nested cross-validation

I am doing feature selection with wrapper method on microarray datasets. I have read several papers and answers here about cross-validation (CV) evaluation on feature selection. Especially the answers ...
107 views

### outer folds errors in nested cross-validation

I have a time series data that I wish to be able to obtain the general performance of it. For that, I use nested cross-validation with time series flavor as described in this amazing blog. As you ...
107 views

### hyper parameter tuning AFTER Nested cross-validation

I have read very well the awesome answers and suggestions by @cbeleites and @Dikran Marsupial here for nested CV but I am still confused about something: Basically now I understand that nested CV is ...
150 views

### Why my validation accuracy and AUC are higher than my training accuracy and AUC?

I have a binary classification problem and I use LightGBM classifier to build my model based on 5 features. I divided my dataset (94 observations) into two parts: Training dataset: 60 observations ...
688 views

### K nearest neighbors with nested cross validation

I'm working on a binary classification problem on this dataset, using the k-nn algorithm. For the performance evaluation and the parameter tuning (i.e. the choosing of k) I'm using the nested cross ...
37 views

### Testing set accuracy by using cross validation using xgboost with caret

I am working on an xgboost model using caret. I'm using cross validation, but don't know if I'm understanding it correctly. As I understand, it creates multiple training and test sets. Does this mean ...
84 views

### Test set vs nested cross-validation

It seems that similar arguments can be made for using nested cross-validation instead of a simple hold-out test set, as the arguments for using cross-validation instead of a single validation set. The ...
753 views

### parameter tuning using nested cross validation

Parameter tuning in SVM has been performed using a nested cross-validation(CV) approach with 45 folds(outer loop) and 13 folds(inner loop). In this process, the outer loop will have 45 prediction ...
595 views

### Which gamma regression model to use for extrapolation?

I'm looking for a regression model which would satify these requirements: My target variable follows the exponential distribution, so to my understanding I should use gamma loss function. I have ...
139 views

### Why not use (nested) cross-validation to update weights when building final model?

I have been trying to find an answer to this question for some time. I understand that cross-validation is primarily used for model selection, i.e. to tune parameters/hyperparameters, but I don’t ...
2k views

### How to build the final model and tune probability threshold after nested cross-validation?

Firstly, apologies for posting a question that has already been discussed at length here, here, here, here, here, and for reheating an old topic. I know @DikranMarsupial has written about this topic ...
641 views

### Cross-validation: Which classifier to use in the end? [duplicate]

This might sound like a very simple question, but I haven't been able to find an answer to it, yet: Assuming I am working on a binary classification task and I am using k-fold cross-validation to ...
674 views

### Feature selection & Cross Validation

this is a popular topic here but I have been reading through the different pages and could not find anything related with what I am wondering now. So, I have a data set with X features and I would ...
6k views

### How to get hyper parameters in nested cross validation?

I have read the following posts for nested cross validation and still am not 100% sure what I am to do with model selection with nested cross validation: Nested cross validation for model selection ...
19k views

### Cross validation and parameter tuning

Can anyone tell me what exactly a cross-validation analysis gives as result? Is it just the average accuracy or does it give any model with parameters tuned? Because, I heard somewhere that cross-...
361 views

### Feature selection using LASSO and PCA on training data or whole data?

I am using LASSO and PCA for performing feature selection on a classification problem. The dataset consist of 20 features and around 5.7k observations. One of the reviewer comments for this approach ...
2k views

### How to obtain optimal hyperparameters after nested cross validation?

In general, if we have a large dataset, we can split it into (1) training, (2) validation, and (3) test. We use validation to identify the best hyperparameters in cross validation (e.g., C in SVM) and ...
940 views

### Cross-validation when splitting data into train/dev/test sets

Background: Train set: data used to train the chosen model Dev set: data used to tune the model's parameters Test set: data used to evaluate the performance of the final model How cross-validation ...
452 views

### An intuitive understanding of each fold of a nested cross validation for parameter/model tuning

There are several questions on this site essentially asking how nested cross validation for parameter tuning works. A lot of the answers use some jargon that I find difficult to understand, but as far ...
2k views

### 10-fold cross validation, why having a validation set?

I have my data stratified in 10 folders. So far I was using 9 of them to train the model, and the remaining one for testing it. A sudden thought just crossed my mind saying "you might be cheating". ...
997 views

### Comparing classification algorithms using cross validation and caret's train

I am having issues understanding some concepts of algorithm comparison/parameter optimization/cross-validation in R Let's say I want to compare two classification algorithms, such as Random Forests ...
1k views

### Feature selection: is nested cross-validation needed?

I have about 150 samples 1000 features (ranked by their importance by Relieff score). My question is, what would be the best approach to: choose the hyper parameters choose the optimal number of ...
485 views

### Does changing the parameter search space after nested CV introduce optimistic bias?

Suppose I am fitting a Ridge and I decide to search a parameter space for c:[1,2,3]. I perform nested CV on my whole dataset and find the performance not so great. I therefore expand my search space ...
175 views

### Toy implementation of nested cross-validation: how to determine number of inner and outer folds and how many iterations to run?

I am reading Cawley and Talbot and saw a post on implementation of nested CV (NCV) as well as numerous posts with good answers on the topic (general NCV, training with full dataset after NCV, how to ...
178 views

### How to validate a model when first exploring model hyperparameter space?

For my class project I am comparing various tree-based ensemble methods such as bagging, boosting, random forest, and AdaBoost against my data set and I can't quite determine my methodology. I know ...
8k views

### Implementation of nested cross-validation

I'm trying to figure out if my understanding of nested cross-validation is correct, therefore I wrote this toy example to see if I'm right: ...
891 views

### Compare different classification algorithms after hyperparameter tuning

Let's say I have a classification problem with $c$ classes. For this, I have a data set containing $N$ distinct feature vectors with $n$ features. Let's say $N$ is of the order of $10^5$, and both $c$ ...
89 views

### How can I conclude my model performance?

I use cross validation to find a best set of parameters for random forest on my dataset. Then I use the best model to fit my train set and got an average AUC of 0.6883. But I can see the variability ...
847 views

### Model selection: before or after nested cross-validation?

I want to build a neural network over a data set. My idea is to use cross-validation on a training set to select the "best" neural network (and evaluate it on a separate test set) and to use nested ...
15k views

### How to do cross-validation with cv.glmnet (LASSO regression in R)?

I'm wondering how to approach properly training and testing a LASSO model using glmnet in R? Specifically, I'm wondering how to do so if a lack of an external test data set necessitates I use cross-...
5k views

### How bad is hyperparameter tuning outside cross-validation?

I know that performing hyperparameter tuning outside of cross-validation can lead to biased-high estimates of external validity, because the dataset that you use to measure performance is the same one ...
59 views

### An Information Criterion that considers how many variables we can choose from

I am running a multiple regression model and looking to use AIC and BIC to select models. However I notice that both measures do not consider the number of variables we can choose from but only ...
275 views

### Cross-validation scheme used in the Introduction to Statistical Learning, Chapter 6, Lab 3

I've been really enjoying the Introduction to Statistical Learning textbook so far, and I'm currently working my way through chapter 6. I realize that I am very confused by the process used in lab 3 ...
265 views

### K-fold CV based model selection with a constraint on the number of features?

I am currently working on project where I need to train a logistic regression classifier with a combined $l_1$/$l_2$-penalty that satisfies a hard on the number of features. Specifically, my dataset ...
1k views

### Model prediction: test for difference in MSE

I have made two regression models. They were made on a training set of 80% of the data. And 20% of the data is the validation set. No test set is made. The models tell me how much premium a ...
435 views

### What is the standard procedure for evaluating a user-based CF algorithm with a dataset offline?

I have read some papers and other materials about the evaluation of recommender systems (RS). Most of them discuss the various properties of RS (e.g. accuracy, diversity, etc.), and metrics for ...
55 views

### Performance of a classifier change heavily

I'm using a data set of 32 face persons and a svm-rbf to classify and a random group of 70% for train and 30% for test. The problem is that my results are heavily dependent of the random group used ...
4k views

### Nested cross-validation - how is it different from model selection via kfold CV on the training set?

I often see people talking about 5x2 cross-validation as a special case of nested cross validation. I assume the first number (here: 5) refers to the number of folds in the inner loop and the second ...
3k views

### Is cross validation needed?

Suppose we have training data set and a test data set. The outcome variable is binary. Is it usually necessary to split the training data set so that there is a cross validation data set? Or can you ...
12k views

### How is the confusion matrix reported from K-fold cross-validation?

Suppose I do K-fold cross-validation with K=10 folds. There will be one confusion matrix for each fold. When reporting the results, should I calculate what is the average confusion matrix, or just sum ...
3k views

### CV for model parameter tuning AND then model evaluation

I have a basic question on using cross-validation for model parameter tuning (model training) and model evaluation (testing) similar to this Model Tuning and Model Evaluation in Machine Learning I ...
1k views

### SVM parameter selection and model testing with cross-validation

I've read: Model selection and cross-validation: The right way Crossvalidation and/or testdata. Always use both or can one exclude the other? but I still don't get it. My problem is to construct a ...
3k views

### AIC BIC Mallows Cp Cross Validation Model Selection

If you have several linear models, say model1, model2 and model3, how would you cross-validate it to pick the best model? (In R) I'm wondering this because my AIC and BIC for each model are not ...