50 questions linked to/from Nested cross validation for model selection
618 views

### Cross-validation: Which classifier to use in the end? [duplicate]

This might sound like a very simple question, but I haven't been able to find an answer to it, yet: Assuming I am working on a binary classification task and I am using k-fold cross-validation to ...
1k views

### Model prediction: test for difference in MSE

I have made two regression models. They were made on a training set of 80% of the data. And 20% of the data is the validation set. No test set is made. The models tell me how much premium a ...
1k views

### SVM parameter selection and model testing with cross-validation

I've read: Model selection and cross-validation: The right way Crossvalidation and/or testdata. Always use both or can one exclude the other? but I still don't get it. My problem is to construct a ...
1k views

### Feature selection: is nested cross-validation needed?

I have about 150 samples 1000 features (ranked by their importance by Relieff score). My question is, what would be the best approach to: choose the hyper parameters choose the optimal number of ...
985 views

### Comparing classification algorithms using cross validation and caret's train

I am having issues understanding some concepts of algorithm comparison/parameter optimization/cross-validation in R Let's say I want to compare two classification algorithms, such as Random Forests ...
812 views

### Model selection: before or after nested cross-validation?

I want to build a neural network over a data set. My idea is to use cross-validation on a training set to select the "best" neural network (and evaluate it on a separate test set) and to use nested ...
876 views

### Cross-validation when splitting data into train/dev/test sets

Background: Train set: data used to train the chosen model Dev set: data used to tune the model's parameters Test set: data used to evaluate the performance of the final model How cross-validation ...
865 views

### Compare different classification algorithms after hyperparameter tuning

Let's say I have a classification problem with $c$ classes. For this, I have a data set containing $N$ distinct feature vectors with $n$ features. Let's say $N$ is of the order of $10^5$, and both $c$ ...
407 views

### An intuitive understanding of each fold of a nested cross validation for parameter/model tuning

There are several questions on this site essentially asking how nested cross validation for parameter tuning works. A lot of the answers use some jargon that I find difficult to understand, but as far ...
859 views

### Nested cross validation vs repeated k-fold

I know there are many topics(1,2,3), papers(1,2,3) and websites(1) that discuss this topic at length. However for the past two days I am reading all I can find about the subject and seems I hit a ...
489 views

### Which gamma regression model to use for extrapolation?

I'm looking for a regression model which would satify these requirements: My target variable follows the exponential distribution, so to my understanding I should use gamma loss function. I have ...
470 views

### Does changing the parameter search space after nested CV introduce optimistic bias?

Suppose I am fitting a Ridge and I decide to search a parameter space for c:[1,2,3]. I perform nested CV on my whole dataset and find the performance not so great. I therefore expand my search space ...
739 views

### parameter tuning using nested cross validation

Parameter tuning in SVM has been performed using a nested cross-validation(CV) approach with 45 folds(outer loop) and 13 folds(inner loop). In this process, the outer loop will have 45 prediction ...