Linked Questions

17
votes
4answers
18k views

Cross validation and parameter tuning

Can anyone tell me what exactly a cross-validation analysis gives as result? Is it just the average accuracy or does it give any model with parameters tuned? Because, I heard somewhere that cross-...
26
votes
4answers
10k views

Internal vs external cross-validation and model selection

My understanding is that with cross validation and model selection we try to address two things: P1. Estimate the expected loss on the population when training with our sample P2. Measure and report ...
20
votes
4answers
5k views

How bad is hyperparameter tuning outside cross-validation?

I know that performing hyperparameter tuning outside of cross-validation can lead to biased-high estimates of external validity, because the dataset that you use to measure performance is the same one ...
9
votes
2answers
8k views

Implementation of nested cross-validation

I'm trying to figure out if my understanding of nested cross-validation is correct, therefore I wrote this toy example to see if I'm right: ...
18
votes
1answer
11k views

How is the confusion matrix reported from K-fold cross-validation?

Suppose I do K-fold cross-validation with K=10 folds. There will be one confusion matrix for each fold. When reporting the results, should I calculate what is the average confusion matrix, or just sum ...
16
votes
3answers
5k views

How to get hyper parameters in nested cross validation?

I have read the following posts for nested cross validation and still am not 100% sure what I am to do with model selection with nested cross validation: Nested cross validation for model selection ...
9
votes
2answers
4k views

Nested cross-validation - how is it different from model selection via kfold CV on the training set?

I often see people talking about 5x2 cross-validation as a special case of nested cross validation. I assume the first number (here: 5) refers to the number of folds in the inner loop and the second ...
2
votes
5answers
1k views

10-fold cross validation, why having a validation set?

I have my data stratified in 10 folders. So far I was using 9 of them to train the model, and the remaining one for testing it. A sudden thought just crossed my mind saying "you might be cheating". ...
10
votes
1answer
14k views

How to do cross-validation with cv.glmnet (LASSO regression in R)?

I'm wondering how to approach properly training and testing a LASSO model using glmnet in R? Specifically, I'm wondering how to do so if a lack of an external test data set necessitates I use cross-...
17
votes
1answer
2k views

How to build the final model and tune probability threshold after nested cross-validation?

Firstly, apologies for posting a question that has already been discussed at length here, here, here, here, here, and for reheating an old topic. I know @DikranMarsupial has written about this topic ...
4
votes
2answers
3k views

Is cross validation needed?

Suppose we have training data set and a test data set. The outcome variable is binary. Is it usually necessary to split the training data set so that there is a cross validation data set? Or can you ...
7
votes
1answer
2k views

CV for model parameter tuning AND then model evaluation

I have a basic question on using cross-validation for model parameter tuning (model training) and model evaluation (testing) similar to this Model Tuning and Model Evaluation in Machine Learning I ...
8
votes
1answer
2k views

How to obtain optimal hyperparameters after nested cross validation?

In general, if we have a large dataset, we can split it into (1) training, (2) validation, and (3) test. We use validation to identify the best hyperparameters in cross validation (e.g., C in SVM) and ...
2
votes
1answer
3k views

AIC BIC Mallows Cp Cross Validation Model Selection

If you have several linear models, say model1, model2 and model3, how would you cross-validate it to pick the best model? (In R) I'm wondering this because my AIC and BIC for each model are not ...
4
votes
0answers
4k views

Backward stepwise regression with cross validation in R

I would like to do model selection using backward stepwise procedure and cross validation. https://www.otexts.org/fpp/5/3 I have used stepAIC in ...
3
votes
5answers
620 views

Cross-validation: Which classifier to use in the end? [duplicate]

This might sound like a very simple question, but I haven't been able to find an answer to it, yet: Assuming I am working on a binary classification task and I am using k-fold cross-validation to ...
1
vote
2answers
1k views

Model prediction: test for difference in MSE

I have made two regression models. They were made on a training set of 80% of the data. And 20% of the data is the validation set. No test set is made. The models tell me how much premium a ...
3
votes
1answer
1k views

SVM parameter selection and model testing with cross-validation

I've read: Model selection and cross-validation: The right way Crossvalidation and/or testdata. Always use both or can one exclude the other? but I still don't get it. My problem is to construct a ...
1
vote
1answer
1k views

Feature selection: is nested cross-validation needed?

I have about 150 samples 1000 features (ranked by their importance by Relieff score). My question is, what would be the best approach to: choose the hyper parameters choose the optimal number of ...
1
vote
1answer
987 views

Comparing classification algorithms using cross validation and caret's train

I am having issues understanding some concepts of algorithm comparison/parameter optimization/cross-validation in R Let's say I want to compare two classification algorithms, such as Random Forests ...
0
votes
1answer
815 views

Model selection: before or after nested cross-validation?

I want to build a neural network over a data set. My idea is to use cross-validation on a training set to select the "best" neural network (and evaluate it on a separate test set) and to use nested ...
1
vote
2answers
880 views

Cross-validation when splitting data into train/dev/test sets

Background: Train set: data used to train the chosen model Dev set: data used to tune the model's parameters Test set: data used to evaluate the performance of the final model How cross-validation ...
4
votes
1answer
412 views

An intuitive understanding of each fold of a nested cross validation for parameter/model tuning

There are several questions on this site essentially asking how nested cross validation for parameter tuning works. A lot of the answers use some jargon that I find difficult to understand, but as far ...
3
votes
1answer
866 views

Compare different classification algorithms after hyperparameter tuning

Let's say I have a classification problem with $c$ classes. For this, I have a data set containing $N$ distinct feature vectors with $n$ features. Let's say $N$ is of the order of $10^5$, and both $c$ ...
6
votes
1answer
864 views

Nested cross validation vs repeated k-fold

I know there are many topics(1,2,3), papers(1,2,3) and websites(1) that discuss this topic at length. However for the past two days I am reading all I can find about the subject and seems I hit a ...
5
votes
2answers
498 views

Which gamma regression model to use for extrapolation?

I'm looking for a regression model which would satify these requirements: My target variable follows the exponential distribution, so to my understanding I should use gamma loss function. I have ...
2
votes
1answer
472 views

Does changing the parameter search space after nested CV introduce optimistic bias?

Suppose I am fitting a Ridge and I decide to search a parameter space for c:[1,2,3]. I perform nested CV on my whole dataset and find the performance not so great. I therefore expand my search space ...
1
vote
1answer
741 views

parameter tuning using nested cross validation

Parameter tuning in SVM has been performed using a nested cross-validation(CV) approach with 45 folds(outer loop) and 13 folds(inner loop). In this process, the outer loop will have 45 prediction ...
5
votes
1answer
425 views

What is the standard procedure for evaluating a user-based CF algorithm with a dataset offline?

I have read some papers and other materials about the evaluation of recommender systems (RS). Most of them discuss the various properties of RS (e.g. accuracy, diversity, etc.), and metrics for ...
1
vote
1answer
634 views

Feature selection & Cross Validation

this is a popular topic here but I have been reading through the different pages and could not find anything related with what I am wondering now. So, I have a data set with X features and I would ...
1
vote
2answers
614 views

K nearest neighbors with nested cross validation

I'm working on a binary classification problem on this dataset, using the k-nn algorithm. For the performance evaluation and the parameter tuning (i.e. the choosing of k) I'm using the nested cross ...
1
vote
1answer
333 views

Feature selection using LASSO and PCA on training data or whole data?

I am using LASSO and PCA for performing feature selection on a classification problem. The dataset consist of 20 features and around 5.7k observations. One of the reviewer comments for this approach ...
3
votes
1answer
452 views

Evaluation of final model in feature selection with nested cross-validation

I am doing feature selection with wrapper method on microarray datasets. I have read several papers and answers here about cross-validation (CV) evaluation on feature selection. Especially the answers ...
3
votes
1answer
273 views

Cross-validation scheme used in the Introduction to Statistical Learning, Chapter 6, Lab 3

I've been really enjoying the Introduction to Statistical Learning textbook so far, and I'm currently working my way through chapter 6. I realize that I am very confused by the process used in lab 3 ...
7
votes
1answer
59 views

An Information Criterion that considers how many variables we can choose from

I am running a multiple regression model and looking to use AIC and BIC to select models. However I notice that both measures do not consider the number of variables we can choose from but only ...
0
votes
1answer
260 views

K-fold CV based model selection with a constraint on the number of features?

I am currently working on project where I need to train a logistic regression classifier with a combined $l_1$/$l_2$-penalty that satisfies a hard on the number of features. Specifically, my dataset ...
0
votes
1answer
178 views

How to validate a model when first exploring model hyperparameter space?

For my class project I am comparing various tree-based ensemble methods such as bagging, boosting, random forest, and AdaBoost against my data set and I can't quite determine my methodology. I know ...
1
vote
1answer
172 views

Toy implementation of nested cross-validation: how to determine number of inner and outer folds and how many iterations to run?

I am reading Cawley and Talbot and saw a post on implementation of nested CV (NCV) as well as numerous posts with good answers on the topic (general NCV, training with full dataset after NCV, how to ...
1
vote
1answer
97 views

Why my validation accuracy and AUC are higher than my training accuracy and AUC?

I have a binary classification problem and I use LightGBM classifier to build my model based on 5 features. I divided my dataset (94 observations) into two parts: Training dataset: 60 observations ...
1
vote
0answers
131 views

Why not use (nested) cross-validation to update weights when building final model?

I have been trying to find an answer to this question for some time. I understand that cross-validation is primarily used for model selection, i.e. to tune parameters/hyperparameters, but I don’t ...
1
vote
2answers
99 views

outer folds errors in nested cross-validation

I have a time series data that I wish to be able to obtain the general performance of it. For that, I use nested cross-validation with time series flavor as described in this amazing blog. As you ...
1
vote
2answers
64 views

hyper parameter tuning AFTER Nested cross-validation

I have read very well the awesome answers and suggestions by @cbeleites and @Dikran Marsupial here for nested CV but I am still confused about something: Basically now I understand that nested CV is ...
0
votes
1answer
87 views

How can I conclude my model performance?

I use cross validation to find a best set of parameters for random forest on my dataset. Then I use the best model to fit my train set and got an average AUC of 0.6883. But I can see the variability ...
1
vote
0answers
80 views

Test set vs nested cross-validation

It seems that similar arguments can be made for using nested cross-validation instead of a simple hold-out test set, as the arguments for using cross-validation instead of a single validation set. The ...
0
votes
0answers
58 views

Nested k-fold cross validation: How to choose hyperparameter for a SVM

I am currently trying to understand how exactly to use nested k-fold cross validation for hyperparameter tuning / model selection. There is one aspect I really cannot get my head around. I found ...
0
votes
1answer
55 views

Performance of a classifier change heavily

I'm using a data set of 32 face persons and a svm-rbf to classify and a random group of 70% for train and 30% for test. The problem is that my results are heavily dependent of the random group used ...
1
vote
0answers
51 views

K-Fold Cross Validation, the right way

I searched for this in many forums, but didn't find any answers that would answer my question (or I didn't understand correctly). So, I will post here. I have a dataset A I have a machine learning ...
1
vote
1answer
33 views

Nested Cross Validation - Which Models Should We Evaluate in the Outer Loop?

Lets assume for example that I am attempting to predict a binary outcome using p predictors in which n>p with methods including a LASSO Regression, a Logistic Regression and SVM with an RBF kernel. ...
1
vote
1answer
29 views

Refit a Neural Network after a (nested) cross validation procedure

During the last years I've been running cross validation procedures (and sometimes nested CV) to have an estimate of my model while also doing hyperparameter search. The usual procedure that people ...
1
vote
1answer
29 views

Testing set accuracy by using cross validation using xgboost with caret

I am working on an xgboost model using caret. I'm using cross validation, but don't know if I'm understanding it correctly. As I understand, it creates multiple training and test sets. Does this mean ...