Linked Questions

6
votes
1answer
917 views

Nested cross validation vs repeated k-fold

I know there are many topics(1,2,3), papers(1,2,3) and websites(1) that discuss this topic at length. However for the past two days I am reading all I can find about the subject and seems I hit a ...
1
vote
0answers
71 views

K-Fold Cross Validation, the right way

I searched for this in many forums, but didn't find any answers that would answer my question (or I didn't understand correctly). So, I will post here. I have a dataset A I have a machine learning ...
1
vote
1answer
41 views

Refit a Neural Network after a (nested) cross validation procedure

During the last years I've been running cross validation procedures (and sometimes nested CV) to have an estimate of my model while also doing hyperparameter search. The usual procedure that people ...
1
vote
1answer
41 views

Nested Cross Validation - Which Models Should We Evaluate in the Outer Loop?

Lets assume for example that I am attempting to predict a binary outcome using p predictors in which n>p with methods including a LASSO Regression, a Logistic Regression and SVM with an RBF kernel. ...
3
votes
1answer
471 views

Evaluation of final model in feature selection with nested cross-validation

I am doing feature selection with wrapper method on microarray datasets. I have read several papers and answers here about cross-validation (CV) evaluation on feature selection. Especially the answers ...
1
vote
2answers
107 views

outer folds errors in nested cross-validation

I have a time series data that I wish to be able to obtain the general performance of it. For that, I use nested cross-validation with time series flavor as described in this amazing blog. As you ...
1
vote
2answers
110 views

hyper parameter tuning AFTER Nested cross-validation

I have read very well the awesome answers and suggestions by @cbeleites and @Dikran Marsupial here for nested CV but I am still confused about something: Basically now I understand that nested CV is ...
1
vote
1answer
156 views

Why my validation accuracy and AUC are higher than my training accuracy and AUC?

I have a binary classification problem and I use LightGBM classifier to build my model based on 5 features. I divided my dataset (94 observations) into two parts: Training dataset: 60 observations ...
1
vote
2answers
719 views

K nearest neighbors with nested cross validation

I'm working on a binary classification problem on this dataset, using the k-nn algorithm. For the performance evaluation and the parameter tuning (i.e. the choosing of k) I'm using the nested cross ...
1
vote
1answer
40 views

Testing set accuracy by using cross validation using xgboost with caret

I am working on an xgboost model using caret. I'm using cross validation, but don't know if I'm understanding it correctly. As I understand, it creates multiple training and test sets. Does this mean ...
1
vote
0answers
84 views

Test set vs nested cross-validation

It seems that similar arguments can be made for using nested cross-validation instead of a simple hold-out test set, as the arguments for using cross-validation instead of a single validation set. The ...
1
vote
1answer
755 views

parameter tuning using nested cross validation

Parameter tuning in SVM has been performed using a nested cross-validation(CV) approach with 45 folds(outer loop) and 13 folds(inner loop). In this process, the outer loop will have 45 prediction ...
5
votes
2answers
604 views

Which gamma regression model to use for extrapolation?

I'm looking for a regression model which would satify these requirements: My target variable follows the exponential distribution, so to my understanding I should use gamma loss function. I have ...
1
vote
0answers
139 views

Why not use (nested) cross-validation to update weights when building final model?

I have been trying to find an answer to this question for some time. I understand that cross-validation is primarily used for model selection, i.e. to tune parameters/hyperparameters, but I don’t ...
17
votes
1answer
2k views

How to build the final model and tune probability threshold after nested cross-validation?

Firstly, apologies for posting a question that has already been discussed at length here, here, here, here, here, and for reheating an old topic. I know @DikranMarsupial has written about this topic ...
3
votes
5answers
647 views

Cross-validation: Which classifier to use in the end? [duplicate]

This might sound like a very simple question, but I haven't been able to find an answer to it, yet: Assuming I am working on a binary classification task and I am using k-fold cross-validation to ...
1
vote
1answer
679 views

Feature selection & Cross Validation

this is a popular topic here but I have been reading through the different pages and could not find anything related with what I am wondering now. So, I have a data set with X features and I would ...
17
votes
3answers
6k views

How to get hyper parameters in nested cross validation?

I have read the following posts for nested cross validation and still am not 100% sure what I am to do with model selection with nested cross validation: Nested cross validation for model selection ...
18
votes
4answers
19k views

Cross validation and parameter tuning

Can anyone tell me what exactly a cross-validation analysis gives as result? Is it just the average accuracy or does it give any model with parameters tuned? Because, I heard somewhere that cross-...
1
vote
1answer
363 views

Feature selection using LASSO and PCA on training data or whole data?

I am using LASSO and PCA for performing feature selection on a classification problem. The dataset consist of 20 features and around 5.7k observations. One of the reviewer comments for this approach ...
8
votes
1answer
2k views

How to obtain optimal hyperparameters after nested cross validation?

In general, if we have a large dataset, we can split it into (1) training, (2) validation, and (3) test. We use validation to identify the best hyperparameters in cross validation (e.g., C in SVM) and ...
1
vote
2answers
949 views

Cross-validation when splitting data into train/dev/test sets

Background: Train set: data used to train the chosen model Dev set: data used to tune the model's parameters Test set: data used to evaluate the performance of the final model How cross-validation ...
5
votes
1answer
459 views

An intuitive understanding of each fold of a nested cross validation for parameter/model tuning

There are several questions on this site essentially asking how nested cross validation for parameter tuning works. A lot of the answers use some jargon that I find difficult to understand, but as far ...
2
votes
5answers
2k views

10-fold cross validation, why having a validation set?

I have my data stratified in 10 folders. So far I was using 9 of them to train the model, and the remaining one for testing it. A sudden thought just crossed my mind saying "you might be cheating". ...
1
vote
1answer
998 views

Comparing classification algorithms using cross validation and caret's train

I am having issues understanding some concepts of algorithm comparison/parameter optimization/cross-validation in R Let's say I want to compare two classification algorithms, such as Random Forests ...
1
vote
1answer
1k views

Feature selection: is nested cross-validation needed?

I have about 150 samples 1000 features (ranked by their importance by Relieff score). My question is, what would be the best approach to: choose the hyper parameters choose the optimal number of ...
2
votes
1answer
487 views

Does changing the parameter search space after nested CV introduce optimistic bias?

Suppose I am fitting a Ridge and I decide to search a parameter space for c:[1,2,3]. I perform nested CV on my whole dataset and find the performance not so great. I therefore expand my search space ...
1
vote
1answer
175 views

Toy implementation of nested cross-validation: how to determine number of inner and outer folds and how many iterations to run?

I am reading Cawley and Talbot and saw a post on implementation of nested CV (NCV) as well as numerous posts with good answers on the topic (general NCV, training with full dataset after NCV, how to ...
0
votes
1answer
179 views

How to validate a model when first exploring model hyperparameter space?

For my class project I am comparing various tree-based ensemble methods such as bagging, boosting, random forest, and AdaBoost against my data set and I can't quite determine my methodology. I know ...
10
votes
2answers
8k views

Implementation of nested cross-validation

I'm trying to figure out if my understanding of nested cross-validation is correct, therefore I wrote this toy example to see if I'm right: ...

15 30 50 per page