Linked Questions

1
vote
2answers
609 views

K nearest neighbors with nested cross validation

I'm working on a binary classification problem on this dataset, using the k-nn algorithm. For the performance evaluation and the parameter tuning (i.e. the choosing of k) I'm using the nested cross ...
1
vote
1answer
330 views

Feature selection using LASSO and PCA on training data or whole data?

I am using LASSO and PCA for performing feature selection on a classification problem. The dataset consist of 20 features and around 5.7k observations. One of the reviewer comments for this approach ...
3
votes
1answer
272 views

Cross-validation scheme used in the Introduction to Statistical Learning, Chapter 6, Lab 3

I've been really enjoying the Introduction to Statistical Learning textbook so far, and I'm currently working my way through chapter 6. I realize that I am very confused by the process used in lab 3 ...
3
votes
1answer
449 views

Evaluation of final model in feature selection with nested cross-validation

I am doing feature selection with wrapper method on microarray datasets. I have read several papers and answers here about cross-validation (CV) evaluation on feature selection. Especially the answers ...
7
votes
1answer
59 views

An Information Criterion that considers how many variables we can choose from

I am running a multiple regression model and looking to use AIC and BIC to select models. However I notice that both measures do not consider the number of variables we can choose from but only ...
0
votes
1answer
260 views

K-fold CV based model selection with a constraint on the number of features?

I am currently working on project where I need to train a logistic regression classifier with a combined $l_1$/$l_2$-penalty that satisfies a hard on the number of features. Specifically, my dataset ...
0
votes
1answer
178 views

How to validate a model when first exploring model hyperparameter space?

For my class project I am comparing various tree-based ensemble methods such as bagging, boosting, random forest, and AdaBoost against my data set and I can't quite determine my methodology. I know ...
1
vote
1answer
172 views

Toy implementation of nested cross-validation: how to determine number of inner and outer folds and how many iterations to run?

I am reading Cawley and Talbot and saw a post on implementation of nested CV (NCV) as well as numerous posts with good answers on the topic (general NCV, training with full dataset after NCV, how to ...
1
vote
1answer
89 views

Why my validation accuracy and AUC are higher than my training accuracy and AUC?

I have a binary classification problem and I use LightGBM classifier to build my model based on 5 features. I divided my dataset (94 observations) into two parts: Training dataset: 60 observations ...
1
vote
0answers
128 views

Why not use (nested) cross-validation to update weights when building final model?

I have been trying to find an answer to this question for some time. I understand that cross-validation is primarily used for model selection, i.e. to tune parameters/hyperparameters, but I don’t ...
1
vote
2answers
98 views

outer folds errors in nested cross-validation

I have a time series data that I wish to be able to obtain the general performance of it. For that, I use nested cross-validation with time series flavor as described in this amazing blog. As you ...
1
vote
2answers
63 views

hyper parameter tuning AFTER Nested cross-validation

I have read very well the awesome answers and suggestions by @cbeleites and @Dikran Marsupial here for nested CV but I am still confused about something: Basically now I understand that nested CV is ...
0
votes
1answer
87 views

How can I conclude my model performance?

I use cross validation to find a best set of parameters for random forest on my dataset. Then I use the best model to fit my train set and got an average AUC of 0.6883. But I can see the variability ...
1
vote
0answers
79 views

Test set vs nested cross-validation

It seems that similar arguments can be made for using nested cross-validation instead of a simple hold-out test set, as the arguments for using cross-validation instead of a single validation set. The ...
0
votes
0answers
57 views

Nested k-fold cross validation: How to choose hyperparameter for a SVM

I am currently trying to understand how exactly to use nested k-fold cross validation for hyperparameter tuning / model selection. There is one aspect I really cannot get my head around. I found ...

15 30 50 per page