30 questions linked to/from Feature selection and cross-validation
68 views

### Cross validation before or after stepwise modeling [duplicate]

I have a dataset of 1931 observations and I intend to predict a binary outcome out of that. There is a list of 128 predictors (both binary and continuous). First I ran logistic regression modeling ...
37k views

### Won't highly-correlated variables in random forest distort accuracy and feature-selection?

In my understanding, highly correlated variables won't cause multi-collinearity issues in random forest model (Please correct me if I'm wrong). However, on the other way, if I have too many variables ...
10k views

### Internal vs external cross-validation and model selection

My understanding is that with cross validation and model selection we try to address two things: P1. Estimate the expected loss on the population when training with our sample P2. Measure and report ...
24k views

### Low classification accuracy, what to do next?

So, I'm a newbie in ML field and I try to do some classification. My goal is to predict the outcome of a sport event. I've gathered some historical data and now try to train a classifier. I got around ...
10k views

### Grid search on k-fold cross validation

I've a dataset of 120 samples in a 10-fold cross validation setting. Currently, I pick the training data of the first holdout and do a 5-fold cross-validation on it to pick the values of gamma and C ...
8k views

### Model selection and cross-validation: The right way

There are numerous threads in CrossValidated on the topic of model selection and cross validation. Here are a few: Internal vs external cross-validation and model selection @DikranMarsupial's top ...
17k views

### Should feature selection be performed only on training data (or all data)?

Should be feature selection performed only on training data (or all data)? I went through some discussions and papers such as Guyon (2003) and Singhi and Liu (2006), but still not sure about right ...
2k views

### Can I perform an exhaustive search with cross-validation for feature selection?

I have been reading some of the posts about feature selection and cross-validation but I still have questions about the correct procedure. Suppose I have a dataset with 10 features and I want to ...
1k views

### Feature selection using cross validation

I am dealing with a typical $p > n$ problem in the medical field. (typically $p \approx 3700$ and $n \approx 100$ ). The dependent variable is binary (healthy/sick) and features are continuous ...
2k views

### Model Tuning and Model Evaluation in Machine Learning

Despite my readings (on stack 1, 2, or in literature (Cawley, 2010; Japkowicz, 2011)), I don't find a clear procedure for tuning and evaluating a model in a classification task. I want to perform a ...
2k views

### Cross-validation and feature selection of a multivariate regression

I've been trying to create a multivariate regression model to fit my training data into the prediction of a value. I've put my data into a matrix X with ...
3k views

### Issues with sequential feature selection [duplicate]

I am trying to do some feature selection in gene expression data with 22215 features. I followed the tutorial here. I initially applied filter method(ttest) to select the features having the best p ...
1k views

### CV on training set with feature selection

I've got a problem with CV on feature selection. I've used a method, but I don't know it's correct... I split my data into 70% training set and 30% test set I work now with my training set. I do on ...