Linked Questions

1
vote
0answers
68 views

Cross validation before or after stepwise modeling [duplicate]

I have a dataset of 1931 observations and I intend to predict a binary outcome out of that. There is a list of 128 predictors (both binary and continuous). First I ran logistic regression modeling ...
33
votes
2answers
37k views

Won't highly-correlated variables in random forest distort accuracy and feature-selection?

In my understanding, highly correlated variables won't cause multi-collinearity issues in random forest model (Please correct me if I'm wrong). However, on the other way, if I have too many variables ...
26
votes
4answers
10k views

Internal vs external cross-validation and model selection

My understanding is that with cross validation and model selection we try to address two things: P1. Estimate the expected loss on the population when training with our sample P2. Measure and report ...
16
votes
4answers
24k views

Low classification accuracy, what to do next?

So, I'm a newbie in ML field and I try to do some classification. My goal is to predict the outcome of a sport event. I've gathered some historical data and now try to train a classifier. I got around ...
16
votes
3answers
10k views

Grid search on k-fold cross validation

I've a dataset of 120 samples in a 10-fold cross validation setting. Currently, I pick the training data of the first holdout and do a 5-fold cross-validation on it to pick the values of gamma and C ...
34
votes
2answers
8k views

Model selection and cross-validation: The right way

There are numerous threads in CrossValidated on the topic of model selection and cross validation. Here are a few: Internal vs external cross-validation and model selection @DikranMarsupial's top ...
10
votes
3answers
17k views

Should feature selection be performed only on training data (or all data)?

Should be feature selection performed only on training data (or all data)? I went through some discussions and papers such as Guyon (2003) and Singhi and Liu (2006), but still not sure about right ...
8
votes
3answers
2k views

Can I perform an exhaustive search with cross-validation for feature selection?

I have been reading some of the posts about feature selection and cross-validation but I still have questions about the correct procedure. Suppose I have a dataset with 10 features and I want to ...
3
votes
2answers
1k views

Feature selection using cross validation

I am dealing with a typical $p > n$ problem in the medical field. (typically $p \approx 3700$ and $n \approx 100$ ). The dependent variable is binary (healthy/sick) and features are continuous ...
3
votes
2answers
2k views

Model Tuning and Model Evaluation in Machine Learning

Despite my readings (on stack 1, 2, or in literature (Cawley, 2010; Japkowicz, 2011)), I don't find a clear procedure for tuning and evaluating a model in a classification task. I want to perform a ...
4
votes
2answers
2k views

Cross-validation and feature selection of a multivariate regression

I've been trying to create a multivariate regression model to fit my training data into the prediction of a value. I've put my data into a matrix X with ...
1
vote
1answer
3k views

Issues with sequential feature selection [duplicate]

I am trying to do some feature selection in gene expression data with 22215 features. I followed the tutorial here. I initially applied filter method(ttest) to select the features having the best p ...
1
vote
2answers
1k views

CV on training set with feature selection

I've got a problem with CV on feature selection. I've used a method, but I don't know it's correct... I split my data into 70% training set and 30% test set I work now with my training set. I do on ...
1
vote
1answer
523 views

How can I use synthetic data to validate my classification model?

Through R and based on a microarray gene expression dataset (60 samples in total-30 cancer and 30 control samples) and R package caret, i have performed a feature selection regarding a binary ...
4
votes
1answer
664 views

Feature selection and training on the same sample

Is feature selection and training on the same sample a bad idea? I want to emphasize that I am not going to use test set for feature selection. If I use the whole train set for feature selection and ...

15 30 50 per page