0
votes
1answer
30 views

Grid Search for hyperparameter and feature selection

So I need to select my hyperparameters and also my features. A full grid search of the space of hyperparameters and features is too computationally intensive, so what I am doing instead is for each ...
1
vote
0answers
24 views

How to determine appropriate number of features and also which features to select?

So I have a dataset which I am using K fold cross validation on to select the number of features and which features should be selected. As I understand it, I would set the number of features to be ...
0
votes
2answers
35 views

Selecting features and estimating their out-of-sample performance with cross-validation

I have only a small dataset. I want to 1. select the most predictive features out of a large candidate pool and 2. get an estimate of their expected predictive performance. In the elements of ...
3
votes
2answers
218 views

Cross-validation and feature selection of a multivariate regression

I've been trying to create a multivariate regression model to fit my training data into the prediction of a value. I've put my data into a matrix X with ...
0
votes
1answer
36 views

How to compare features and classifiers which achieve perfect accuracy?

So I'm looking to compare different combinations of features and classifiers. But I'm getting a lot of combinations that achieve 100% cross validation accuracy. I'm trying to figure out how I would ...
2
votes
3answers
21 views

How to compare features and classifiers which achieve perfect accuracy?

So I'm looking to compare different combinations of features and classifiers. But I'm getting a lot of combinations that achieve 100% cross validation accuracy. I'm trying to figure out how I would ...
0
votes
0answers
17 views

Neural net: variable distribution, validation and number of variables - unexpected results

I have been experimenting with ANNs on one of my datasets, they seem to have the potential to be quite effective in explaining my Y variation. Something i am finding is that they very much benefit ...
1
vote
2answers
107 views

CV on training set with feature selection

I've got a problem with CV on feature selection. I've used a method, but I don't know it's correct... I split my data into 70% training set and 30% test set I work now with my training set. I do on ...
0
votes
1answer
123 views

Feature selection and cross validation

I'm working on a project and I would like to know if the following strategy is good/correct. Sorry if this is a basic/stupid idea (I'm new to this). The input is a dataset with 2.500 features and ...
1
vote
0answers
106 views

Cross validation for variable selection and coefficient shrinkage?

Is cross validation an appropriate technique for variable selection and regression coefficient shrinkage? A former colleague of mine used 10-fold CV to compare the regression coefficients from the ...
3
votes
2answers
157 views

Should feature selection be performed only on training data (or all data)?

Should be feature selection performed only on training data (or all data)? I went through some discussions and papers such as [1,2], but still not sure about right answer. My experiment setup is as ...
3
votes
1answer
148 views

Feature selection in the training set

I have a classifier, and I am using leave one out cross-validation to assess its performance. On each iteration, I divide the dataset into training and testing sets. The testing set is just the ...
3
votes
1answer
179 views

Is it possible to compare two feature selections algorithms by cross-validations?

Assume I have two feature selection algorithms, A and B, which are developed based on SVM. I applied these two algorithms on the same dataset, a Liver Cancer dataset (400 features & 150 samples), ...
1
vote
2answers
769 views

Feature selection using caret + repeatedcv

I am using caret and repeatedcv with repeats for feature selection. That is, ...
1
vote
1answer
226 views

How to define in R the most important variables?

I have a data set with my target gene and more than thousand transcriptional factors somehow correlated with this gene. There is data of these factors in more than 70 variable conditions. What I'm ...
5
votes
5answers
863 views

Is using the same data for feature selection and cross-validation biased or not?

We have a small dataset (about 250 samples * 100 features) on which we want to build a binary classifier after selecting the best feature subset. Lets say that we partition the data into: Training, ...
5
votes
3answers
264 views

Can I perform an exhaustive search with cross-validation for feature selection?

I have been reading some of the posts about feature selection and cross-validation but I still have questions about the correct procedure. Suppose I have a dataset with 10 features and I want to ...
11
votes
2answers
546 views

Model stability when dealing with large $p$, small $n$ problem

Intro: I have a dataset with a classical "large p, small n problem". The number available samples n=150 while the number of possible predictors p=400. The outcome is a continuous variable. I want ...
18
votes
3answers
3k views

Feature selection and cross-validation

I have recently been reading a lot on this site (@Aniko, @Dikran Marsupial, @Erik) and elsewhere about the problem of overfitting occuring with cross validation - (Smialowski et al 2010 ...
4
votes
1answer
191 views

Is this a correct procedure for feature selection using cross-validation?

I was looking for information about feature selection and crossvalidation, when I found this post: Feature selection for "final" model when performing cross-validation in machine learning. ...
2
votes
1answer
496 views

How to select the final model with elastic net feature selection, cross validation and SVM?

I have a dataset of some 100 samples, each with >10,000 features, some of which highly correlated. Here's what I am doing currently. Split the data set into three folds. For each fold, 2.1 Run ...
7
votes
4answers
2k views

Is there a way to use cross validation to do variable/feature selection in R?

I have a data set with about 70 variables that I'd like to cut down. What I'm looking to do is use CV to find most useful variables in the following fashion. 1) Randomly select say 20 variables. ...
11
votes
2answers
740 views

Best approach for model selection Bayesian or cross-validation?

When trying to select among various models or the number of features to include for, say prediction I can think of two approaches. Split the data into training and test sets. Better still, use ...
13
votes
2answers
1k views

Significance testing or cross validation?

Two common approaches for selecting correlated variables are significance tests and cross validation. What problem does each try to solve and when would I prefer one over the other?
27
votes
8answers
3k views

Feature selection for “final” model when performing cross-validation in machine learning

I am getting a bit confused about feature selection and machine learning and I was wondering if you could help me out. I have a microarray dataset that is classified into two groups and has 1000s of ...