Methods and principles of selecting a subset of attributes for use in further modelling

learn more… | top users | synonyms (2)

28
votes
8answers
3k views

Feature selection for “final” model when performing cross-validation in machine learning

I am getting a bit confused about feature selection and machine learning and I was wondering if you could help me out. I have a microarray dataset that is classified into two groups and has 1000s of ...
20
votes
3answers
4k views

Variables are often adjusted (e.g. standardised) before making a model - when is this a good idea, and when is it a bad one?

In what circumstances would you want to, or not want to scale or standardize a variable prior to model fitting? And what are the advantages / disadvantages of scaling a variable?
18
votes
3answers
4k views

Feature selection and cross-validation

I have recently been reading a lot on this site (@Aniko, @Dikran Marsupial, @Erik) and elsewhere about the problem of overfitting occuring with cross validation - (Smialowski et al 2010 ...
18
votes
3answers
3k views

What are disadvantages of using the lasso for variable selection for regression?

From what I know, using lasso for variable selection handles the problem of correlated inputs. Also, since it is equivalent to Least Angle Regression, it is not slow computationally. However, many ...
17
votes
5answers
2k views

Detecting significant predictors out of 300 independent variables

In a dataset of two non-overlapping populations (patients & healthy, total $n=60$) I would like to find (out of $300$ independent variables) significant predictors for a continuous dependent ...
17
votes
3answers
1k views

Why is variable selection necessary?

Common data-based variable selection procedures (for example, forward, backward, stepwise, all subsets) tend to yield models with undesirable properties, including: Coefficients biased away from ...
17
votes
2answers
2k views

Variable importance from SVM

How to obtain a variable (attribute) importance using SVM?
16
votes
6answers
2k views

Variable selection procedure for binary classification

What are the variable/feature selection that you prefer for binary classification when there are many more variables/feature than observations in the learning set? The aim here is to discuss what is ...
16
votes
4answers
526 views

When wouldn't I use LASSO for model selection?

Assume that you need to build a linear model to make predictions for new observations, and that there is uncertainty about which subset of variables should be included in the model. You are only ...
15
votes
3answers
238 views

Is building a multiclass classifier better than several binary ones?

I need to classify URLs into categories. Say I have 15 categories that I'm planning to zero down every URL to. Is a 15-way classifier better? Where I have 15 labels and generate features for each ...
14
votes
2answers
1k views

Significance testing or cross validation?

Two common approaches for selecting correlated variables are significance tests and cross validation. What problem does each try to solve and when would I prefer one over the other?
13
votes
2answers
560 views

Why does the Lasso provide Variable Selection?

I've been reading Elements of Statistical Learning, and I would like to know why the Lasso provides variable selection and ridge regression doesn't. Both methods minimize the residual sum of squares ...
12
votes
2answers
6k views

How to deal with multicollinearity when performing variable selection?

I have a dataset with 9 continuous independent variables. I'm trying to select amongst these variables to fit a model to a single percentage (dependent) variable, ...
12
votes
2answers
643 views

Model stability when dealing with large $p$, small $n$ problem

Intro: I have a dataset with a classical "large p, small n problem". The number available samples n=150 while the number of possible predictors p=400. The outcome is a continuous variable. I want ...
11
votes
2answers
834 views

Best approach for model selection Bayesian or cross-validation?

When trying to select among various models or the number of features to include for, say prediction I can think of two approaches. Split the data into training and test sets. Better still, use ...
11
votes
2answers
461 views

Bayesian variable selection — does it really work?

I thought I might toy with some Bayesian variable selection, following a nice blog post and the linked papers therein. I wrote a program in rjags (where I am quite a rookie) and fetched price data ...
10
votes
4answers
1k views

Application of machine learning techniques in small sample clinical studies

What do you think about applying machine learning techniques, like Random Forests or penalized regression (with L1 or L2 penalty, or a combination thereof) in small sample clinical studies when the ...
9
votes
3answers
331 views

What can cause PCA to worsen results of a classifier?

I have a classifier that I'm doing cross-validation on, along with a hundred or so features that I'm doing forward selection on to find optimal combinations of features. I also compare this against ...
9
votes
2answers
233 views

Domain-agnostic feature engineering that retains semantic meaning?

Feature engineering is often an important component to machine learning (it was used heavily to win the KDD Cup in 2010). However, I find that most feature engineering techniques either destroy ...
9
votes
3answers
3k views

Improving the SVM classification of diabetes

I am using SVM to predict diabetes. I am using the BRFSS data set for this purpose. The data set has the dimensions of $432607 \times 136$ and is skewed. The percentage of ...
9
votes
1answer
583 views

Random permutation test for feature selection

I am confused about permutation analysis for feature selection in a logistic regression context. Could you provide a clear explanation of the random permutation test and how does it applies to feature ...
9
votes
2answers
530 views

Best methods of feature selection for nonparametric regression

A newbie question here. I am currently performing a nonparametric regression using the np package in R. I have 7 features and using a brute force approach I identified the best 3. But, soon I will ...
8
votes
4answers
2k views

Test accuracy higher than training. How to interpret?

I've a dataset containing at most 150 examples (split into training & test), with many features (higher than 1000). I need to compare classifiers and feature selection methods which perform well ...
8
votes
2answers
2k views

Finding the best features in interaction models

I have list of proteins with their feature values. A sample table looks like this: ...
8
votes
3answers
2k views

How to apply LASSO to IRLS (logistic regression)?

I have programmed a logistic regression using the IRLS algorithm. I would like to apply a LASSO penalization in order to automatically select the right features. At each iteration, the following is ...
8
votes
3answers
1k views

The use of median polish for feature selection

In a paper I was reading recently I came across the following bit in their data analysis section: The data table was then split into tissues and cell lines, and the two subtables were separately ...
8
votes
4answers
1k views

Features for time series classification

I consider the problem of (multiclass) classification based on time series of variable length $T$, that is, to find a function $$f(X_T) = y \in [1..K]\\ \text{for } X_T = (x_1, \dots, x_T)\\ ...
8
votes
2answers
1k views

Soft-thresholding vs. Lasso penalization

I am trying to summarize what I understood so far in penalized multivariate analysis with high-dimensional data sets, and I still struggle through getting a proper definition of soft-thresholding vs. ...
8
votes
1answer
435 views

Dealing with very large time-series datasets

I have access to a very large dataset. The data is from MEG recordings of people listening to musical excerpts, from one of four genres. The data is as follows: 6 Subjects 3 Experimental repetitions ...
8
votes
0answers
1k views

How do you select variables in a regression model?

The traditional approach to variable selection is to find variables that contribute the most to predicting a new response. Recently I learned of an alternative to this. In modeling variables that ...
7
votes
6answers
2k views

Algorithms and methods for attribute/feature selection?

I have data with continuous class and I'm searching for good methods to reduce number of attributes. Now I'm using correlation based filters, random forests and Gram–Schmidt algorithm. What I want ...
7
votes
3answers
955 views

Computing best subset of predictors for linear regression

For the selection of predictors in multivariate linear regression with $p$ suitable predictors, what methods are available to find an 'optimal' subset of the predictors without explicitly testing all ...
7
votes
2answers
2k views

Feature selection with Random Forests

I have a dataset with mostly financial variables (120 features, 4k examples) which are mostly highly correlated and very noisy (technical indicators, for example) so I would like to select about max ...
7
votes
4answers
1k views

Low classification accuracy, what to do next?

So, I'm a newbie in ML field and I try to do some classification. My goal is to predict the outcome of a sport event. I've gathered some historical data and now try to train a classifier. I got around ...
7
votes
4answers
2k views

Is there a way to use cross validation to do variable/feature selection in R?

I have a data set with about 70 variables that I'd like to cut down. What I'm looking to do is use CV to find most useful variables in the following fashion. 1) Randomly select say 20 variables. ...
7
votes
3answers
962 views

Feature selection using mutual information in Matlab

I am trying to apply the idea of mutual information to feature selection, as described in these lecture notes (on page 5). My platform is Matlab. One problem I find when computing mutual information ...
7
votes
1answer
179 views

How to quantify redundancy of features?

I have three features that I use to solve a classification problem. Originally, these features produced boolean values, so I could evaluate their redundancy by looking at how much the sets of positive ...
7
votes
1answer
349 views

Automatic feature selection for anomaly detection

What is the best way to automatically select features for anomaly detection? I normally treat Anomaly Detection as an algorithm where the features are selected by human experts: what matters is the ...
6
votes
2answers
390 views

Fisher Distance for feature selection

I'm currently working for EEG signal classification from 3 electrodes. I want to have a simple feature selection algorithm that is independent with the classification process. From the feature ...
6
votes
1answer
584 views

If p > n, the lasso selects at most n variables

One of the motivations for the elastic net was the following limitation of LASSO: "In the p > n case, the lasso selects at most n variables before it saturates, because of the nature of the convex ...
6
votes
1answer
2k views

What's the forward stagewise regression algorithm?

Maybe it's just that I'm tired, but I'm having trouble trying to understand the Forward Stagewise Regression algorithm. From "Elements of Statistical Learning" page 60: Forward-stagewise ...
6
votes
0answers
55 views
+50

Lasso-ing the order of a lag?

Suppose I have longitudinal data of the form $\mathbf Y = (Y_1, \ldots, Y_J) \sim \mathcal N(\mu, \Sigma)$ (I have multiple observations, this is just the form of a single one). I'm interested in ...
5
votes
2answers
4k views

Using PCA for feature selection

I'm new to feature selection and I was wondering how you would use PCA to perform feature selection. Does PCA compute a relative score for each input variable that you can use to filter out ...
5
votes
2answers
2k views

Significance of categorical predictor in logistic regression

I am having trouble interpreting the z values for categorical variables in logistic regression. In the example below I have a categorical variable with 3 classes and according to the z value, CLASS2 ...
5
votes
6answers
561 views

What machine learning algorithms are good for estimating which features are more important?

I have data with a minimum number of features that don't change, and a few additional features that can change and have a big impact on the outcome. My data-set looks like this: Features are A, B, C ...
5
votes
3answers
314 views

Can I perform an exhaustive search with cross-validation for feature selection?

I have been reading some of the posts about feature selection and cross-validation but I still have questions about the correct procedure. Suppose I have a dataset with 10 features and I want to ...
5
votes
2answers
373 views

Feature selection for low probability event prediction

I'm currently trying to predict the probability for low probability events (~1%). I have large DB with ~200,000 vectors (~2000 plus examples) with ~200 features. I'm trying to find the the best ...
5
votes
5answers
1k views

Is using the same data for feature selection and cross-validation biased or not?

We have a small dataset (about 250 samples * 100 features) on which we want to build a binary classifier after selecting the best feature subset. Lets say that we partition the data into: Training, ...
5
votes
3answers
1k views

How to avoid overfitting when using crossvalidation within Genetic Algorithms

This is a long set-up, but the pure intellectual challenge will make it worthwhile I promise ;-) I have marketing data where there is a treatment and a control (i.e a customer gets no treatment). The ...
5
votes
3answers
748 views

Selecting the best subset of variables for parsimonious binary logistic regression models

In addition to PROC VARCLUS, randomForest, glmnet, and assessing multicollinearity among potential predictor variables (without regards to the outcome of interest), I am seeking other methods of ...