Methods and principles of selecting a subset of attributes for use in further modelling

learn more… | top users | synonyms (2)

19
votes
3answers
18k views

Using principal component analysis (PCA) for feature selection

I'm new to feature selection and I was wondering how you would use PCA to perform feature selection. Does PCA compute a relative score for each input variable that you can use to filter out ...
37
votes
3answers
11k views

Variables are often adjusted (e.g. standardised) before making a model - when is this a good idea, and when is it a bad one?

In what circumstances would you want to, or not want to scale or standardize a variable prior to model fitting? And what are the advantages / disadvantages of scaling a variable?
34
votes
3answers
10k views

Feature selection and cross-validation

I have recently been reading a lot on this site (@Aniko, @Dikran Marsupial, @Erik) and elsewhere about the problem of overfitting occuring with cross validation - (Smialowski et al 2010 ...
44
votes
8answers
10k views

Feature selection for “final” model when performing cross-validation in machine learning

I am getting a bit confused about feature selection and machine learning and I was wondering if you could help me out. I have a microarray dataset that is classified into two groups and has 1000s of ...
26
votes
2answers
3k views

Why does the Lasso provide Variable Selection?

I've been reading Elements of Statistical Learning, and I would like to know why the Lasso provides variable selection and ridge regression doesn't. Both methods minimize the residual sum of squares ...
11
votes
2answers
13k views

Significance of categorical predictor in logistic regression

I am having trouble interpreting the z values for categorical variables in logistic regression. In the example below I have a categorical variable with 3 classes and according to the z value, CLASS2 ...
1
vote
2answers
245 views

How to account for participants in a study design?

I have a conceptual problem. I want to find out if stress during the day leads to (stronger) teeth grinding (bruxism) at night. I have a number of participants. They will fill in a self-report ...
22
votes
4answers
2k views

How can SVM 'find' an infinite feature space where linear separation is always possible?

What is the intuition behind the fact that an SVM with a Gaussian Kernel has infinite dimensional feature space?
26
votes
4answers
3k views

Why is variable selection necessary?

Common data-based variable selection procedures (for example, forward, backward, stepwise, all subsets) tend to yield models with undesirable properties, including: Coefficients biased away from ...
15
votes
5answers
1k views

What can cause PCA to worsen results of a classifier?

I have a classifier that I'm doing cross-validation on, along with a hundred or so features that I'm doing forward selection on to find optimal combinations of features. I also compare this against ...
25
votes
5answers
7k views

Detecting significant predictors out of many independent variables

In a dataset of two non-overlapping populations (patients & healthy, total $n=60$) I would like to find (out of $300$ independent variables) significant predictors for a continuous dependent ...
12
votes
3answers
22k views

How does one interpret SVM feature weights?

I am trying to interpret the variable weights given by fitting a linear SVM. (I'm using scikit-learn): ...
2
votes
2answers
1k views

How to select a subset of variables from my original long list in order to perform logistic regression analysis?

My situation: small sample size: 116 binary outcome variable long list of explanatory variables: 44 explanatory variables did not come from the top of my head; their choice was based on the ...
8
votes
5answers
21k views

Feature Selection Packages in R, which do both regression and classification

I am very new to R. I am learning machine learning right now. Very sorry, if this question appears to be very basic. I am trying to find a good feature selection package in R. I went through Boruta ...
9
votes
4answers
361 views

How can top principal components retain the predictive power on a dependent variable?

Suppose I am running a regression $Y \sim X$. Why by selecting top $k$ principle components of $X$, does the model retain its predictive power on $Y$? I understand that from ...
1
vote
3answers
4k views

Use of PCA analysis to select variables for a regression analysis [duplicate]

I have too many environmental variables to use in a multiple regression analysis. If I use all the variables the models are just too complex. The use of the PCA axes in the regression analysis was ...
14
votes
2answers
14k views

How to deal with multicollinearity when performing variable selection?

I have a dataset with 9 continuous independent variables. I'm trying to select amongst these variables to fit a model to a single percentage (dependent) variable, ...
14
votes
2answers
11k views

Feature selection with Random Forests

I have a dataset with mostly financial variables (120 features, 4k examples) which are mostly highly correlated and very noisy (technical indicators, for example) so I would like to select about max ...
6
votes
4answers
4k views

Variablity in cv.glmnet results

I am using cv.glment to find predictors. The set-up I use is as follows: ...
6
votes
1answer
11k views

The main effect will be non-significant if the interaction is significant? [duplicate]

I am using linear mixed models to identify important factors, and it turns out that: A: significant B: not significant ...
4
votes
1answer
82 views

How to fit weights into Q-values with linear function approximation

In reinforcement learning, linear function approximation is often used when large state spaces are present. (When look up tables become unfeasible.) The form of the $Q-$value with linear function ...
1
vote
3answers
650 views

Feature selection before SVM

I have a simple but difficult question. Does feature selection before SVM help? I have a data set that has ~1100 features but a lot of these are redundant data / uncorrelated data. Can someone give me ...
24
votes
7answers
6k views

Variable selection procedure for binary classification

What are the variable/feature selection that you prefer for binary classification when there are many more variables/feature than observations in the learning set? The aim here is to discuss what is ...
17
votes
2answers
2k views

Best approach for model selection Bayesian or cross-validation?

When trying to select among various models or the number of features to include for, say prediction I can think of two approaches. Split the data into training and test sets. Better still, use ...
15
votes
5answers
28k views

Using LASSO from lars (or glmnet) package in R for variable selection

Sorry if this question comes across a little basic. I am looking to use LASSO variable selection for a multiple linear regression model in R. I have 15 predictors, one of which is categorical(will ...
17
votes
2answers
3k views

Significance testing or cross validation?

Two common approaches for selecting correlated variables are significance tests and cross validation. What problem does each try to solve and when would I prefer one over the other?
6
votes
3answers
3k views

Logistic regression performance with high number of predictors

I'm trying to understand the behavior of logistic regression in high dimensional problems (i.e. when you are fitting a logistic regression to data with a high number of predictor variables). Every ...
3
votes
0answers
244 views

Paradox in model selection (AIC, BIC, to explain or to predict?)

Having read Galit Shmueli's "To Explain or to Predict" (2010) I am puzzled by an apparent contradiction. There are three starting points, AIC- versus BIC-based model choice (end of p. 300 - start of ...
6
votes
2answers
2k views

Is it possible to use kernel PCA for feature selection?

Is it possible to use kernel principal component analysis (kPCA) for Latent Semantic Indexing (LSI) in the same way as PCA is used? I perform LSI in R using the ...
1
vote
2answers
2k views

Issues with feature selection in matlab

I am trying to use sequentialfs to do some feature selection in matlab. I have huge dimensional data of 22215 features. When I tried to use sequentialfs with svm as classifier so that it selects the ...
2
votes
1answer
617 views

How to perform step() when n < p in R?

I am trying to perform stepwise regression for variable selection in R. In matlab, the stepwisefit function is able to work in ...
1
vote
1answer
334 views

Which variable relative importance method to use?

Following is a plot from relaimpo package of R which shows relative importance of predictor variables for regression of mpg on all other variables in mtcars dataset. The relative importance is ...
28
votes
4answers
2k views

When wouldn't I use LASSO for model selection?

Assume that you need to build a linear model to make predictions for new observations, and that there is uncertainty about which subset of variables should be included in the model. You are only ...
22
votes
2answers
7k views

Variable importance from SVM

How to obtain a variable (attribute) importance using SVM?
4
votes
1answer
561 views

What to conclude from this lasso plot (glmnet)

Following is the plot of glmnet with default alpha(1, hence lasso) using mtcars data set in R with mpg as the DV and others as ...
7
votes
5answers
3k views

Is using the same data for feature selection and cross-validation biased or not?

We have a small dataset (about 250 samples * 100 features) on which we want to build a binary classifier after selecting the best feature subset. Lets say that we partition the data into: Training, ...
6
votes
6answers
1k views

What machine learning algorithms are good for estimating which features are more important?

I have data with a minimum number of features that don't change, and a few additional features that can change and have a big impact on the outcome. My data-set looks like this: Features are A, B, C ...
6
votes
1answer
2k views

When does LASSO select correlated predictors?

I'm using the package 'lars' in R with the following code: ...
6
votes
3answers
793 views

Can I perform an exhaustive search with cross-validation for feature selection?

I have been reading some of the posts about feature selection and cross-validation but I still have questions about the correct procedure. Suppose I have a dataset with 10 features and I want to ...
6
votes
3answers
2k views

How to avoid overfitting when using crossvalidation within Genetic Algorithms

This is a long set-up, but the pure intellectual challenge will make it worthwhile I promise ;-) I have marketing data where there is a treatment and a control (i.e a customer gets no treatment). The ...
5
votes
3answers
291 views

Regression in $p\gg N$ setting (predicting drug efficiency from gene expression with 30k predictors and ~30 samples)

I have a dataset of 29 cell lines and the IC50 values of a test drug. I want to find a relation between the gene expression profiles of each cell line (nearly 31000 genes) and the IC50 values. My ...
4
votes
3answers
3k views

How to use principal components as predictors in GLM?

How would I use the output of a principal components analysis (PCA) in a generalized linear model (GLM), assuming the PCA is used for variable selection for the GLM? Clarification: I want to use PCA ...
1
vote
1answer
645 views

What does the varImp function in the caret package actually compute for a glmnet (elastic net) object

I am fitting an elastic net model with glmnet via the caret package with 189 predictors and a binomial criteria (a,b) ...
4
votes
1answer
427 views

Variable selection for regression - the subselect package

No regular here will be unaware of the perils of using stepwise and similar automatic methods for variable selection in regression analysis. But preferred alternatives, such as the lasso or ...
6
votes
1answer
4k views

Gini decrease and Gini impurity of children nodes

I'm working on the Gini feature importance measure for random forest. Therefore, I need to calculate the Gini decrease in node impurity. Here is the way I do so, which leads to a conflict with the ...
5
votes
2answers
870 views

Why does increasing the number of features reduce performance?

I'm trying to gain an intuition as to why increasing the number of features could reduce performance. I'm currently using an LDA classifier which performs better bivariately among certain features ...
3
votes
6answers
145 views

How to compare features and classifiers which achieve perfect accuracy?

So I'm looking to compare different combinations of features and classifiers. But I'm getting a lot of combinations that achieve 100% cross validation accuracy. I'm trying to figure out how I would ...
2
votes
2answers
617 views

Model Tuning and Model Evaluation in Machine Learning

Despite my readings (on stack 1, 2, or in literature (Cawley, 2010; Japkowicz, 2011)), I don't find a clear procedure for tuning and evaluating a model in a classification task. I want to perform a ...
1
vote
2answers
311 views

GBM: Predict the response variable measured in {0,20}

I need to predict the response that has values in {0,20}. Should it be used as a factor or as a numeric value? How does it influence on the prediction error? I am using GBM with the Gaussian ...
0
votes
1answer
61 views

What is the best way to select variables for clogit model?

I am doing clogit model (clogit of survival package) with around 150 independent variables which are highly correlated. I have to select the combinations of the variables so that the model will be the ...