Methods and principles of selecting a subset of attributes for use in further modelling

learn more… | top users | synonyms (2)

18
votes
3answers
3k views

Feature selection and cross-validation

I have recently been reading a lot on this site (@Aniko, @Dikran Marsupial, @Erik) and elsewhere about the problem of overfitting occuring with cross validation - (Smialowski et al 2010 ...
19
votes
3answers
3k views

Variables are often adjusted (e.g. standardised) before making a model - when is this a good idea, and when is it a bad one?

In what circumstances would you want to, or not want to scale or standardize a variable prior to model fitting? And what are the advantages / disadvantages of scaling a variable?
27
votes
8answers
3k views

Feature selection for “final” model when performing cross-validation in machine learning

I am getting a bit confused about feature selection and machine learning and I was wondering if you could help me out. I have a microarray dataset that is classified into two groups and has 1000s of ...
17
votes
3answers
1k views

Why is variable selection necessary?

Common data-based variable selection procedures (for example, forward, backward, stepwise, all subsets) tend to yield models with undesirable properties, including: Coefficients biased away from ...
1
vote
3answers
1k views

Use of PCA analysis to select variables for a regression analysis

I have too many environmental variables to use in a multiple regression analysis. If I use all the variables the models are just too complex. The use of the PCA axes in the regression analysis was ...
12
votes
2answers
5k views

How to deal with multicollinearity when performing variable selection?

I have a dataset with 9 continuous independent variables. I'm trying to select amongst these variables to fit a model to a single percentage (dependent) variable, ...
12
votes
2answers
391 views

Why does the Lasso provide Variable Selection?

I've been reading Elements of Statistical Learning, and I would like to know why the Lasso provides variable selection and ridge regression doesn't. Both methods minimize the residual sum of squares ...
16
votes
6answers
2k views

Variable selection procedure for binary classification

What are the variable/feature selection that you prefer for binary classification when there are many more variables/feature than observations in the learning set? The aim here is to discuss what is ...
13
votes
2answers
1k views

Significance testing or cross validation?

Two common approaches for selecting correlated variables are significance tests and cross validation. What problem does each try to solve and when would I prefer one over the other?
5
votes
2answers
780 views

Significance of categorical predictor in logistic regression

I am having trouble interpreting the z values for categorical variables in logistic regression. In the example below I have a categorical variable with 3 classes and according to the z value, CLASS2 ...
9
votes
3answers
298 views

What can cause PCA to worsen results of a classifier?

I have a classifier that I'm doing cross-validation on, along with a hundred or so features that I'm doing forward selection on to find optimal combinations of features. I also compare this against ...
5
votes
5answers
857 views

Is using the same data for feature selection and cross-validation biased or not?

We have a small dataset (about 250 samples * 100 features) on which we want to build a binary classifier after selecting the best feature subset. Lets say that we partition the data into: Training, ...
4
votes
1answer
67 views

Variable selection for regression - the subselect package

No regular here will be unaware of the perils of using stepwise and similar automatic methods for variable selection in regression analysis. But preferred alternatives, such as the lasso or ...
3
votes
3answers
4k views

How does one interpret SVM feature weights?

I am trying to interpret the variable weights given by fitting a linear SVM. (I'm using scikit-learn): ...
16
votes
5answers
2k views

Detecting significant predictors out of 300 independent variables

In a dataset of two non-overlapping populations (patients & healthy, total $n=60$) I would like to find (out of $300$ independent variables) significant predictors for a continuous dependent ...
7
votes
4answers
759 views

Features for time series classification

I consider the problem of (multiclass) classification based on time series of variable length $T$, that is, to find a function $$f(X_T) = y \in [1..K]\\ \text{for } X_T = (x_1, \dots, x_T)\\ ...
10
votes
4answers
1k views

Application of machine learning techniques in small sample clinical studies

What do you think about applying machine learning techniques, like Random Forests or penalized regression (with L1 or L2 penalty, or a combination thereof) in small sample clinical studies when the ...
7
votes
4answers
1k views

Low classification accuracy, what to do next?

So, I'm a newbie in ML field and I try to do some classification. My goal is to predict the outcome of a sport event. I've gathered some historical data and now try to train a classifier. I got around ...
5
votes
2answers
1k views

Feature selection with Random Forests

I have a dataset with mostly financial variables (120 features, 4k examples) which are mostly highly correlated and very noisy (technical indicators, for example) so I would like to select about max ...
3
votes
3answers
3k views

Feature Selection Packages in R

I am very new to R. I am learning machine learning right now. Very sorry, if this question appears to be very basic. I am trying to find a good feature selection package in R. I went through Boruta ...
11
votes
2answers
543 views

Model stability when dealing with large $p$, small $n$ problem

Intro: I have a dataset with a classical "large p, small n problem". The number available samples n=150 while the number of possible predictors p=400. The outcome is a continuous variable. I want ...
5
votes
6answers
519 views

What machine learning algorithms are good for estimating which features are more important?

I have data with a minimum number of features that don't change, and a few additional features that can change and have a big impact on the outcome. My data-set looks like this: Features are A, B, C ...
1
vote
1answer
75 views

Using Principal Components Analysis for feature selection

I have a dataset D made of m samples, and n features with n >> m. For each sample I have a score s which I would like to ...
3
votes
3answers
1k views

Logistic regression performance with high number of predictors

I'm trying to understand the behavior of logistic regression in high dimensional problems (i.e. when you are fitting a logistic regression to data with a high number of predictor variables). Every ...
3
votes
1answer
601 views

Feature selection methods for document classtification

I have a simple document classification problem where i need to classify some documents to a definite set of classes. I need to perform a feature selection (where I will select the most important ...
3
votes
1answer
403 views

How can I assess how descriptive feature vectors are?

I am assessing how good different features are for unsupervised classification of a set of objects. For each different feature I test, I have computed a feature vector that describes the object. I ...
2
votes
2answers
427 views

Variables importance: who can do the most pushups?

I don't know enough math to formulate an intelligent question on this so I'll give an example. I'd like an answer to my example but also I'd like to know the jargon I need to be able to research it ...
5
votes
3answers
262 views

Can I perform an exhaustive search with cross-validation for feature selection?

I have been reading some of the posts about feature selection and cross-validation but I still have questions about the correct procedure. Suppose I have a dataset with 10 features and I want to ...
3
votes
2answers
85 views

Random search for the optimal number of input features and optimal number of hidden layers for a MLP?

I've performed a random search in hypothesis space $$\{(c,h)| c \in U[1,256]; h\in U[1,100];c \in \mathrm{Z} \text{ and } h \in \mathrm{Z}\}$$ that defines the parameters of a standard multilayer ...
2
votes
1answer
1k views

What is “feature space”?

What is the definition of "feature space"? For example, When reading about SVMs, I read about "mapping to feature space". When reading about CART, I read about "partitioning to feature space". I ...
2
votes
1answer
155 views

Selecting optimal number of input features and optimal number of hidden layers for a MLP?

What is the best way to select parameters for a binary neural network classifier? More specifically I have 265 features ranked according to Mutual Information Criterion. I have to determine the ...
2
votes
1answer
495 views

How to select the final model with elastic net feature selection, cross validation and SVM?

I have a dataset of some 100 samples, each with >10,000 features, some of which highly correlated. Here's what I am doing currently. Split the data set into three folds. For each fold, 2.1 Run ...
1
vote
2answers
701 views

Issues with feature selection in matlab

I am trying to use sequentialfs to do some feature selection in matlab. I have huge dimensional data of 22215 features. When I tried to use sequentialfs with svm as classifier so that it selects the ...
0
votes
1answer
116 views

How to model a multi-dimensional feature set for classification

I am new to statistical modelling and so please pardon if the question appears trivial. I have a set of multi-dimensional data ($T$) where each dimension represents features ($f_i$) obtained from a ...
8
votes
3answers
1k views

The use of median polish for feature selection

In a paper I was reading recently I came across the following bit in their data analysis section: The data table was then split into tissues and cell lines, and the two subtables were separately ...
4
votes
2answers
198 views

Why does increasing the number of features reduce performance?

I'm trying to gain an intuition as to why increasing the number of features could reduce performance. I'm currently using an LDA classifier which performs better bivariately among certain features ...
2
votes
1answer
163 views

How to perform step() when n < p in R?

I am trying to perform stepwise regression for variable selection in R. In matlab, the stepwisefit function is able to work in ...
1
vote
0answers
207 views

Feature importance

The extremely randomized trees classifier (scikitlearn) provides a (multivariate) feature importance measurement Ensemble methods/feature importance evaluation. For each feature, the classifier ...
1
vote
2answers
873 views

How to integrate principal components with GLM?

How would I integrate the output of a principal components analysis with a GLM (assuming the PCA is used for variable selection for the GLM)?
1
vote
3answers
649 views

How to identify suitable variables to assess confounding, mediation and effect modification?

Imagine that you are planning a study about risk behaviours among HIV positive injecting drug users. All the individuals included in the sample are injecting drugs and all are HIV positive. The main ...
0
votes
3answers
205 views

Feature selection before SVM

I have a simple but difficult question. Does feature selection before SVM help? I have a data set that has ~1100 features but a lot of these are redundant data / uncorrelated data. Can someone give me ...