Methods and principles of selecting a subset of attributes for use in further modelling

learn more… | top users | synonyms (2)

6
votes
1answer
1k views

Feature selection and parameter tuning with caret for random forest

I have data with a few thousand features and I want to do recursive feature selection (RFE) to remove uninformative ones. I do this with caret and RFE. However, I started thinking, if I want to get ...
21
votes
5answers
3k views

Detecting significant predictors out of many independent variables

In a dataset of two non-overlapping populations (patients & healthy, total $n=60$) I would like to find (out of $300$ independent variables) significant predictors for a continuous dependent ...
4
votes
1answer
929 views

Variable selection with LASSO

I am trying to fit a predictive gene-based model in survival analysis. My question is: Can I use LASSO as a variable selection method, and then run a multivariate Cox regression to get the ...
5
votes
3answers
1k views

How to avoid overfitting when using crossvalidation within Genetic Algorithms

This is a long set-up, but the pure intellectual challenge will make it worthwhile I promise ;-) I have marketing data where there is a treatment and a control (i.e a customer gets no treatment). The ...
3
votes
2answers
408 views

Variable analysis in multiple linear regression

I'm investigating how some weather variables (15) affect electricity demand in a specific area during the last 20 years. I was thinking to perform the following steps: 1. Perform Multiple Linear ...
3
votes
1answer
521 views

Are randomForest variable importance values comparable across same variables on different dates?

Are randomForest variable importance comparable across same variables on different dates? I have a data array X which is of size $T\times N\times K$, where $T=1500$, $N=1500$ and $K=10$. ...
6
votes
1answer
3k views

What's the forward stagewise regression algorithm?

Maybe it's just that I'm tired, but I'm having trouble trying to understand the Forward Stagewise Regression algorithm. From "Elements of Statistical Learning" page 60: Forward-stagewise ...
2
votes
0answers
271 views

Procedure for variable selection + logistic regression when n is small, p is large, and data are unbalanced?

I have data that have been collected using case-control procedures, in which the population of positive cases is collected with a random sample of negative cases. This yields 62 positive cases and 179 ...
0
votes
1answer
128 views

Lucene-based text feature construction

When doing the feature construction for text mining, does Lucene has a better performance in terms of classification/clustering result than the traditional bag-of-word approach?
15
votes
3answers
272 views

Is building a multiclass classifier better than several binary ones?

I need to classify URLs into categories. Say I have 15 categories that I'm planning to zero down every URL to. Is a 15-way classifier better? Where I have 15 labels and generate features for each ...
4
votes
1answer
934 views

When does LASSO select correlated predictors?

I'm using the package 'lars' in R with the following code: ...
0
votes
1answer
75 views

Feature selection given non-normal variables?

I'm trying to reduce noise (improve separability) among groups in a data set with 26 numerical variables and 10.000 samples. Each sample is a chemical profile, with each variable indicating the ...
3
votes
1answer
101 views

How to determine significant subgroups of data inputs? [duplicate]

I have a large $(10000 \times 5001)$ table representing $10000$ samples and $5001$ different features of these samples. One of these features represents an output variable of each sample. In other ...
14
votes
3answers
972 views

Model stability when dealing with large $p$, small $n$ problem

Intro: I have a dataset with a classical "large p, small n problem". The number available samples n=150 while the number of possible predictors p=400. The outcome is a continuous variable. I want ...
10
votes
2answers
3k views

Finding the best features in interaction models

I have list of proteins with their feature values. A sample table looks like this: ...
3
votes
1answer
160 views

Choosing the number of features

I remember reading about a general rule of thumb for choosing number of features. It was something like $\sqrt{\log_2(n)}$, where $n$ is the number of samples or $\log_2(n)$. Does anyone remember the ...
4
votes
1answer
2k views

Information gain as a feature selection for 3-class classification problem

I am facing a sentiment analysis task where I am using Naive Bayes to classify documents as Positive, Negative or Neutral. I have thought of using Information Gain as my filter for feature selection. ...
1
vote
0answers
269 views

Guassian Process Regression - feature selection

I'm using guassian process regression to do some modeling. One issue I'm encountering is feature selection for some of my models, which often have many relevant features. I'm not sure what the best ...
2
votes
2answers
562 views

Variable selection in large datasets

I'm looking for an overview of some methods of variable selection. I use datasets with around 6000 variables (the level of missing values is satisfying i.e. there are no variables with 100% missing ...
25
votes
3answers
5k views

Feature selection and cross-validation

I have recently been reading a lot on this site (@Aniko, @Dikran Marsupial, @Erik) and elsewhere about the problem of overfitting occuring with cross validation - (Smialowski et al 2010 ...
10
votes
2answers
1k views

How do you select variables in a regression model?

The traditional approach to variable selection is to find variables that contribute the most to predicting a new response. Recently I learned of an alternative to this. In modeling variables that ...
5
votes
3answers
1k views

Logistic regression performance with high number of predictors

I'm trying to understand the behavior of logistic regression in high dimensional problems (i.e. when you are fitting a logistic regression to data with a high number of predictor variables). Every ...
10
votes
3answers
8k views

Using principal component analysis (PCA) for feature selection

I'm new to feature selection and I was wondering how you would use PCA to perform feature selection. Does PCA compute a relative score for each input variable that you can use to filter out ...
2
votes
1answer
180 views

Feature construction for text mining

In the text mining, besides N-gram model, what are the state-of-art models for building feature space while capturing the dependence among the different words, or capturing the semantic meaning in the ...
4
votes
1answer
938 views

SVM for Image Segmentation?

I turn to this forum for advice with the following problem. If you could please shed some light on any aspect of this question I'd be very grateful. Problem decription: I'm trying to use an SVM to ...
2
votes
1answer
391 views

Is SVM-RFE a “filter” or a “wrapper” feature selection algorithm?

I am wondering whether SVM based recursive feature elimination (Guyon et al., 2000) would be considered a filter or a wrapper method. On the one hand, it "wraps" around an SVM to get the ...
0
votes
1answer
240 views

Missing data and feature selection

My data is 1,785,000 records with 271 features. I'm trying to reduce number of features used to build the model. Q1. while exploring the data I found that some features are almost all missing data, ...
-1
votes
1answer
112 views

High-dimensional dependent binary explanatory variable

I am dealing with a data set containing roughly $n=4000$ binary observations $Y_1, \ldots, Y_n$ with $p=1000$ binary explanatory variables. I suspect that a lot of these explanatory variables are not ...
1
vote
1answer
351 views

SVM importance of predictor variables

I am building a model in R using support vector machine (SVM) with KBF kernel. The model seems to work quite well. I would like to assess the relative importance of predictor variables. Can anyone ...
3
votes
1answer
1k views

Feature selection methods for document classtification

I have a simple document classification problem where i need to classify some documents to a definite set of classes. I need to perform a feature selection (where I will select the most important ...
1
vote
0answers
56 views

What are some common tools, initial approaches to data in a prediction problem when facing too many predictors?

If one is given several hundred features (of both categorical and continuous type) what are some approaches to determining which features to keep or even drop? Data as such is difficult to visualize ...
2
votes
1answer
129 views

Feature selection without target variable

Let's assume I have a NxD matrix X with the N rows being observations and the D columns being features. I would now like to know which are the most "interesting" features of this dataset. I.e. which ...
1
vote
1answer
253 views

Feature selection weighting 2 filters in Naive Bayes

I am trying to do text classification using Naive Bayes. Before training, I would like to make feature selection in order to reduce the feature space dimension. In order to do so, I have thought of ...
12
votes
2answers
9k views

How to deal with multicollinearity when performing variable selection?

I have a dataset with 9 continuous independent variables. I'm trying to select amongst these variables to fit a model to a single percentage (dependent) variable, ...
1
vote
1answer
357 views

Is there a rule of thumb on the relationship between the number of instances and the number of features?

If we build a classifier based on a very small number of instances (say, fewer than 300) and the number of features we are using is very large (say, larger than 100k features). If we decide to ...
4
votes
2answers
1k views

Feature selection for the text mining?

Before performing the task of text mining, we need to select the features for characterizing each given document. Are there any systematic guidance on choosing the document features? How does the ...
0
votes
1answer
147 views

Statistics regarding different data sets having a common response variable

Lets assume one has 2 datasets: with different number of rows (samples) and columns (features). Each of these 2 datasets have a column as a binary response variable. Lets say healthy or not. What sort ...
4
votes
1answer
257 views

Is this a correct procedure for feature selection using cross-validation?

I was looking for information about feature selection and crossvalidation, when I found this post: Feature selection for "final" model when performing cross-validation in machine learning. ...
3
votes
2answers
291 views

Feature selection for disease classification based on tests

I have a dataset of around 100 different subjects Some of them have a disease, some do not (roughly 60:40 disease:no disease) They are subjected to a battery of 15 tests, to see if they are outside ...
1
vote
0answers
209 views

References about univariable vs multivariable variable selection

Suppose I have variables $X_j$, $j=1,\ldots,p$, some of which are correlated, and some continuous output $y$. I want to rank the variables by importance. One way is to do an association test of each ...
2
votes
4answers
3k views

How to convert nominal dataset into numerical dataset?

For my work, im using the multilabel dataset from this webpage. Few dataset which are listed in the page (for, e.g bibtex) have nominal attributes, i.e attribute values are 0 and 1. My queries are ...
4
votes
0answers
689 views

How exactly does Chi-square feature selection work?

I know that for each feature-class pair, the value of the chi-square statistic is computed and compared against a threshold. I am a little confused though. If there are $m$ features and $k$ classes, ...
2
votes
1answer
510 views

Open source implementation elastic net in C or C++

Can anyone provide or point me to a freely available implemention of Elastic Net in C or C++?
4
votes
4answers
951 views

How to know when to stop reducing dimensions with PCA?

I'm using PCA to reduce dimensionality before I feed the data into a classifier. My bootstrap/cross-validation has shown a significant reduction in test error as a result of applying PCA and keeping ...
5
votes
1answer
817 views

How to select the final model with elastic net feature selection, cross validation and SVM?

I have a dataset of some 100 samples, each with >10,000 features, some of which highly correlated. Here's what I am doing currently. Split the data set into three folds. For each fold, 2.1 Run ...
2
votes
0answers
165 views

Non-linear (e.g. RBF kernel) SVM with SCAD penalties implementation

Is there one? I think there's a penalizedSVM package in R but it looks to use a linear kernel. Can't quite tell from the documentation. If it's linear, is there a R package that lets me calculate the ...
1
vote
3answers
2k views

Use of PCA analysis to select variables for a regression analysis [duplicate]

I have too many environmental variables to use in a multiple regression analysis. If I use all the variables the models are just too complex. The use of the PCA axes in the regression analysis was ...
3
votes
1answer
2k views

R knn variable selection

I have a data set that's 200k rows X 50 columns. I'm trying to use a knn model on it but there is huge variance in performance depending on which variables are ...
8
votes
1answer
587 views

Dealing with very large time-series datasets

I have access to a very large dataset. The data is from MEG recordings of people listening to musical excerpts, from one of four genres. The data is as follows: 6 Subjects 3 Experimental repetitions ...
7
votes
1answer
519 views

Automatic feature selection for anomaly detection

What is the best way to automatically select features for anomaly detection? I normally treat Anomaly Detection as an algorithm where the features are selected by human experts: what matters is the ...