Methods and principles of selecting a subset of attributes for use in further modelling

learn more… | top users | synonyms (2)

7
votes
0answers
88 views

Speed, computational expenses of PCA, LASSO, elastic net

I am trying to compare computational complexity / estimation speed of three groups of methods for linear regression as distinguished in Hastie et al. "Elements of Statistical Learning" (2nd ed.), ...
5
votes
0answers
31 views

Bayesian lasso vs spike and slab

Suppose I have the likelihood: $$y\sim\mathcal{N}(Xw,\sigma^2I)$$ where I can put either one of the priors: $$ w_i\sim \pi\delta_0+(1-\pi)\mathcal{N}(0,100)\\ \pi=0.9 $$ or $$ w_i\sim ...
5
votes
0answers
352 views

When would I choose Lasso over Elastic Net

What are the scenarios where Lasso is likely to perform better than Elastic Net (out of sample prediction)?
4
votes
0answers
123 views

Which variables are driving correlations within groups

I'm running an analysis on a few data sets that each typically have 100-200 cases measured across 120-160 variables - something similar to looking at gene expressions. Each variable is a non-centered ...
4
votes
0answers
179 views

Variable Selection One by One vs Simultaneously

The high dimensional variable selection problem is really popular now. But I have a question: If I do simple linear regression regressing one response variable on 1 covariate at a time first and then ...
4
votes
0answers
725 views

How exactly does Chi-square feature selection work?

I know that for each feature-class pair, the value of the chi-square statistic is computed and compared against a threshold. I am a little confused though. If there are $m$ features and $k$ classes, ...
3
votes
0answers
59 views

Why is lasso in matlab much slower than glmnet in R (10 min versus ~1 s)?

I observed that the function lasso in MATLAB is relatively slow. I run many regression problems, with typically 1 to 100 predictors and 200 to 500 observations. In some cases, lasso turned out to be ...
3
votes
0answers
48 views

How can I estimate the influence/significance of the every observation on classification?

There are many ways to estimate the significance of the features on the classification model. But how I can estimate the influence of the every observation on the classification quality? My thinking ...
3
votes
0answers
68 views

Random Forest: Class specific feature importance

I'm using the bigrf R-package to analyse a dataset with ca. 50.000 observations x 120 variables, classified into two groups. After growing a forest of 1000 trees, ...
3
votes
0answers
53 views

Development data set for feature engineering and data exploration

I dont hear this being talked about much: If you want to engineer features and visually explore the data, should you do this on a development set separate from the training and test set? If ...
3
votes
0answers
101 views

LASSO prediction model question

I am trying to create a prediction model with 33 predictors (brain metabolite levels in various regions) and 8 observations (cognitive test scores) with p>>n problem using LASSO in MATLAB (...
3
votes
0answers
136 views

Does full subset selection suffer from the same handicaps as stepwise regression?

Let's assume $p$ potential predictor variables $X_1,...,X_p$ and a single dependent variable $Y$. Now I evaluate the performance of all possible linear models considering all possible combinations of ...
3
votes
0answers
155 views

Variable reduction by means of ANOVA?

I have a typical problem with several variables and a large amount of data which are not important right now. The goal of the study is to relate variable $Y$ with variables $X_1,X_2,...,X_n$. I have ...
3
votes
0answers
126 views

Mutual information/pointwise mutual information for measuring prediction

I want to measure how well I predict a vector $Y$ (vector not a label) for observation $X$. Both $X$ and $Y$ have the same set of features ($1\times n$). For that, I thought of "scoring" the ...
3
votes
0answers
348 views

Microarray data: suggestions on Feature selection + Model training scheme?

I have a microarray expression dataset (46 samples, thousands of attributes) and I want to perform feature selection first, and, based on this subset of features (shouldn't be more than 4 or 5, based ...
3
votes
0answers
679 views

Variable selection / Dataset reduction for large datasets (in R)

I'm working on a behavoural scorecard modelling exercise, and many of the decisions taken to date have been based on the experience of a consulting credit analyst (whose experience software-wise is ...
3
votes
0answers
306 views

How does LASSO select among collinear predictors?

I'm looking for an intuitive answer why a GLM LASSO model selects a specific predictor out of a group of highly correlated ones, and why it does so differently then the best subset feature selection. ...
3
votes
0answers
220 views

Fast algorithm for variable selection

The (training) data contains 1280 observations with 1415 features. The test set has additional 380 observations. The data is sparse, that is, many of the variables has many zeros and few positive ...
3
votes
0answers
229 views

Sensible to include ratio as a variable in logistic regression?

I'm creating a generalised linear regression using a binomial link function for two variables A and B. From looking at the data it appears that A/B may have discriminatory effect. Is it sensible to ...
3
votes
0answers
147 views

Random forest like techniques (bagging, random feature subset) for SGD methods

Are there any well-known results/tools/literature on using bagging and random feature subset selection for regression or SGD-based methods?
2
votes
0answers
83 views

How to use LDA results for feature selection?

I am working on the Forest type mapping dataset which is available in the UCI machine learning repository. I have 27 features to predict the 4 types of forest. I am performing a Linear Discriminant ...
2
votes
0answers
50 views

Selecting multiple hyper-parameters via successive nested cross-validation

Selecting multiple hyper-parameters via successive nested cross-validation I am currently working in a classification task on motion data. Each sample to classify is represented by a set of features ...
2
votes
0answers
150 views

Paradox in model selection (AIC, BIC, to explain or to predict?)

Having read Galit Shmueli's "To Explain or to Predict" (2010) I am puzzled by an apparent contradiction. There are three starting points, AIC- versus BIC-based model choice (end of p. 300 - start of ...
2
votes
0answers
103 views

How do I use weight vector of SVM and logistic regression for feature importance?

I have trained a SVM and logistic regression classifier on my dataset for binary classification. Both classifier provide a weight vector which is of the size of the number of features. I can use this ...
2
votes
0answers
46 views

Feature selection of SVM

My question is three-fold In the context of "Kernelized" support vector machines Is variable/feature selection desirable - especially since we regularize the parameter C to prevent overfitting and ...
2
votes
0answers
106 views

Generating features: What level of interaction?

I have multi (3) level data indexed by i,j,t. As such, I can generate fixed effects (dummies) for either ij, it, or jt, (and still achieve identification). I can also do i,j,t separately as well. ...
2
votes
0answers
648 views

What kind of feature selection can Chi square test be used for?

Here I am asking about what others commonly do to use chi squared test for feature selection wrt outcome in supervised learning. If I understand correctly, do they test the independence between ...
2
votes
0answers
66 views

How to make use of less data of a particular class for better modeling?

I have a dataset, say 9000 rows, with some features. Around 8000 belong to class 1 and 1000 to class 0. So, if I am creating a model with any method say SVM, LR, Random forest the model has a tendency ...
2
votes
0answers
32 views

Can I reuse the dataset set aside for performing t-test based on the following condition?

I have a small number of samples and large number features. For doing the feature selection I'm going to divide my total set into a feature selection set and a test set.I run the t-test on the former ...
2
votes
0answers
87 views

category selection with LASSO

Suppose one has two features: color = {R, G, B} and t-shirt size = {S, M, L} and wants to regress these features on the probability of a sale, call it p. So the model is p ~ color + size. Now, the ...
2
votes
0answers
100 views

Quantify the information lost given by the Kullback-Leibler divergence measure

Consider there are $N$ individuals and these measure a quantity $X\in \mathbb{R}^{N\times M}$ where $M$ is the number of measurements and let $P(X)$ denote a probability distribution over $X$. The ...
2
votes
0answers
472 views

Linear regression for feature selection

Imagine we regress y on x1...x4. Now, we want to find out if ...
2
votes
0answers
101 views

Confusion related to feature selection

Well my objective is to predict solar energy radiation at a particular location given some features like wind, temperature, humidity ... I have a total data for 10 years where I have the measurement ...
2
votes
0answers
274 views

Features selection by filter methods for multivariate time series

I have a data set in which the samples are multivariate (about 30 variable/features) time series. These samples refer to two classes. I would like to select the variables more relevant to discriminate ...
2
votes
0answers
94 views

Out-of-bag estimate biased by correlated features

I have a data set with a small number of samples (322) and a large number of features (318.976). My data consists of images, and I want to train a binary classifier. Since I have such a small amount ...
2
votes
0answers
44 views

Correct order of performing imputation and variable selection

This is a general question about performing data analysis. I have a data set with ~1000 sample size and 200 features. Some of features have more than 50% missing or even higher. The missing pattern is ...
2
votes
0answers
400 views

Feature importance scores of SVM multiclass one-vs-one design

Info about dataset: 5 classes, 200 trials, 100 features. (I know about the trial to feature ratio being very low, but can not avoid this here and still got well enough classification results.) ...
2
votes
0answers
44 views

Why do correlations matter in simulating data to compare classifiers when p >> N?

In genomics and computational biology, expression data sets contain a much larger number of features (p) than the number of observations (N). I wanted to simulate data where p>>N to compare the ...
2
votes
0answers
375 views

LASSO vs AIC for feature selection with the Cox model

I have some questions about the Lasso. After using the AIC or BIC to select a model, the model is fit with the variables selected in order to get the standard errors of the estimates with CIs, ...
2
votes
0answers
332 views

Univariate feature ranking in classification

Scikit-learn has function to evaluate the F-statistics for univariate feature importance feature selection. According to the web page they are calculating ANOVA F value. If I understood correctly, ...
2
votes
0answers
360 views

Kernel in PenalizedSVM R package

There is not option to select kernel in penalizedSVM R package. What kernel do they use? Is there some other R package with penalized SVM methods where I can choose various kernels?
2
votes
0answers
284 views

Procedure for variable selection + logistic regression when n is small, p is large, and data are unbalanced?

I have data that have been collected using case-control procedures, in which the population of positive cases is collected with a random sample of negative cases. This yields 62 positive cases and 179 ...
2
votes
0answers
183 views

Non-linear (e.g. RBF kernel) SVM with SCAD penalties implementation

Is there one? I think there's a penalizedSVM package in R but it looks to use a linear kernel. Can't quite tell from the documentation. If it's linear, is there a R package that lets me calculate the ...
2
votes
0answers
869 views

Error using rfe in caret package in R

I am doing some exploratory data analysis in the Heritage Health Prize , and have come across a weird error using R's caret package. In the dataset, I've created a dataframe counting how many times a ...
2
votes
0answers
125 views

New development in variable selection in clustering using MCMC?

The latest general framework I know in MCMC-based wrapper method(doing variable selection and clustering simultaneously) are the paper "Bayesian variable selection in clustering high-dimensional data" ...
2
votes
0answers
164 views

What should I do to compare different sets of data?

I am a beginner in statistics, and I want to learn machine learning :). Therefore, I have gathered some sample data to practice. But, the problem is I want to create a feature (or attribute), which is ...
1
vote
0answers
16 views

Choosing between feature selection and regularization to overcome over-fitting in categorical regression

In order to overcome over-fitting during a regression process over categorical features, one can either 1) Apply L1/L2/Elastic regularization during the regression, for example as answered here ...
1
vote
0answers
19 views

Variable importance in regression with large number of missing values

I have a dataset with multiple (approximately 20) categorical and ordinal predictors and a numerical outcome and I am trying to understand which and how each of these predictors affect the outcome ...
1
vote
0answers
24 views

grouping attributes in RF and GBM

i have a dataset with 1000 samples and ~11k features (SNP markers). i have identified 100 additional binary features describing the markers themselves so i have a ...
1
vote
0answers
30 views