Methods and principles of selecting a subset of attributes for use in further modelling

learn more… | top users | synonyms (2)

10
votes
0answers
199 views

Speed, computational expenses of PCA, LASSO, elastic net

I am trying to compare computational complexity / estimation speed of three groups of methods for linear regression as distinguished in Hastie et al. "Elements of Statistical Learning" (2nd ed.), ...
5
votes
0answers
67 views

Bayesian lasso vs spike and slab

Suppose I have the likelihood: $$y\sim\mathcal{N}(Xw,\sigma^2I)$$ where I can put either one of the priors: $$ w_i\sim \pi\delta_0+(1-\pi)\mathcal{N}(0,100)\\ \pi=0.9 $$ or $$ w_i\sim \exp(-\lambda|...
5
votes
0answers
405 views

When would I choose Lasso over Elastic Net

What are the scenarios where Lasso is likely to perform better than Elastic Net (out of sample prediction)?
5
votes
0answers
748 views

How exactly does Chi-square feature selection work?

I know that for each feature-class pair, the value of the chi-square statistic is computed and compared against a threshold. I am a little confused though. If there are $m$ features and $k$ classes, ...
4
votes
0answers
298 views

Paradox in model selection (AIC, BIC, to explain or to predict?)

Having read Galit Shmueli's "To Explain or to Predict" (2010) I am puzzled by an apparent contradiction. There are three starting points, AIC- versus BIC-based model choice (end of p. 300 - start of ...
4
votes
0answers
2k views

What kind of feature selection can Chi square test be used for?

Here I am asking about what others commonly do to use chi squared test for feature selection wrt outcome in supervised learning. If I understand correctly, do they test the independence between each ...
4
votes
0answers
138 views

Which variables are driving correlations within groups

I'm running an analysis on a few data sets that each typically have 100-200 cases measured across 120-160 variables - something similar to looking at gene expressions. Each variable is a non-centered ...
4
votes
0answers
186 views

Variable Selection One by One vs Simultaneously

The high dimensional variable selection problem is really popular now. But I have a question: If I do simple linear regression regressing one response variable on 1 covariate at a time first and then ...
3
votes
0answers
65 views

Is there an appropriate order to apply bagging and filter feature selection?

I'm training a (regression) learner on a $p \gg n$ problem, including bagging and filter feature selection (information gain). I'm in doubt though regarding the order of the procedures: Apply the ...
3
votes
0answers
22 views

Does this pattern indicate over-fitting in machine learning?

I am working on a diagnostics project, and trying to improve the performance of a classifier(s). We have over a million features to choose from, so feature selection is a real challenge. To look ...
3
votes
0answers
19 views

Feature Selection Among Groups

I'm trying to do feature selection along a dataset which has: ...
3
votes
0answers
93 views

Important question regarding feature selection methodologies in R concerning the randomness of the results

I'm currently testing some feature selection methodologies/algorithms in R, like the Recursive Feature Elimination from the R caret package, and also the RRF R package, to select a subset of features ...
3
votes
0answers
66 views

How to do dimensionality reduction on a huge data set?

I am working with fMRI data of ~1000 subject. Each subject has a feature vector of ~150 million dimension. So I can only keep the feature vectors of ~10 subjects in memory. What are some algorithms ...
3
votes
0answers
197 views

Why is lasso in matlab much slower than glmnet in R (10 min versus ~1 s)?

I observed that the function lasso in MATLAB is relatively slow. I run many regression problems, with typically 1 to 100 predictors and 200 to 500 observations. In some cases, lasso turned out to be ...
3
votes
0answers
48 views

How can I estimate the influence/significance of the every observation on classification?

There are many ways to estimate the significance of the features on the classification model. But how I can estimate the influence of the every observation on the classification quality? My thinking ...
3
votes
0answers
85 views

Random Forest: Class specific feature importance

I'm using the bigrf R-package to analyse a dataset with ca. 50.000 observations x 120 variables, classified into two groups. After growing a forest of 1000 trees, ...
3
votes
0answers
188 views

How do I use weight vector of SVM and logistic regression for feature importance?

I have trained a SVM and logistic regression classifier on my dataset for binary classification. Both classifier provide a weight vector which is of the size of the number of features. I can use this ...
3
votes
0answers
117 views

How can I get feature importance for Gaussian Naive Bayes classifier?

I have a dataset consisting of 4 classes and around 200 features. I have implemented a Gaussian Naive Bayes classifier. I want now calculate the important of each feature for each pair of classes ...
3
votes
0answers
64 views

Development data set for feature engineering and data exploration

I dont hear this being talked about much: If you want to engineer features and visually explore the data, should you do this on a development set separate from the training and test set? If ...
3
votes
0answers
138 views

LASSO prediction model question

I am trying to create a prediction model with 33 predictors (brain metabolite levels in various regions) and 8 observations (cognitive test scores) with p>>n problem using LASSO in MATLAB (...
3
votes
0answers
122 views

Quantify the information lost given by the Kullback-Leibler divergence measure

Consider there are $N$ individuals and these measure a quantity $X\in \mathbb{R}^{N\times M}$ where $M$ is the number of measurements and let $P(X)$ denote a probability distribution over $X$. The ...
3
votes
0answers
149 views

Does full subset selection suffer from the same handicaps as stepwise regression?

Let's assume $p$ potential predictor variables $X_1,...,X_p$ and a single dependent variable $Y$. Now I evaluate the performance of all possible linear models considering all possible combinations of ...
3
votes
0answers
171 views

Variable reduction by means of ANOVA?

I have a typical problem with several variables and a large amount of data which are not important right now. The goal of the study is to relate variable $Y$ with variables $X_1,X_2,...,X_n$. I have ...
3
votes
0answers
154 views

Mutual information/pointwise mutual information for measuring prediction

I want to measure how well I predict a vector $Y$ (vector not a label) for observation $X$. Both $X$ and $Y$ have the same set of features ($1\times n$). For that, I thought of "scoring" the ...
3
votes
0answers
470 views

Feature importance scores of SVM multiclass one-vs-one design

Info about dataset: 5 classes, 200 trials, 100 features. (I know about the trial to feature ratio being very low, but can not avoid this here and still got well enough classification results.) ...
3
votes
0answers
401 views

Microarray data: suggestions on Feature selection + Model training scheme?

I have a microarray expression dataset (46 samples, thousands of attributes) and I want to perform feature selection first, and, based on this subset of features (shouldn't be more than 4 or 5, based ...
3
votes
0answers
819 views

Variable selection / Dataset reduction for large datasets (in R)

I'm working on a behavoural scorecard modelling exercise, and many of the decisions taken to date have been based on the experience of a consulting credit analyst (whose experience software-wise is ...
3
votes
0answers
248 views

Fast algorithm for variable selection

The (training) data contains 1280 observations with 1415 features. The test set has additional 380 observations. The data is sparse, that is, many of the variables has many zeros and few positive ...
3
votes
0answers
240 views

Sensible to include ratio as a variable in logistic regression?

I'm creating a generalised linear regression using a binomial link function for two variables A and B. From looking at the data it appears that A/B may have discriminatory effect. Is it sensible to ...
2
votes
0answers
59 views

PCA for questionnaire reduction

I'd like to have some opinion regarding if I'm in the right way with my questionnaire reduction. I have a questionnaire with 275 questions and 34 issues (so a couple of questions are related to each ...
2
votes
0answers
31 views

Is stagewised feature engineering/ selection an invalid approach?

Suppose we want to build a regression or classification model. However, the features (independent variables used) are not all ready at one time. This is very realistic in business, because the data ...
2
votes
0answers
33 views

Feature Selection with Categorical Variables: Multicollinearity and Statistical Significance

Building a logistic regression model with three categorical features and one continuous. For simplicity, let's say I have the following features and variables: ...
2
votes
0answers
39 views

How to improve the sensitivity of minority class on imbalanced datasets

I am working on a classifier which stratifies a population of samples into different classes. The class distribution (ground truth) is imbalanced, and the prevalence of each class is: $$\begin{...
2
votes
0answers
64 views

Optimal feature selection

I am working on classification issue. My training set contains of 10D features vectors. As a training model I am going to use Fisher or Neural Network. Here is a plot of the correlation matrix for a ...
2
votes
0answers
28 views

Combinatorial optimization: split features into several subsets to maximize overall score

I tried to reformulate my original problem (that is quite difficult to explain) to a simpler one. Please, take it into account when you may think that the final goal doesn't make too much sense. ...
2
votes
0answers
153 views

Coalitional effect in logistic regression and assessing explanarory variable contribution

I have a problem that could be described as logistic regression with all dichotomous variables: 1 response variable (DV) Y (I would call it later as a feature/violet star) and 5 explanatory variables (...
2
votes
0answers
54 views

Variable selection with multi-variate time series

I currently have a data.frame with 273 variables and 94 rows: ...
2
votes
0answers
128 views

How to use LDA results for feature selection?

I am working on the Forest type mapping dataset which is available in the UCI machine learning repository. I have 27 features to predict the 4 types of forest. I am performing a Linear Discriminant ...
2
votes
0answers
73 views

Selecting multiple hyper-parameters via successive nested cross-validation

Selecting multiple hyper-parameters via successive nested cross-validation I am currently working in a classification task on motion data. Each sample to classify is represented by a set of features ...
2
votes
0answers
32 views

What is the general procedure or general rules for grouping factor levels?

I am attempting to build a predictive (machine-learning) logistic regression model that contains mostly categorical (non-ordinal) variables. As part of a variable selection process I run a Pearson ...
2
votes
0answers
62 views

Feature selection of SVM

My question is three-fold In the context of "Kernelized" support vector machines Is variable/feature selection desirable - especially since we regularize the parameter C to prevent overfitting and ...
2
votes
0answers
148 views

AIC, model selection and variable scale

In looking at the formula for the AIC=-2*(LL)-2k and the formula for log likelihood, LL=-n/2*log(2*pi) - n/2*log(sse/n) - n/2, I notice that the term with sse is sensitive to the scale of the ...
2
votes
0answers
129 views

Generating features: What level of interaction?

I have multi (3) level data indexed by i,j,t. As such, I can generate fixed effects (dummies) for either ij, it, or jt, (and still achieve identification). I can also do i,j,t separately as well. ...
2
votes
0answers
74 views

How to make use of less data of a particular class for better modeling?

I have a dataset, say 9000 rows, with some features. Around 8000 belong to class 1 and 1000 to class 0. So, if I am creating a model with any method say SVM, LR, Random forest the model has a tendency ...
2
votes
0answers
37 views

Can I reuse the dataset set aside for performing t-test based on the following condition?

I have a small number of samples and large number features. For doing the feature selection I'm going to divide my total set into a feature selection set and a test set.I run the t-test on the former ...
2
votes
0answers
104 views

category selection with LASSO

Suppose one has two features: color = {R, G, B} and t-shirt size = {S, M, L} and wants to regress these features on the probability of a sale, call it p. So the model is p ~ color + size. Now, the ...
2
votes
0answers
51 views

Is mRMR feature selection sensitive to imbalanced dataset?

I wish to perform feature selection on a dataset using mRMR but my classes are imbalanced roughly 3:1. Should I down sample the majority class or use the whole dataset? If I use the whole dataset will ...
2
votes
0answers
95 views

What are some classic examples of feature selection in classification?

Is there a classic example showing the importance of good feature selection in classification? The ideal example would be simple, and very easy to understand. I've been volunteered/instructed to put ...
2
votes
0answers
115 views

Confusion related to feature selection

Well my objective is to predict solar energy radiation at a particular location given some features like wind, temperature, humidity ... I have a total data for 10 years where I have the measurement ...
2
votes
0answers
347 views

Features selection by filter methods for multivariate time series

I have a data set in which the samples are multivariate (about 30 variable/features) time series. These samples refer to two classes. I would like to select the variables more relevant to discriminate ...