Questions tagged [feature-selection]

Methods and principles of selecting a subset of attributes for use in further modelling

813 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
15
votes
1answer
3k views

How can I get feature importance for Gaussian Naive Bayes classifier?

I have a dataset consisting of 4 classes and around 200 features. I have implemented a Gaussian Naive Bayes classifier. I want now calculate the importance of each feature for each pair of classes ...
8
votes
1answer
2k views

Feature selection using chi squared for continuous features

I'm looking at univariate feature selection. A method that is often described, is to look at the p-values for a $\chi^2$-test. However, I'm confused as to how this works for continuous variables. 1. ...
8
votes
1answer
1k views

Random Forest: Class specific feature importance

I'm using the bigrf R-package to analyse a dataset with ca. 50.000 observations x 120 variables, classified into two groups. After growing a forest of 1000 trees, ...
6
votes
0answers
94 views

Features for binary time-series event prediction

This question is somewhat inspired by the answer to Features for time series classification. The difference to that question is that I have a dataset with multi-dimensional time-series where I have ...
6
votes
0answers
1k views

When would I choose Lasso over Elastic Net

What are the scenarios where Lasso is likely to perform better than Elastic Net (out of sample prediction)?
5
votes
1answer
761 views

Does Attention Help with standard auto-encoders

I understand the use of attention mechanisms in the encoder-decoder for sequence-to-sequence problem such as a language translator. I am just trying to figure out whether it is possible to use ...
5
votes
0answers
103 views

Using covariates from penalized regression model in unpenalized model

The good news where I am is that researchers are doing less stepwise covariate selection now that I've introduced penalized regression. The bad news is that researchers want to use elastic-net ...
5
votes
0answers
270 views

Consistency of variable selection based on Lasso estimator for high-dimensional data

According to this paper by Meinshausen and Bühlmann (2006) the variable subset selection coming out of a Lasso is not always consistent in high-dimensional cases. It is bounded by the neighbourhood ...
5
votes
0answers
356 views

Is there an appropriate order to apply bagging and filter feature selection?

I'm training a (regression) learner on a $p \gg n$ problem, including bagging and filter feature selection (information gain). I'm in doubt though regarding the order of the procedures: Apply the ...
4
votes
0answers
114 views

Boruta Algorithm for Logistic Regression?

Is it okay to use a Boruta algorithm to select features for a logistic regression? I read several sources, including the source package as well as this site explaining what Boruta does. My ...
4
votes
2answers
2k views

Feature Selection in unbalanced data

I was always taught 3 things: Training algorithms (rf, trees, etc) don't perform well with unbalanced data. I should balance data only after performing feature selection (mainly to keep variables ...
4
votes
1answer
38 views

How do these matrices form an order-$4$-tensor?

I'm reading this paper on a convolutional neural network for modelling sentences, and I'm having some trouble understanding section $3.5$. Please consider the following text: We denote a feature map ...
4
votes
0answers
494 views

Is it valid to use random forests for feature selection in a time series problem?

I'm working on a time series problem, with additional predictors. While I'm exploring various ways to approach the problem, one possible way is to turn the time series problem into a supervised ...
4
votes
0answers
410 views

What does the Cholesky decomposition of a correlation matrix tell you?

In this answer, the Cholesky decomposition of a correlation matrix is suggested as the means for testing for multicollinearity. I have a dataset that I am certain has high collinearity. I did the ...
4
votes
0answers
737 views

Effectiveness of Standardization and Normalization in Machine Learning

I am just studying the basics of machine learning and had a question about the standardisation and normalisation of the features and its effectiveness. I have read this CrossValidated question and ...
4
votes
0answers
318 views

The method of knock-offs by Barber & Candes for variable selection and FDR control

The knock-off method is a recent approach to variable selection and FDR control presented in two papers to be found here https://statweb.stanford.edu/~candes/papers/FDR_regression.pdf and here https://...
4
votes
0answers
318 views

classification: concatenating descriptors vs. using multiple classifiers

Consider a typical machine learning problem where you try to do object classification from a high-dimensional set of features. Suppose we know that the features are actually a collection of distinct "...
4
votes
0answers
658 views

Most important original features in PCA: can one multiply eigenvectors by the explained variance?

I would like to know the importance of the original features in principle component analysis. See this Stackoverflow link for an example of what I mean (with a code example). The question is: can you ...
4
votes
0answers
838 views

Gibbs sampling for spike and slab priors

In Spike and slab variable selection (equation 4) there is a model setup of the form $\beta_k | \lambda_k, \tau_k \sim \text{Normal} (0, \lambda_k \tau_k^2)$ $\lambda_k | \nu_0, w \sim (1-w)\delta_{\...
4
votes
1answer
166 views

Match model selection strategies with modelling objectives

I am confused trying to match different model selection strategies with different modelling objectives. (Unfortunately, my confusion is reflected in the length of the post. Please be patient.) Model ...
4
votes
0answers
356 views

Akaike information criterion for categorical and numerical data

How should I compute AIC for categorical and for numeric variables in classification problems? I see in Chapter 6 of Zumel and Mount that they use AIC before they train classification algorithms (...
4
votes
0answers
228 views

Quantify the information lost given by the Kullback-Leibler divergence measure

Consider there are $N$ individuals and these measure a quantity $X\in \mathbb{R}^{N\times M}$ where $M$ is the number of measurements and let $P(X)$ denote a probability distribution over $X$. The ...
4
votes
0answers
229 views

Does full subset selection suffer from the same handicaps as stepwise regression?

Let's assume $p$ potential predictor variables $X_1,...,X_p$ and a single dependent variable $Y$. Now I evaluate the performance of all possible linear models considering all possible combinations of ...
4
votes
0answers
181 views

Which variables are driving correlations within groups

I'm running an analysis on a few data sets that each typically have 100-200 cases measured across 120-160 variables - something similar to looking at gene expressions. Each variable is a non-centered ...
4
votes
0answers
388 views

Fast algorithm for variable selection

The (training) data contains 1280 observations with 1415 features. The test set has additional 380 observations. The data is sparse, that is, many of the variables has many zeros and few positive ...
4
votes
1answer
282 views

Determining conserved features using a Bayesian approach

I would like to perform some sort of binary classification, and my data set consists of 100 examples (for each class), which are vectors with 2500 elements. Ideally, I would like to determine which ...
4
votes
0answers
673 views

Kernel in PenalizedSVM R package

There is not option to select kernel in penalizedSVM R package. What kernel do they use? Is there some other R package with penalized SVM methods where I can choose various kernels?
4
votes
0answers
433 views

Sensible to include ratio as a variable in logistic regression?

I'm creating a generalised linear regression using a binomial link function for two variables A and B. From looking at the data it appears that A/B may have discriminatory effect. Is it sensible to ...
3
votes
0answers
32 views

Variable Importance: Bias towards variables with many categories / continuous scale

It seems to be a well-established phenomenon that variable importance measures based on "increased node impurity" tend to be biased in favor of categorical variables with relatively many categories, ...
3
votes
1answer
43 views

In linear regression, what is the difference between performing variable selection before assessing multicollinearity or vice versa?

If you have a number of variables you're interested in and want to perform linear regression, is there a clear preference between: Method A. Perform variable selection techniques (e.g. using ...
3
votes
0answers
163 views

Reducing variables before LASSO

I have around 1200 cases, 150 events, each with roughly 100 predictors. Some of the predictors are highly correlated, and the end-game is to derive a predictive model with as few predictors as ...
3
votes
0answers
80 views

Why does lasso return unstable features when using the same data?

I am using scikit-learn to shrink my data set having around 800 features. It is a very noisy data (market and economic data) To my best knowledge, lasso returns same features for the same data set. ...
3
votes
0answers
199 views

Elastic Net and collinearity

I am performing elastic net for variable selection on a dataset of 95 records and 41 variables. The response is a continuous numerical. I choose the alpha and lambda parameters through 10 fold cross ...
3
votes
0answers
208 views

Gradient Boosting: When to stop doing manual feature engineering?

As an example Let's assume we do have feature-columns $X_1$ and $X_2$ and $X_3$ and target $Y$, and it just so happens that $Y$ is the noisy spearman correlation of the three features. Would it help ...
3
votes
0answers
55 views

Variable selection with tree-structured covariates?

Let's say I want to do regression and that there's a categorical variable which has an inherent tree structure. Using an example from my field of linguistics, let's say I'm trying to predict a binary ...
3
votes
0answers
280 views

Evolutionary algorithms for model selection

I've recently come across a few encounters where people are using genetic programming or genetic algorithms to build "best" performing models. gplearn is an example of genetic programming used for ...
3
votes
0answers
36 views

logistic regression - inclusion of wrongly-informative variable

So I am not sure if this question makes sense at all, but I will try to explain. I want to build a logistic regression scoring model that learns automatically (updates) when sample is updated with ...
3
votes
0answers
1k views

Feature Selection Using Principal Feature Analysis and Variables Factor Map

I am trying to select the most important features that explain the variability of my data using an unsupervised approach in python (would consider R though). This is after I performed a PCA and ...
3
votes
0answers
1k views

Recursive feature elimination (rfe) performs poorly with binary outcome

Backward elimination with random forests does not work as expected in a simple test case with binary outcome and three continuous predictors. Below, I generate a binary outcome based on a single ...
3
votes
0answers
640 views

Random forest feature importance vs. feature correlation to PCA eigenvectors

I have recently built a random forest model for classifying two types of cancer based on patients' genetic profile. I used patients' genetic markers (SNPs) as my features and disease status as my ...
3
votes
1answer
662 views

Evaluation of final model in feature selection with nested cross-validation

I am doing feature selection with wrapper method on microarray datasets. I have read several papers and answers here about cross-validation (CV) evaluation on feature selection. Especially the answers ...
3
votes
0answers
102 views

Feature Selection in Mixture of Experts Model

I am just beginning to learn about Mixture of Experts Models, so I apologize if parts of my question are elementary. My understanding is that a MEM is a mixture model where the components are linear ...
3
votes
0answers
238 views

Variable selection with right-censored data?

I have the typical linear regression model: $$y_i = x_i^T\beta + e_i,$$ where $e_i\sim N(0,\sigma^2)$, iid. However, in my case, some (not all of them, only around 1/3 of them) response variables $...
3
votes
0answers
102 views

Is stagewised feature engineering/ selection an invalid approach? What to do when all the features are not ready at one time?

Suppose we want to build a regression or classification model. However, the features (independent variables used) are not all ready at one time. This is very realistic in business, because the data ...
3
votes
1answer
968 views

Is iterating LASSO a reasonable idea?

Can Lasso regression be performed multiple times to systematically clean/remove parameters from a model? Would there be downsides to doing so/would it be considered poor practice?
3
votes
0answers
482 views

Understanding the approach behind variable importance returned with Xgboost method in R package caret

I recently implemented the R package caret, for a binary categorical outcome regarding a transcriptomic microarray dataset. As i used the method from the xgboost package(method="xgbtree"), then i used ...
3
votes
0answers
62 views

Does this pattern indicate over-fitting in machine learning?

I am working on a diagnostics project, and trying to improve the performance of a classifier(s). We have over a million features to choose from, so feature selection is a real challenge. To look ...
3
votes
0answers
492 views

Important question regarding feature selection methodologies in R concerning the randomness of the results

I'm currently testing some feature selection methodologies/algorithms in R, like the Recursive Feature Elimination from the R caret package, and also the RRF R package, to select a subset of features ...
3
votes
0answers
590 views

Elastic net regularization - variables penalization

I have a data with ~ 3000 factor predictors with ~ 6 levels, many rows (300k+), and binary Y (trying to predict probability of event). There are many groups of variables that are highly correlated. I ...
3
votes
0answers
65 views

How can I estimate the influence/significance of the every observation on classification?

There are many ways to estimate the significance of the features on the classification model. But how I can estimate the influence of the every observation on the classification quality? My thinking ...

1
2 3 4 5
17