Questions tagged [feature-selection]

Methods and principles of selecting a subset of attributes for use in further modelling

Filter by
Sorted by
Tagged with
1 vote
0 answers
20 views

How many zeroes from lasso linear regression?

Given a dataset $X$ with $d$-dimensional features $x \in R^d$, and a response variable $y$ you can perform a lasso regression, ie linear regression with L1 regularization, as $$ \min_{\beta} (X\beta - ...
alexmolas's user avatar
  • 306
0 votes
0 answers
11 views

Feature selection on an explicit example

I'm trying to learn about feature selection and am having a terribly difficult time wrapping my head around how to understand the marginal benefit of adding an extra feature in the context of ...
redbull_nowings's user avatar
0 votes
1 answer
20 views

How to handle correlated variables before using Recursive Feature Elimination?

I have seen a few Kaggle notebooks that list without reason that RFE works better when removing correlated variables. I struggle to see the reason why so I conducted some of my own research and would ...
AvanishM's user avatar
0 votes
0 answers
15 views

Is the random forest classfier affected by related samples or biological replicates?

Correlation or collinearity between features can impact the results of random forest. So can having unbalanced data. However, I have not found a clear answer on whether having related samples can ...
Tal's user avatar
  • 11
1 vote
0 answers
64 views

Why does feature importance decrease for highly correlated variables?

I am investigating the relationship between correlation between features and its impact on their feature importances using sklearn's DecisionTreeClassifier algorithm. I manipulated the correlation of ...
AvanishM's user avatar
0 votes
0 answers
12 views

Model Significance for Svy logistic regression

I am working on the survey logistic regression. More than ten predictor variables are identified based on a P-value of < 0.25 on bivariate analysis. However, when I try multivariate analysis the ...
Melese Siyoum's user avatar
0 votes
1 answer
31 views

Feature selection using backward feature selection in scikit-learn and PCA

I have calculated the scores of all the columns in my dataframe, which has 312 columns and 650 rows, using PCA. I used the following code: ...
Mostafa Bouzari's user avatar
0 votes
1 answer
31 views

Is it necessary to remove redundant variables from a random forest classification model?

I am running a random forest model in the following variables(attached): Is it necessary with Random Forests Classifier to remove highly correlated variables or should I leave the model as is? If I ...
John Gallop's user avatar
0 votes
0 answers
37 views

Variable selection based on PLS

What is the logical way to select the variables from PLS? Does choosing the feature from loadings and loading weights make sense? Loadings... Loading weights... Regression coefficients... Variable ...
vdu16's user avatar
  • 13
0 votes
1 answer
50 views

Comparing two groups by the counts of their features

Imagine there are two different groups of individual samples. I know they are different but I don't know why. For example in biology a group of sick individuals and a group of healthy ones. Now for ...
lesolorzanov's user avatar
0 votes
0 answers
31 views

What are the best ways to perform feature selection for a binary classification problem with extremely imbalanced dataset

I have a classification problem where the size of the dataset is about 1 million lines but the target group is only about 0.6% of the dataset. I have about 40 feature including both categorical and ...
peiman razavi's user avatar
2 votes
0 answers
15 views

Transforming discrete optimisation problem into continuous optimisation problem

In Sparse Hilbert-Schmidt Independence Criterion Regression (Poignard and Yamada, AISTATS 2020), the authors consider a way to perform feature selection by taking the subset of features that maximises ...
LoveRKHS's user avatar
13 votes
4 answers
1k views

Why does Harrell Argue for "Ignoring Y during data reduction"?

In Regression Modeling Strategies page 79 (4.7 Data Reduction) reads: Data reduction is aimed at reducing the number of parameters to estimate in the model, without distorting statistical inference ...
purple-blade's user avatar
0 votes
1 answer
27 views

Using XGB or a Random Forest Model for Interaction Minion to later include in Logistic Regression Model

I am currently developing a logistic regression model, I already have some expert based interactuion terms I will implement but I want to further figure out possible interactions I can implement in ...
Habenzu's user avatar
  • 43
0 votes
0 answers
19 views

Measuring features effect and importance in Partial Least Square (PLS) regression

Context: it is possible to assess features importance and effect for a model using model-independent scoring techniques such as Partial Dependence (PD) profile, Acculumated Local Effect (ALE) profile, ...
Paul's user avatar
  • 101
5 votes
2 answers
178 views

How to deal with correlated variables

I would like to know how to deal with correlated variables, with this kind of correlation matrix: Is there a way to combine the correlated variables such as all the AV.. variables or FF.. variables? ...
Siueg's user avatar
  • 51
2 votes
1 answer
70 views

Why author discards useful column?

I am confused with this logical thinking: The quote is from the book “The art of ML” by Matloff He is working with the dataset https://www.kaggle.com/datasets/joniarroba/noshowappointments I agree ...
Chicago1988's user avatar
1 vote
0 answers
34 views

Handling Mixed-Frequency time series data for Feature Selection

I'm currently working on a project where I aim to apply LASSO regularization and conduct variable importance analysis on WTI crude oil prices. My challenge is dealing with datasets that have different ...
Dome's user avatar
  • 21
1 vote
0 answers
13 views

Feature importance in expectation maximization

The context is using EM algorithm for a mixture model - more precisely Dirichlet Multinomial Mixture, as discussed in Dirichlet Multinomial Mixtures: Generative Models for Microbial Metagenomics. One ...
Roger V.'s user avatar
  • 3,773
0 votes
0 answers
9 views

Addressing High Missingness in Data Analysis for Feature Selection Methods

I'm working with a comprehensive dataset of 12,000 companies and 600 ESG metrics from Refinitiv, segmented into 10 sectors. In my feature selection process for determining influential ESG scores, I've ...
Question Anxiety's user avatar
0 votes
0 answers
9 views

How to use RFE for RF and SVM

Considering I have a big data (lots of OTUs and clinical), which I will be using to input into RF and SVM for prediction (classification), will it make sense to perform RFE as a feature selection step?...
Tori's user avatar
  • 11
1 vote
0 answers
26 views

Feature selection before ML (RF and SVM)

I am new to machine learning and have to work with big data (lots of OTUs along with clinical) which I will input into 2 different machine learning models (RF and SVM) that will be used for prediction ...
Tori's user avatar
  • 11
2 votes
1 answer
29 views

Interpreting the formula for Riemannian metric tensor

In Improving support vector machine classifiers by modifying kernel functions, the authors defined Riemannian metric tensor for a kernel as follows: $$ \begin{align} g(\vec{x}) &= \text{det}|g_{ij}...
Omar Shehab's user avatar
2 votes
0 answers
43 views

Variable Selection for Longitudinal Data with a Binary Outomce

I have a large longitudinal dataset (100,000 observations) with firm IDs and Years with about 1000 features (most numeric and ...
thatsroughbuddy's user avatar
2 votes
0 answers
10 views

Is spiked tensor decomposition a special case of INDSCAL decomposition?

I understand that "Spiked" often refers to the presence of a dominant component (or a few dominant components) in a tensor decomposition. Spiked tensor decomposition is applied to multi-way ...
Omar Shehab's user avatar
1 vote
0 answers
40 views

Does anyone recognize this significance test between two regression betas?

I came across the following test for statistical significance between two betas (predictors) from a multiple regression model. Note that $R^2$ is the model coefficient of determination, $r$ is the ...
mvanaman's user avatar
  • 183
0 votes
0 answers
14 views

Random forest for feature selection over large inhomogeneous data set [duplicate]

I have a very large dataset (500,000 examples, 3000 features with a lot of missing values). I want to run a random forest algorithm for feature selection with sklearn. Unfortunately, I cannot load the ...
Constantinus Spanakis's user avatar
0 votes
0 answers
37 views

Fourier coeficients as feature to classification model

I have to solve a timeseries multiclass classification problem. Some of these classes (Class 0, Class 1 and Class 4) have timeseries that are really similar to each other, as follows: ...
Murilo's user avatar
  • 139
1 vote
0 answers
127 views

What reference is best to convince someone that variable selection on the same data you use for inference is dangerous? [duplicate]

I need to convince colleagues that variable selection on the same data you use for inference is a bad idea. I know of some general references on the problems with model selection, listed below -- but ...
Mohan's user avatar
  • 865
7 votes
2 answers
276 views

Intuition and reasoning why LASSO can only select $n$ features when $n \ll p$

I'm struggling to grasp the intuition behind why LASSO can only select at most $n$ features when $n << p$, where $n$ is the number of samples and $p$ is the number of features. I've read through ...
Jacob Bumgarner's user avatar
3 votes
1 answer
56 views

Selecting variables using lasso algorithm

I have a question concerning a large dataset with 94 observation and 15000 variables. For data mining models (boosting, trees, neural networks...) this number of variables are too much and I have to ...
ali's user avatar
  • 61
0 votes
0 answers
5 views

Faster loss convergence in the beginning leads to worse local minimum?

When doing feature selections, I've noticed that sometimes the same model converges faster with feature set A, than with feature set B, in the very beginning, but ends up with a worse loss later. Here'...
autoencoder's user avatar
0 votes
0 answers
25 views

Sklearn feature selection performs strangely with 2 groups (and with SVC)

Previously I've successfully performed support vector classification with recursive feature elimination in R using the e1071 package, but I'm now hoping to move over to SciKit Learn given that Python ...
Benjamin Taylor's user avatar
0 votes
0 answers
11 views

Should unprocessed features be kept in the dataset along engineered ones?

I'm working on a Machine Learning classification problem that has five Service Spending features (among many others) in its dataset, each sample is a customer. For instance, here are the first three ...
spengler's user avatar
0 votes
1 answer
69 views

Appropriate feature selection for classification

in univariate feature ranking for classification, it is common to use the χ²-test(MATLAB-sklearn) to calculate importance scores based on the negative log of the associated p-value: $Importance = -\ln(...
Tino D's user avatar
  • 113
1 vote
0 answers
21 views

Automated Code for Logistic Regression [closed]

My Y variable (output) is binary (0 or 1). I have 10 input variables in total, 3 of them are scaled variable, 2 of them are ordinal number therefore being written with C( ). Rather than running the ...
sct 22n's user avatar
  • 11
0 votes
0 answers
28 views

How do i prioritize which features to use in my machine learning model before the feature engineering stage?

I am encountering a probably fairly common problem where I have too many features, lets say 500 possible features. I only want to pick the top 10-50 features that would be the most predictive of y, or ...
Katsu's user avatar
  • 861
1 vote
0 answers
15 views

How to deal with different orders of integration between explained and explaining variables?

Is there a standard, or at least a valid, regression approach if you are trying to regress a dependent variable with a unit root against a set of stationary independent variables? I know I could ...
andrewH's user avatar
  • 3,097
1 vote
1 answer
64 views

Lasso for feature selection in classification models

I want to perform classification of breast cancer cases by using models like SVM or Random Forest. When I was browsing the web I saw that one could use Lasso for feature selection, and then applied ...
Layla's user avatar
  • 621
1 vote
0 answers
26 views

Variable Selection for GEE or other Longitudinal Methods When Dealing with Many Variables?

I see the question of model selection for GEE come up and answers seem to usually involve QIC/MLIC for non-nested models, or LRT for nested. That's all fine, but when I have 50 predictors it's a bit ...
purple-blade's user avatar
0 votes
0 answers
7 views

Should main effects and interaction terms be included in Kmeans clustering? (hierarchical principle in clustering analysis)

Let's say I'm trying to cluster observations based on five features, including: n_emps: Number of employees n_cust: Number of ...
Arturo Sbr's user avatar
0 votes
0 answers
25 views

BorutaPy selects different features in different iterations

I came up to something strange. I know Boruta should select everything that is important. I have a dataframe of 200 observations and 2000 features. if I shuffle the order of the features in the ...
Noob Programmer's user avatar
2 votes
1 answer
43 views

What is the Mutual Information for one instance?

I'm computing mutual information for several features where one of my datasets has one instance. One instance is because of a specific filtering criterion I used. I'm using ...
Sachz's user avatar
  • 121
0 votes
0 answers
20 views

Nested Cross Validation performance on each Sequential Feature Selection subset

I want to get cross-validated performance values of my model after hyperparameter tuning and sequential feature selection on each feature subset. Following this example, I want to use an outer-CV ...
Charlie's user avatar
  • 132
2 votes
1 answer
46 views

Model Building Process Without Feature Names/Domain Knowledge

I am working with a dataset of $\approx 10^7$ samples and $\approx 120$ features. All samples have a binary classification $0 - 1$. I am attempting to build a model that minimizes out-of-sample error ...
Bepop's user avatar
  • 197
0 votes
0 answers
26 views

Linear Regression with Learned Feature Selection?

I have a question related to linear regression that stems from my research. Suppose I have two real matrices: Design matrix $X \in \mathbb{R}^{N \times D}$ Target matrix $Y \in \mathbb{R}^{N \times E}...
Rylan Schaeffer's user avatar
1 vote
0 answers
17 views

Need help to validate if my understanding is correct regarding random forest and feature importance

I'm currently doing binary classification task with about 40 features. The main goal of this task is finding important features. I built few tree based and binary classification models and one of my ...
dspeanut's user avatar
4 votes
2 answers
119 views

Variable selection with a theoretical DAG vs algorithmically discovered DAG

I'm analysing data from an electronic health record and determining what variables to include in a model to close back doors and omit bias. I've read that it is important to have a subject specific ...
Geoff's user avatar
  • 591
1 vote
0 answers
34 views

Proof of Variable Selection Consistency for LASSO in Zhao & Yu, 2006

I'm going through the proof of Proposition 1 in Zhao & Yu, 2006 (https://www.jmlr.org/papers/volume7/zhao06a/zhao06a.pdf), titled On Model Selection Consistency of LASSO. The proof is in Appendix ...
gchandra97's user avatar
0 votes
0 answers
11 views

Feature Selection before Hyper-parameter Tuning and Error Estimation

I have a huge pool of features for a classification task. For error estimation of my models, I am using Nested Cross Validation (Nested CV), where I have an inner loop of CV for hyper-parameter tuning ...
Mehdi Ejtehadi's user avatar

1
2 3 4 5
48