0
votes
0answers
13 views

Feature selection when bagging trees/random forest

I want to get a better understanding of feature selection and how the number of features affect performance when bagging trees. I am using Matlab's treebagger and I ...
0
votes
1answer
29 views

mtry tuning given by caret higher than the number of predictors

According to this discussion, it seems that the train function of the caret package returns a ...
0
votes
0answers
27 views

feature selection for longitudinal data

I have a longitudinal data which looks like this. Number of time points are different for each ID. Y is the binary response variable (take values 0 & 1) and X1-X20 are either continuous or ...
1
vote
0answers
11 views

Does within-group heterogeneity negatively impact random forest classification?

I have two rather conceptual questions about random forest classifiers. Before we get there, I quickly want to lay out the problem I am working on: I have large a large data set consisting of 300 ...
0
votes
0answers
19 views

Considering non-i.i.d. covariates in random forests

Random forests are theoretically funded on the assumption that the data are i.i.d. realizations from a multivariate random vector $(X_1, \ldots, X_p, Y)$. Does it make sense to use random forests (for ...
0
votes
0answers
22 views

How to deal with a feature relating to what type of expert labelled the data that becomes unavailable at point of classification?

Essentially I have a data set, that has a feature vector, and label indicating whether it is spam or non-spam. To get the labels for this data, 2 distinct types of expert were used each using ...
2
votes
4answers
98 views

Interpreting conflicting results from Random Forest & Logistic Regression?

I am using SKLearn and Statsmodel in python to build a RF and Logistic Regression, respectively. I have a feature that the RF indicates is important (feature importance of 0.202, closely behind #1 ...
0
votes
0answers
19 views

On population variable importance

Consider we run a random forest on $n$ independent realizations of a random vector $(X_1,X_2,X_3,Y)$ assuming $Y$ is a numerical response variable. Let $f$ be the best theoretical classifier defined ...
0
votes
0answers
12 views

Boruta score goes to minus infinity

I'm running the Boruta algorithm with a $179\times 36$ predictor matrix and a numerical response. Most of the variables have a score going to -Inf. Should I ...
0
votes
0answers
38 views

Dropping predictor variables, based on variable of importance, effect of Random Forest Accuracy

I am trying to use Random Forest to accurately predict forested land cover classes using Landsat 7, climatic and geographical data. I have 23 predictor variables and 1 response variable. When I drop ...
4
votes
2answers
130 views

Feature selection with partial permutation

For feature selection, permutation tests are biased in favor of those categorical variables with a large number of levels [White1994]. Besides, it has been proposed [Deng2011] that partial ...
3
votes
1answer
187 views

Gini decrease and Gini impurity of children nodes

I'm working on the Gini feature importance measure for random forest. Therefore, I need to calculate the Gini decrease in node impurity. Here is the way I do so, which leads to a conflict with the ...
1
vote
1answer
54 views

Does Boruta feature selection (in R) take into account the correlation between variables?

I am a bit of a novice in R and feature selection, and have tried the Boruta package to select (diminish) my number of variables (n= 40). I thought that this method also took into account the possible ...
3
votes
1answer
112 views

Finding interactions using randomForest

I am trying to use randomForest in R to find interaction terms to add to a model. My plan was to fit trees with maxnodes=4 (two ...
0
votes
0answers
63 views

Random Forests and Feature selection [duplicate]

First, I split my training set into 10 parts. 9 parts of it, I use as LS and the other one as TS. I now want to do feature selection, so I do feature selection on 9 parts. I use Random Forest to do ...
0
votes
1answer
85 views

Random forest importance differs between rf$importance and importance()

My model is working ok (the AUC is 0.7) but the importances from a randomForest run for my binary classification problem differ depending on how I retrieve them. Is ...
0
votes
2answers
912 views

Random Forest: IncNodePurity and Feature Selection for Binary Logistic Regression

After creating a Random Forest object using randomForest with around 500 candidate variables, I used importance(object) to ...
2
votes
1answer
198 views

Combining Exploratory Factor Analysis with Random Forest for Binary Logistic Regression Feature Selection

For those of you familiar with Exploratory Factor Analysis (EFA) and Random Forest (RF), I have recently had an idea of combining these two methods to reduce the number of potential predictor ...
7
votes
2answers
3k views

Feature selection with Random Forests

I have a dataset with mostly financial variables (120 features, 4k examples) which are mostly highly correlated and very noisy (technical indicators, for example) so I would like to select about max ...
3
votes
1answer
275 views

Using Mutual Information for Binary Logistic Regression Variable Selection

In addition to proc varclus, randomForest, and assessing multicollinearity among potential predictor variables, I am seeking ...
2
votes
0answers
66 views

Out-of-bag estimate biased by correlated features

I have a data set with a small number of samples (322) and a large number of features (318.976). My data consists of images, and I want to train a binary classifier. Since I have such a small amount ...
0
votes
1answer
388 views

What is a good Gini decrease cutoff for feature inclusion based upon random forests?

I am using random forests to try and determine variable importance as part of feature selection for a model I'm working on, and while I can get ranked variable importance by mean decrease in Gini from ...
3
votes
1answer
453 views

Can we use random forest for classification in combination with distance matrix between classes?

With a colleague, we are working on a dataset containing ~5000 continuous variables for 120 individuals belonging to 8 classes. We want to estimate the relative importance of each variable to explain ...
0
votes
1answer
163 views

Relative importance weight with cforest

I am new in using RF. I want to use it to compute the relative importance of the features. I found the weight is very small ("party" package, cforest). Is there anyway to get these weights in a range ...
3
votes
1answer
302 views

Variable importance randomForest negative values

I am asking myself if it is a good idea to remove those variables with a negative variable importance value ("%IncMSE") in a regression context. And if it gives me a better prediction? What do you ...
-1
votes
1answer
160 views

Random forest like procedure for regression or other statistical models

I'm wondering if there exist methods similar to one used in random forest algorithm - I mean taking simultaneously bootstrap sample and random subset of features, then building statistisal model. Have ...
1
vote
0answers
520 views

How to select the best variables by RandomForest in R?

I have a table of mRNA levels of my target gene and it's transcription factors in many different condition. What I want to do is to select the most important ...
1
vote
0answers
270 views

Feature importance

The extremely randomized trees classifier (scikitlearn) provides a (multivariate) feature importance measurement Ensemble methods/feature importance evaluation. For each feature, the classifier ...
1
vote
1answer
185 views

highly correlated features and high ranking

I am classifying different texts and I wondering about some features that are highly correlated. I have 49 features. Some features are absolute counters (integers) but most features are relative ...
4
votes
2answers
2k views

Number of trees for Random Forest optimization using recursive feature elimination

How many trees would you suggest to pick to perform recursive feature elimination (RFE) in order to optimize Random Forest classifier (for binary classification problem). My dataset is very ...
2
votes
0answers
95 views

Random forest like techniques (bagging, random feature subset) for SGD methods

Are there any well-known results/tools/literature on using bagging and random feature subset selection for regression or SGD-based methods?
7
votes
4answers
1k views

Low classification accuracy, what to do next?

So, I'm a newbie in ML field and I try to do some classification. My goal is to predict the outcome of a sport event. I've gathered some historical data and now try to train a classifier. I got around ...
6
votes
1answer
939 views

Feature selection and parameter tuning with caret for random forest

I have data with a few thousand features and I want to do recursive feature selection (RFE) to remove uninformative ones. I do this with caret and RFE. However, I started thinking, if I want to get ...
2
votes
1answer
335 views

Are randomForest variable importance values comparable across same variables on different dates?

Are randomForest variable importance comparable across same variables on different dates? I have a data array X which is of size $T\times N\times K$, where $T=1500$, $N=1500$ and $K=10$. ...
2
votes
0answers
182 views

Procedure for variable selection + logistic regression when n is small, p is large, and data are unbalanced?

I have data that have been collected using case-control procedures, in which the population of positive cases is collected with a random sample of negative cases. This yields 62 positive cases and 179 ...
8
votes
4answers
2k views

Is there a way to use cross validation to do variable/feature selection in R?

I have a data set with about 70 variables that I'd like to cut down. What I'm looking to do is use CV to find most useful variables in the following fashion. 1) Randomly select say 20 variables. ...