3
The question you may ask first is what defines "important feature".
Random forest is a supervised learning algorithm, you need to specify a label first then the algorithm will tell you which feature is more important respect the given label. In other words, specifying different label will have different results for variable importance.
Without using the ...
2
Let us to use $X$ to represent the feature and $Y$ to represent the label. Essentially, if $P(Y|X)=P(Y)$ or $X$ and $Y$ are independent, we can drop $X$.
What you described
feature has zero variance only for cases that belong to one class but not in the other
Just tells this feature is an important feature that can differentiate different classes, i.e....
2
Standard Feature Importances simply tell you which ones of your features were more useful when building the model. They are not to be interpreted as a direct dependence between predictor and target.
As a consequence:
They are completely useless if your model is weak. If your model does not generalize to validation data - like in the case you mentioned of ...
2
Quoting Guyon in the paper that introduced RFE:
This [RFE] iterative procedure is an instance of backward feature elimination (Kohavi, 2000 and
references therein)
Indeed, when introducing RFE, Guyon does so using Support Vector Machines, and proposes two different methods to rank the single predictors.
At the same time Kohavi tests backward ...
1
My questions is this right approach to do feature selection when data volume is high?
Simply, no.
Basing feature selection on p values is a bad idea, especially when data are large. First, p-values tell you nothing about the effect of the variable. I can always construct a model with a highly significant feature but which performs negligibly different with ...
1
The technique you used is called Best Subset Selection.
I would say that the most popular technique to reach the same objective is LASSO.
See these techniques and other friends here.
You may also select features considering the importance of the features for out of sample prediction. In this context, I suggest two very interesting and general methods that ...
1
For tree based model, it can automatically handle redundant features, i.e. less useful features will not be selected as a split point. So you do not need to manually handle feature selection problems.
In many implementations of random forest or tree based boosting, the algorithm will automatically select a subset of features to build each tree. Therefore, ...
1
Is there a correct way / order to do [two kind of hyperparameter optimization]?
Yes: unless you know for sure that the different hyperparameters do not interact, they should to be optimized together.
Here, they do interact => optimize together
You can also optimize sequentially, but that should then become an iterative procedure:
optimize one type of ...
answered Feb 23 at 10:18
cbeleites supports Monica
29k2 gold badges60 silver badges117 bronze badges
1
Use ColumnTransformer, applying SelectKBest only to the continuous variables. You can apply some other feature selection/transformation to the categorical variables, or set remainder=passthrough to send the categorical features through untouched (aside from the order of the columns).
To the broader question, some feature selection methods should be ...
1
Yes, it is valid to apply different penalties to different coefficients. From the documentation for glmnet:
penalty.factor: Separate penalty factors can be applied to each
coefficient. This is a number that multiplies ‘lambda’ to
allow differential shrinkage. Can be 0 for some variables,
which implies no shrinkage, and ...
1
Even if that feature have zero variance for one of the classes (or even if it has zero variance for both classes!), it could still have very different values between the classes, so be a good discriminator. You should probably keep it!
1
What you apparently want to do is to start by evaluating the relationship between the 3 classes and each of your features individually. For each continuous feature you are proposing a one-way ANOVA of that feature against the known classification (3 classes). ANOVA is not appropriate for a similar evaluation of the binary features, as interpretation of ANOVA ...
1
I'd say that the default should be to treat preprocessing as part of model training, i.e. do this inside the cross validation loop.
You can save computation in some cases by "pulling" the transformation before the cross validation. This is allowed as long as the transformation does not violate statistical independence. So transformations that involve each ...
answered Feb 13 at 15:21
cbeleites supports Monica
29k2 gold badges60 silver badges117 bronze badges
1
There is nothing specifically about one-hot encoding that makes ANOVA inappropriate.
Most software (certainly R and SAS) does some version of coding of categorical variables for you. With R and SAS (and maybe others) you can choose among various parameterizations for categorical variables (effect coding, dummy coding, Helmert contrasts etc.)
answered Feb 8 at 14:28
1
We find a very broad definition of "adversarial examples" in Justin Gilmer, Ryan P. Adams, Ian Goodfellow, David Andersen, George E. Dahl "Motivating the Rules of the Game for Adversarial Example Research"
Of particular recent interest has been the investigation of errors arising from maliciously crafted inputs, or “adversarial examples”. Although there ...
answered Jan 29 at 2:57
Sycorax says Reinstate Monica
53.6k16 gold badges130 silver badges236 bronze badges
1
Partial dependence doesn't tell you the full story. You are looking at the marginal effect of one predictor on the response variable. It's just like the marginal distribution doesn't tell you the full story of the joint distribution. The graphs that motivate individual conditional expectation here (https://blogs.sas.com/content/subconsciousmusings/2018/06/12/...
Only top voted, non community-wiki answers of a minimum length are eligible
