Tag Info

2

You lost information by binning, see Why should binning be avoided at all costs?. For an alternative, use logistic regression and spline the day of contact variable. For details see Logistic Regression with regression splines in R or Using splines to address non-linearity in logistic regression

0

I am thinking about stability selection more like a statistical procedure to give you a set of significant variables in a high dimensional setting, where traditional linear regression would fail. That it uses LASSO in the paper, which is usually used for machine learning purposes is kind of a red herring. Now, as you correctly noticed, it cannot be easily ...

2

A way to gauge, how useful a predictor $x_j$ is within a given model $M$ is by comparing the performance of the model $M$ with and without a predictor $x_j$ being included (say model $M^{-x_j}$). If we have multiple predictors though we are face with a situation we would have to create $p$ different $M^{-x_j}$ models going back and forth. The cost of this re-...

2

Tree models' measures of feature importance have been called into question in general. But also, xgboost's python implementation get_score defaults to "weight", which measures the number of splits a feature makes. This obviously hurts small-cardinality features like sex (which should be highly predictive in titanic): it can only be used to split ...

3

Not necessarily. Correlation measures the strength of a linear relationship. Age appears to have a weak correlation but, the relationship between age and the outcome may not be linear. See the wikipedia entry for correlation for some examples in which x and y are related but the correlation is 0. I'm not a big fan of correlation. Feature importance via ...

0

I would determine the ROC-AUC for all possible class comparisons using each feature as a separate input. Below is an ROC-AUC plot of AUC curves for all possible class comparsons for a 4-class problem, but it's based on a run with multiple features.

1

Whether or not the importances of gradient boosting are strongly influenced by randomness depends on the hyper-parameter configuration of the gradient boosting model. A gradient boosted model which uses random subsampling of features (or other randomized components) will estimate feature importances which vary to a greater or lesser degree upon repeated ...

0

The statement: For all other categories, the corresponding beta value will be representative of the feature importance isn't quite right. With dummy coding, the intercept term represents the outcome (log-odds in logistic regression; principle holds for all regression models) when the categorical predictor is at its reference level. The category-specific ...

1

Also, in XGBoost the default measure of feature importance is average gain whereas it's total gain in sklearn. See, https://xgboost.readthedocs.io/en/latest/python/python_api.html#module-xgboost.sklearn importance_type (string, default "gain") – The feature importance type for the feature_importances_ property: either “gain”, “weight”, “cover”, “...

Top 50 recent answers are included