14

In addition to John's answer, you may wish to obtain the squared semi-partial correlations for each predictor. Uncorrelated predictors: If the predictors are orthogonal (i.e., uncorrelated), then the squared semi-partial correlations will be the same as the squared zero-order correlations. Correlated predictors: If the predictors are correlated, then the ...


14

The first thing to note is that you don't use logistic regression as a classifier. The fact that $Y$ is binary has absolutely nothing to do with using this maximum likelihood method to actually classify observations. Once you get past that, concentrate on the gold standard information measure which is a by-product of maximum likelihood: the likelihood ...


13

No, this isn't a bug. The values given in fit$importance are unscaled, while the values given by importance(fit) are expressed in terms of standard deviations (as given by fit$importanceSD). This is usually a more meaningful measure. If you want the "raw" values, you can use importance(fit, scale=FALSE). In general, it's a very bad idea to rely on the ...


10

Random Forest importance metrics as implemented in the randomForest package in R have quirks in that correlated predictors get low importance values. http://bioinformatics.oxfordjournals.org/content/early/2010/04/12/bioinformatics.btq134.full.pdf I have a modified implementation of random forests out on CRAN which implements their approach of estimating ...


9

The part of the overall random forest algorithm that uses mtry is (adapted from The Elements of Statistical Learning): At each terminal node that is larger than minimal size, 1) Select mtry variables at random from the $p$ regressor variables, 2) From these mtry variables, pick the best variable and split point, 3) Split the node into two daughter nodes ...


9

Standardized regression coefficients do not work for categorical variables or for nonlinear effects. You are assuming everything has a linear effect, which is unlikely. Standardization also assumes that the SD is the right scaling constant. To me standardized coefficients are harder to interpret than the original coefficients, and the standardization is ...


9

AUROC ($c$-index; concordance probability, Somers' $D_{xy}$ rank correlation) is not a valid objective for optimization. It is fooled by a terribly miscalibrated model and is inefficient. Maximum likelihood estimation exists for a reason: optimizing the log likelihood function results in optimality properties of the estimators. And don't scale indicator ...


8

Run this code and assert that RF variable importance do incorporate interactions. library(randomForest) obs=1000 vars =4 X = data.frame(replicate(vars,rnorm(obs))) ysignal = with(X,sign(X1*X2)) ynoise = 0.1 * rnorm(obs) y = ysignal + ynoise RF = randomForest(X,y,importance=T) varImpPlot(RF) You should see X1 and X2 are found the important and X3 and X4 ...


8

One way to quantify the usefulness of each feature (= variable = dimension), from the book Burns, Robert P., and Richard Burns. Business research methods and statistics using SPSS. Sage, 2008. (mirror), usefulness being defined by the features' discriminative power to tell clusters apart. We usually examine the means for each cluster on each dimension ...


8

For the standard linear regression model the absolute value of the coefficient estimates and the p-value are not related in the way you describe. It is very possible to have absolutely large coefficients which are insignificant and absolutely small coefficients which are very significant. What your missing in your interpretation is the effect of the ...


8

This is completely anecdotal, but I've found variable importance useful in identifying mistakes or weaknesses in GBMs. Variable importance gives you a kind of huge cross-sectional overview of the model that would be hard to get otherwise. Variables higher in the list are seeing more activity (whether or not they are more 'important' is another question). ...


8

I have argued that variable importance is a slippery concept, as this question posits. The tautological first type of response that you get to your question and the unrealistic hopes of those who would interpret variable-importance results in terms of causality, as noted by @DexGroves, need little elaboration. In fairness to those who would use backward ...


8

When working on "feature importance" generally it is helpful to remember that in most cases a regularisation approach is often a good alternative. It will automatically "select the most important features" for the problem at hand. Now, if we do not want to follow the notion for regularisation (usually within the context of regression), random forest ...


7

To get the coefficient in a space that lets you directly compare their importance, you have to standardize them. I wrote a note on Thinklab to discuss standardization of logistic regression coefficients. (Very) Long story short, I advise to use the Agresti method: # if X is the input matrix of the glmnet function, # and cv.result is your glmnet object: sds ...


6

The short answer is that is that there isn't a single, "right" way to answer this question. For the best review of the issues see Ulrike Groemping's papers, e.g., Estimators of Relative Importance in Linear Regression Based on Variance Decomposition. The options she discusses range from simple heuristics to sophisticated, CPU intensive, multivariate ...


6

It does not use a separate training and testing set. Instead, standard accuracy estimation in random forests takes advantage of an important feature: bagging, or bootstrap aggregation. To construct a random forest, a large number of data subsets are generated by sampling with replacement from the full dataset. A separate decision tree is fit to each ...


6

Linear Regression are already highly interpretable models. I recommend you to read the respective chapter in the Book: Interpretable Machine Learning (avaiable here). In addition you could use a model-agnostic approach like the permutation feature importance (see chapter 5.5 in the IML Book). The idea was original introduced by Leo Breiman (2001) for random ...


5

Variable importance in Random forest is calculated as follows: Initially, MSE of the model is calculated with the original variables Then, the values of a single column are permuted and the MSE is calculated again. For example, If a column (Col1) takes the values 1,2,3,4, and a random permutation of the values results in 4,3,1,2. This results in an MSE1. ...


5

You can just get the two separate correlations and square them or run two separate models and get the R^2. They will only sum up if the predictors are orthogonal.


5

For linear models you can use the absolute value of the t-statistics for each model parameter. Also, you can use something like a random forrest and get a very nice list of feature importances. If you are using R check out (http://caret.r-forge.r-project.org/varimp.html), if you are using python check out (http://scikit-learn.org/stable/auto_examples/...


5

The variable importance obtained by permutations is computed only by permuting values for a single variable. Thus, it computes some importance measure of the given variable in the context that all other data is fixed. I think it is reasonable to state that the importance measure includes in the measurement also interactions, if such interactions exists. I ...


5

In short, yes, you can get some measure of variable importances for RNN based models. I won't iterate through all of the listed suggestions in the question, but I will walk through an example of sensitivity analysis in depth. The data The input data for my RNN will be composed of a time-series with three features, $x_1$, $x_2$, $x_3$. Each feature will be ...


5

Boruta and random forrest differences Boruta algorithm uses randomization on top of results obtained from variable importance obtained from random forest to determine the truly important and statistically valid results. For details of the difference please refer to Section 2 of the article: Kursa, Miron B., and Witold R. Rudnicki. "Feature selection with ...


5

I assume all predictors have been standardized (thus, centered and scaled by the sample standard deviations). Let $\mathbf{x}$ be the vector of predictors and $y$ the response, conditionally Bernoulli-distributed wrt $\mathbf{x}$. Then if $\mu=\mathbb{E[y|\mathbf{x}]}=p(y=1|\mathbf{x})$, then clearly $$\frac{\partial \mu}{\partial x_i}=\beta_i \frac{\exp{...


4

I can think of two other possibilities that focus more on which variables are important to which clusters. Multi-class classification. Consider the objects that belong to cluster x members of the same class (e.g., class 1) and the objects that belong to other clusters members of a second class (e.g., class 2). Train a classifier to predict class membership (...


4

Given that you have panel data, the "standard approach" would be to use a fixed effects regression in order to eliminate time-invariant unobserved candidate and state characteristics. The earliest example I can provide you with is Levitt (1994). He wanted to know how campaign spending by incumbent candidates and challengers affects the election outcome. If ...


4

Since you were specifically asking for an interpretation on the probability scale: In a logistic regression, the estimated probability of success is given by $\hat{\pi}(\mathbf{x})=\frac{exp(\beta_0+ \mathbf{\beta x})}{1+exp(\beta_0+ \mathbf{\beta x})}$ With $\beta_0$ the intercept, $\mathbf{\beta}$ a coefficient vector and $\mathbf{x}$ your observed ...


4

Your question seems to reflect the mistaken understanding that the statistical "significance" of the p-value somehow means "meaningful", "important", or "relevant to real life". This is a false but very widely held misunderstanding. P-values are a standardized representation of how reliable the effect size measures are. I say that the p-value is "...


4

Variable importance rankings have a definite role in the applied business world whenever there is a need to prioritize the potentially large number of inputs to a process, any process. This information provides direction in terms of a focused strategy for attacking a problem, working down from most to least important, e.g., process cost reduction, given that ...


Only top voted, non community-wiki answers of a minimum length are eligible