3
$\begingroup$

There's many questions on related topics but I have been unable to find one that precisely answers my question.

Let's say I'm performing a regression on multiple predictor variables $x_1...x_n$ for explanatory purposes. My intent is that the size of the coefficient determines the importance of that variable in the outcome $y$. I want to be able to say something like "This coefficient $b_2$ is the largest, therefore this variable $x_2$ has the biggest effect on $y$". I do not care about the intercept.

My $x$ variables are of different types: Categorical binary (either 0 or 1), continuous on [0,1] and continuous over $(-\infty,\infty)$ and I am not sure how much normalisation or standardisation is necessary to make the coefficients of these variables comparable to each other in the model.

Below is a list of statements in decreasing order of how sure I am they are true.

It is valid to compare the coefficients of:

  • The categorical variables with each other
  • The continuous variables on [0,1] with each other
  • The continuous variables on [0,1] with the categorical variables
  • The continuous variables on $(-\infty,\infty)$ with each other, providing they are standardised.
  • The standardised continuous variables on $(-\infty,\infty)$ with all the other variables

My questions are:

1. Are the above statements true?, and

2. If they are not true, what transformation is necessary (if any exists) to make them true?

$\endgroup$
  • $\begingroup$ There are many possible ways to compute some "importance" measure for a predictor in multiple regression. Search for multiple regression importance. One popular easy measure is the relative size of the squared part correlation of the predictor with the Y. For each predictor i in the model, compute importance I_i = SSE_without_i - SSE_full_model. Then normalize by dividing by the sum of the I's of all the predictors. $\endgroup$ – ttnphns Aug 28 '18 at 10:35
  • $\begingroup$ Check also this paper by Andrew Gelman: Scaling regression inputs by dividing by two standard deviations. $\endgroup$ – Dimitris Rizopoulos Sep 3 '18 at 13:39
1
+50
$\begingroup$

Standardizing continuous predictor by subtracting their mean and dividing by their standard deviation is a procedure that is often used to put all predictors on a common scale. In that sense, the magnitude of the estimated coefficients could be compared to see which predictors are associated with the largest change in your outcome when all of them are increased by the magnitude of one standard deviation.

However, there are some important issues associated when doing this standardization. Some of them have been highlighted above. An additional issue is that such standardization is data specific. I.e., if someone else did in another dataset, he would standardize the predictors differently because in his sample the mean and standard deviation of the predictor would be different than in your sample.

For more on this, check also this paper by Andrew Gelman: Scaling regression inputs by dividing by two standard deviations.

$\endgroup$
1
$\begingroup$

The reason why you're not finding answers to these questions is definitions: neither importance nor effect of a variable are defined statistical terms. Regression coefficients measure the effect of some unit of change in a variable, and scaling transformations simply change that unit. What people refer to as "importance" is mostly determined by the practical, moral and philosophical issues surrounding the problem, and not only by the regression coefficients.

For a practical example, let's say $y$ is some clinical outcome of interest (disease risk or continuous measure), and you ran a regression on potentially causal factors $x_i$. Here's some properties of $x$ that could be of interest to you, but will not be reflected in the $\beta$s:

1) Is a variable "important" if it cannot be changed? Height (in adults) has an effect on various disease risks, but short of eugenics we have no way to change it in well-nourished populations.

2) Is a variable "important" if it is not causal for $y$, yet strongly associated with it? For prediction purposes, it might, for explanatory - no.

3) Is a category "important" if it has a large effect (relative to reference), yet is very rare? Developing drugs against rare conditions could be very costly.

4) Is a variable "important" if we already know about its effects and have policies targeting it (e.g. smoking)?

...

Standardised coefficients and $R^2$ are useful because they provide a meaningful interpretation as "variance of $y$ explained by $x$". This partially takes care of issue 3), as rare categories or low-variation continuous features do not offer much predictive power. The drawback is that such measures depend on the distribution of predictors, and thus will not be comparable across populations or experimental settings.
As for the other issues, one can convert anything into a single "importance" objective, but that will depend on the particular user needs (and morals, virtues and everything else), so don't expect any universal statistical answer.

$\endgroup$
  • $\begingroup$ I'm not a formally trained statistician, so I'm not always familiar with the best words to use, though I had hoped that this question was clear enough. By importance, I mean (and this is non-technical language) the strength of association of the outcome variable y with the variable $x_n$. I don't really know what else to call it. "The influence of $x_n$ on y", "The biggest predictor of y". $\endgroup$ – Ingolifs Aug 28 '18 at 8:35
  • $\begingroup$ So let's say $x_1$ is binary, $x_2$ - continuous. All I'm saying is: it's certainly valid to compare raw $\beta_1$ and $\beta_2$, as long as you understand that $\beta_1$ reflects a change of category 0 to 1 and $\beta_2$ - a change of one unit in $x_2$. Any further interpretation will have to take into account how easy / relevant / ethical is to enact these changes in $x_1$ and $x_2$. If "importance" is limited to prediction in an identical population, then measures such as standardised coefficients, $R^2$ or ttnphns's suggestion above are fine. $\endgroup$ – juod Aug 28 '18 at 23:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.