2
$\begingroup$

Suppose that we want to predict $y$ using a subset of the variables $x_1 \dots x_n$ using linear regression. Suppose I regress $y$ on $x_1$, from which I obtain fit $f_1$, regress $y$ on $x_2$, from which I obtain $f_2$ and then regress $y$ on $x_3$, from which I obtain $f_3$.

Facts:

The slope of $f_1$ is almost $0$ (that is, the best fit line is almost constant). The slope of $f_3$ is greater in magnitude than the slope of $f_2$.

Questions:

1) Can I claim that $x_1$ is not an important predictor?

2) Can I claim that $x_3$ is a more important predictor than $f_2$?

I believe the answer to 2) is "no." Simply fabricate a relationship $y \sim 0.00001(x_1)$, create a variable $x_2 \sim N(0,10)$ and then $y \sim x_2$ will have a greater slope than $y \sim x_1$, even though $x_2$ is clearly unimportant.


Concrete Examples

Here are some concrete examples of an output variable (yield_rainfed_ana) regressed on three different, input variables. The best fit lines associated with these 3, following examples mimic, respectively, the lines $f_1,f_2,f_3$ that I described earlier.


enter image description here

enter image description here

enter image description here

$\endgroup$
1
  • 1
    $\begingroup$ Did you tried loess or similar piecewise predictions first to see if the dependency is linear? Also, with so many data point it is recommended to plot them as circles, not dots, and with very thin borders - to try to see what's going on in the middle of the cloud. $\endgroup$
    – ttnphns
    Commented Jul 22, 2018 at 8:09

1 Answer 1

2
$\begingroup$

This page does a good job of explaining variable importance:

http://statisticsbyjim.com/regression/identifying-important-independent-variables/

In summary, you first need to define what variable importance means to you, and decide between a purely statistical importance definition and a definition that contains something about your particular domain that makes a variable important in another way.

If you go the route of comparing coefficients, make sure to standardize your variables to put them all on a level playing field.

I think comparing the change in r-squared when you add a variable to a model last is a reasonable way of deciding on the importance of a given variable. Essentially, you’re looking at how much explanation of the target variable’s variation comes from adding the given predictor this way.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.