What happens when there is a linear regression of the kind

y ~ x1 + x2 + x3 + x4

What determines which variables will be granted the explained variance rather than the others given a correlation matrix (not assuming independence)?

In case this really is unpredictable, why do we insist on using regression rather than simply looking at the correlation matrix: at least then you know you are not accounting for effects of other variables when looking at the correlation coefficient, while using regression you'd be oblivious to which variable gets the explained variance (and how correctly or wrong this is!).

  • $\begingroup$ How it will come out is completely 'predictable' in the sense I think you intend it - no worries there. $\endgroup$ – Glen_b Jul 14 '13 at 0:08

There are several properties of regression that are directly relevant to questions of the relative "importance" or "impact" of predictors but are widely misunderstood:

  1. First and foremost, all the variables that truly matter must be present in the regression equation. If any important variables are omitted then the results can be misleading unless all the omitted variables are uncorrelated with all the included variables. There is no point in attempting to discover the relative importance of some predictors for which you have data unless you already know that these are the only predictors that matter.

  2. R^2 can be partitioned into components representing the unique contribution of each predictor only when all the predictors are mutually uncorrelated. The problem is not partitioning R^2 -- that can always be done. The problem is that the results do not always represent the unique contribution of each predictor. However intuitively straightforward the notion of unique contributions may seem, there is no mathematical definition that is entirely satisfactory when the predictors are correlated.

  3. Importance ratings obtained by comparing semipartial correlations or changes in R^2 (i.e., squared semipartials) depend on the joint distribution of the predictors. Contrary to what is often implicit in the importance question, the results are not inherent properties of the variables alone, but joint properties of the variables and the particular multivariate distribution they happen to have. This is especially important when the distribution of the predictors is an artifact -- either a direct artifact, because the investigator set the values of the predictors; or an indirect artifact, because the investigator selected cases or sampled nonrandomly. And even if the sample distribution is a valid estimate of some "natural" population distribution, if the population distribution changes then the true semipartials can also change, even though the mechanism relating the predictors to the outcome variable has not changed.

  4. If the predictors are in the same units (possibly after data-independent unit-equating transformations, which excludes sample-specific standardization), then comparing the raw-score regression weights can lead to conclusions of relative importance that are inherent properties of the variables alone. However, the definition of importance that is implicit in comparisons of the regression weights is the expected change in the outcome variable for a unit change in the predictor in question, with all other predictors held constant. In some situations this definition may be appropriate; in others, not.

So where does this leave us? It may often mean that questions about importance can not be answered -- at least not in the sense that they were asked. Unfortunately, regression seems to have been "sold" to many as a way to answer all such questions. It can't.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.