3
$\begingroup$

I have 3 independent variables A, B, C and want to run a multiple linear regression to predict Y. After studying the correlations between:

1) A, Y
2) B, Y
3) A/B, Y
4) (A-B)/(A+B), Y

It turns out that 4) has the highest correlation than all other cases by at least > 0.10. Both 3) and 4) make sense to me as variables A and B are complementary: that is, they represent the items bought in store A versus store B and there are only 2 stores in this problem.

Now, in a simple linear regression I have little doubt that the higher the correlation of the single independent variable the better the fit. But in a multiple linear regression, does it make sense to select the independent variables by looking at the formulas or ratios that shows the highest correlations against Y? In this case using 4) in the regression over 1), 2) or 3) because it has the highest correlation.

Improve this question
$\endgroup$
5
  • $\begingroup$ I'm no sure that I understand what you mean by refining. I'm also a little confused of how to interpret $\frac{A-B}{A+B}$. Perhaps having standard deviation at the bottom would be more intuitive (although with only two stores it probably will be the same)... $\endgroup$
    – Max Gordon
    Dec 6, 2011 at 7:00
  • $\begingroup$ Do you really want just to predict Y, or do you want to explain the mechanism by which one or more variables explain Y? If the former, then there's no reason not to use 4). If the latter, you'll have some work to do to make it clear to your audience just how 4) is helpful. It's not always the case that the most useful solution is the one with the highest r-squared. $\endgroup$
    – rolando2
    Dec 6, 2011 at 13:05
  • $\begingroup$ I want to get the best r-squared. All four cases are sound, but I want to understand if the highest correlated variables produce the highest r-squared in a multiple regression, or if, more in general, analyzing each independent variable correlation to Y is a good way to prepare the best fit model. $\endgroup$
    – Robert Kubrick
    Dec 6, 2011 at 13:57
  • $\begingroup$ Also I am aware that there are ways to select/discard the independent based on anova and the t-test, after the regression is run. I would like to understand if the general idea of increasing the correlation against Y like in my example is a sound method to increase r-squared. $\endgroup$
    – Robert Kubrick
    Dec 6, 2011 at 15:37
  • 1
    $\begingroup$ Robert, R-squared is meaningless when you are re-expressing variables. For some discussion of R-squared, see stats.stackexchange.com/questions/13314/…. You should prefer to obtain a model which has a good fit overall and acceptably small residuals. $\endgroup$
    – whuber ♦
    Dec 6, 2011 at 20:22

1 Answer 1

Reset to default
6
$\begingroup$

Although manipulating variables to maximize correlation has an intuitive appeal, it is not usually a good approach, for several reasons:

  1. In multiple regression the individual correlations can be low between independent and dependent variables, yet the least-correlated IVs can be the most important predictors of the DVs. See this thread for an example and this one for a theoretical discussion. This suggests that looking at the individual (bivariate) correlations can be useless or misleading.

  2. You can (accidentally or arbitrarily) make the correlation arbitrarily close to $\pm 1$ by means of a transformation of the dependent values that creates a single extreme outlier. In the following example $A$ (blue) and $B$ (red) are normally distributed--but always positive--and $C=A+B$ plus normally distributed error, except for a single outlying value at $(A,B,C)=(1, 1/16,20)$. Despite this strong relationship between $C$ and untransformed values of $A$ and $B$, the correlation of $A$ and $C$ is -0.2 (notice the sign is wrong!), the correlation of $B$ and $C$ is -0.25 (wrong sign again), and the correlation of $(A-B)/(A+B)$ and $C$ is +.45 (much stronger than either of the other correlations).

    A and B vs. C

    (A-B)/(A+B) vs. C

  3. Typically, one re-expresses independent variables in order to establish a more linear relationship with the dependent variable. You can test that visually if you like, making sure to discount high-leverage outlying values that might appear with some re-expressions.

Improve this answer
$\endgroup$
4
  • $\begingroup$ I gather that although correlation in itself is not a good approach to filter DVs, linearity against the observed outcomes and outliers analysis is the way to go. That is a DV with a correlation of 0.05 should not be discarded at-priori from the regression. On the other hand, if the DV shows a number of heavy outliers or a clearly non-linear relationship against Y, then it should be discarded? Or should we conclude that there is no pratical way to filter and work out DVs before running the regression (that is, the only "way to go" is AIC, BIC, residuals analysis and so on)? $\endgroup$
    – Robert Kubrick
    Dec 7, 2011 at 21:02
  • $\begingroup$ That's a heavy set of questions :-). Some general principles are to identify and discount outliers during the process, but not to use their presence for determining the model itself (you can fit models without the outliers and you can use robust fitting methods such as IWLS/IRLS); that if a bunch of outliers seems to appear, that could indicate a need for a skewness-reducing transformation of the data; and that residuals analysis is always something worth doing (even if only visually when inspecting plots). $\endgroup$
    – whuber ♦
    Dec 7, 2011 at 22:57
  • $\begingroup$ For reference I would like to add a number of suggestions that I've found in Makridakis, Wheelwright and Hyndman regarding DVs selection: 1. Principal Component analysis (Draper and Smith, 1981)2. Distributed lead/lag analysis 3. Best subset and stepwise regression (although I believe these techniques require to run the regression first). $\endgroup$
    – Robert Kubrick
    Dec 8, 2011 at 3:25
  • $\begingroup$ Thanks, Robert. I believe you meant "IVs" instead of "DVs." You will find that the stepwise regression recommendation has been debated, or at least commented on, extensively on these pages, usually negatively. I have commented on PCA for this purpose, making the point that correlations among the IVs tell us practically nothing about their correlations with the DV, so we had better not hope for too much from that approach. The lead-lag relationships seem appropriate only for specialized problems (such as time series analysis). $\endgroup$
    – whuber ♦
    Dec 8, 2011 at 7:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.