Linear relationship between explanatory variables in multiple regression

Question

I was reading the multiple regression chapter of Data Analysis and Graphics Using R: An Example-Based Approach and was a bit confused to find out that it recommends checking for linear relationships between explanatory variables (using a scatterplot) and, in case there aren't any, transforming them so they do become more linearly related. Here are some excerpts of this:

6.3 A strategy for fitting multiple regression models

(...)

Examine the scatterplot matrix involving all the explanatory variables. (Including the dependent variable is, at this point, optional.) Look first for evidence of non-linearity in the plots of explanatory variables against each other.

(...)

This point identifies a model search strategy - seek models in which regression relationships between explanatory variables follow a "simple" linear form. Thus, if some pairwise plots show evidence of non-linearity, consider use of transformation(s) to give more nearly linear relationships. While it may not necessarily prove possible, following this strategy, to adequately model the regression relationship, this is a good strategy, for the reasons given below, to follow in starting the search.

(...)

If relationships between explanatory variables are approximately linear, perhaps after transformation, it is then possible to interpret plots of predictor variables against the response variable with confidence.

(...)

It may not be possible to find transformations of one or more of the explanatory variables that ensure the the (pairwise) relationships shown in the panels appear linear. This can create problems both for the interpretation of the diagnostic plots for any fitted regression equation and for the interpretation of the coefficients in the fitted equation. See Cook and Weisberg(1999).

Shouldn't I be worried about linear relationships between dependent variables (because of the risk of multicollinearity) instead of actively pursuing them? What are the advantages of having approximately linearly related variables?

The authors do address the issue of multicollinearity later in the chapter, but this recommendations seem to be at odds with avoiding multicollinearity.

Answer 1

There are two points here:

1. The passage recommends transforming IVs to linearity only when there is evidence of nonlinearity. Nonlinear relationships among IVs can also cause collinearity and, more centrally, may complicate other relationships. I am not sure I agree with the advice in the book, but it's not silly.

2. Certainly very strong linear relationships can be causes of collinearity, but high correlations are neither necessary nor sufficient to cause problematic collinearity. A good method of diagnosing collinearity is the condition index.

EDIT in response to comment

Condition indexes are described briefly here as "square root of the maximum eigenvalue divided by the minimum eigenvalue". There are quite a few posts here on CV that discuss them and their merits. The seminal texts on them are two books by David Belsley: Conditioning diagnostics and Regression Diagnostics (which has a new edition, 2005, as well).

Answer 2

Linear relations between each of the explanatory variables and the dependent variable will ensure also linear relations between the explanatory variables. The reverse is not of course true.

It is true that transformation(s) designed to give approximate linearity will increase collinearity. In the absence of such transformation(s), however, the collinearity is hidden. Insisting on keeping the collinearlity thus hidden can result in a complicated and uninterpretable regression equation, where a simple form of equation is available.

Suppose that y is close to a linear function of log(x1), in a case where x ranges over values that differ by a factor of 10 or more. Then if x is used as a regressor, other explanatory variables will if at all possible be invoked to account for the nonlinearity in the relationship with x1. The result may be a very complicated regression relationship, with uninterpretable coefficients, in place of a simple form of regression equation that captures all the available explanatory power.

The bizarre consequences that may result from failure to find and work with linearly related variables are well illustrated in the recent paper that claimed a femaleness of hurricane name effect in data on deaths from 94 Atlantic hurricanes that made landfall over the US over 1950-2012. See http://www.pnas.org/content/111/24/8782.abstract. The data are available as part of the supplementary information. Note that working with log(deaths) and using a normaL theory linear model (R's function lm()) is roughly equivalent to Jung et al's use of a negative binomial regression model.

If one regresses log(E[deaths]) on log(NDAM), there is nothing left for the minimum pressure variable, the femaleness variable, and interactions, to explain. The variable log(NDAM), not NDAM, appears in a scatterplot matrix as linearly related to the minimum pressure variable. Its distribution is also much less skew, much closer to symmetric.

Jung et al regressed log(E[deaths]) on NDAM (normalised damage), plus those other variables and interactions. The equation that then emerged was used to tell a story in which the femaleness of the name has a large effect.

To see how bizarre it is to use NDAM as an explanatory variable in a regression where the outcome variable is log(E[deaths]), plot log(deaths+0.5) or log(deaths+1) against NDAM. Then repeat the plot with log(NDAM) in place of NDAM. The contrast is even more striking if Katrina and Audrey, which Jung et al omitted as outliers, are included in the plot. By insisting on using NDAM as the explanatory variable, rather than log(NDAM), Jung et al passed up the opportunity to find a very simple form of regression relationship.

NB that E[deaths] is the number of deaths predicted by the model.

In the Jung et al data, the transformations needed can be identified from a scatterplot matrix of all variables. Try perhaps the R function spm() in the latest release of the car package for R, with transform=TRUE and (with deaths as a variable) family="yjPower". Or experiment with the transformations suggested by an initial scatterplot matrix. In general, the preferred advice may be to look first for explanatory variables that satisfy the linear predictors requirement, then attend to the outcome variable, perhaps using the car function invTranPlot().

See, in addition to "Data Analysis and Graphics Using R" that was referenced by the questioner:

• Weisberg: Applied Linear Regression. 4th edn, Wiley 2014, pp.185-203.
• Fox and Weisberg: An R Companion to Applied Regression. 2nd edn, Sage, 2011, pp.127-148.

Answer 3

I find this whole passage rather cryptic if not downright questionable. Ideally, you want your independent variables to be as uncorrelated as possible from one another so as to provide incremental and additional information to the model in estimating the dependent variable. You raise the issue of multicollinearity through high correlations between independent variables, and you are perfectly right to raise that issue in this circumstance.

It is more critical to examine the scatter plot and related linear relationship between each of the independent variables and the dependent variable, but not between the independent variables. When looking at such scatter plots (independent on X-axis and dependent on Y-axis) at such time there may be opportunities to transform the independent variable to observe a better fit whether it is through a log, an exponent, or polynomial form.