# Multiple regression model and correlation between predictors

I am reading the Introduction to Statistical Learning in Python (ISLP) book. I am reading the below paragraph:

Now suppose that the multiple regression is correct and newspaper advertising is not associated with sales, but radio advertising is associated with sales. Then in markets where we spend more on radio our sales will tend to be higher, and as our correlation matrix shows, we also tend to spend more on newspaper advertising in those same markets. Hence, in a simple linear regression which only examines sales versus newspaper, we will observe that higher values of newspaper tend to be associated with higher values of sales, even though newspaper advertising is not directly associated with sales.

My question here is this: how does a multiple regression model know that the coefficient for news paper ads is $$0$$ and not the coefficient for radio? The correlation between radio and news ad can be in any direction.

Setting their causal assumptions to the side (which aren't well explained in this part of the book), the correlation between radio spending and sales is more than double that of newspaper spending based on the scatters on the raw data from ISLR, the R equivalent to ISLP (James et al., 2013).

Looking at the strong associations of all indicators except for newspaper spending, this leads one to believe that the effect of newspaper spending was suppressed after other stronger variables entered the model. Suppression is when an independent variable has a weak correlation with the outcome variable, is correlated with another independent variable, and increases the $$R^2$$ (variance explained). It can also occur when a regression produces a negative coefficient despite the fact that all correlations between the predictor and outcome variables are non-negative, which I show as the case in the regression fitting below (Friedman & Wall, 2005). In terms of what the regression is doing, we can reconsider a simple case of two predictors algebraically as such (lecture notes here):

$$\beta_1 = \frac{r_{y_1} - r_{y_2} r_{12}}{1 - r^2_{12}}$$

Where the $$\beta_1$$ in some part functions as a measure of the partial associations:

One can think of the above equation for the standardized regression coefficient for the first predictor, $$\beta_1$$, as a correction to the correlation between $$x_1$$ and $$y$$ (i.e., $$r_{y_1}$$), where the second term in the numerator, $$r_{y_2} r_{12}$$, is subtracted. Assuming for a moment that the correlation $$r_{y_1}$$ is positive, the value of $$\beta_1$$ will tend to increase if either $$r_{y_1}$$ or $$r_{12}$$ are negative, because subtraction of a negative quantity is the same as adding that quantity. In such a case, $$\beta_1$$ will be larger than $$r_{y_1}$$, which is not the expected direction. As a second example, assuming again that $$r_{y_1}$$ is positive, but, this time, both $$r_{y_2}$$ and $$r_{12}$$ are both positive. If their product, $$r_{y_2} r_{12}$$, happens to be larger than $$r_{y_1}$$, then $$\beta_1$$ will be negative, in essence changing an initial positive association with the dependent variable into a negative (partial) association. Note also that the correlation between $$x_1$$ and $$x_2$$ can make a difference in the estimate of $$\beta_1$$, because of its appearance in the denominator of the equation. Larger values of $$r^2_{12}$$ will tend to increase the value of $$\beta_1$$.

Therefore, based on the correlations between the predictors with each other and the outcome, the regression will "figure out" which one is more predictive of the outcome and suppress the other, as it to a degree obscures the primary effects. This seems to be the case when you fit this in R:

Call:
lm(formula = Sales ~ TV + Newspaper + Radio, data = Advertising)

Residuals:
Min      1Q  Median      3Q     Max
-8.8277 -0.8908  0.2418  1.1893  2.8292

Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept)  2.938889   0.311908   9.422   <2e-16 ***
TV           0.045765   0.001395  32.809   <2e-16 ***
Newspaper   -0.001037   0.005871  -0.177     0.86
Radio        0.188530   0.008611  21.893   <2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 1.686 on 196 degrees of freedom
Multiple R-squared:  0.8972,    Adjusted R-squared:  0.8956
F-statistic: 570.3 on 3 and 196 DF,  p-value: < 2.2e-16


We can see that $$\beta_{\text{Newspaper}} = -0.001$$ despite that it's correlation is $$r = .23$$. This doesn't happen with TV spending because it is about as strong at predicting sales as the radio spending. Much of this is due to the other strong sales indicators "absorbing" the explained variance of the outcome and leaving less for the newspaper spending to determine, even if the overall $$R^2$$ may improve with inclusion of the suppressed effect. Another example of sign-switching from other effects can be found here.

Does that actually mean that newspaper spending has no effect on sales? No. Mathematically we can say less is predicted by newspaper spending after accounting for the other spending indicators. Causally, we can't infer this from purely observational data, of which any number of contaminating factors could be influencing the outcome.

#### References

• Friedman, L., & Wall, M. (2005). Graphical views of suppression and multicollinearity in multiple linear regression. The American Statistician, 59(2), 127–136. https://doi.org/10.1198/000313005X41337
• James, G. M., Witten, D., Hastie, T. J., & Tibshirani, R. (2013). An introduction to statistical learning: With applications in R (Corrected at 6th printing 2015). Springer Springer Science+Business Media.

First: Models don't "know" things.

Second: The question in the book is clearly about real data and Shawn did a great job analyzing that data. However, you say:

The correlation between radio and news ad can be in any direction.

Which indicates that you are interested in the more general case, as well. In this situation I would be very leery of collinearity. This particular case doesn't seem to have high collinearity, but it wouldn't hurt to look at condition indexes. However, a general problem looking at the associations between three kinds of advertising and sales might well have very high collinearity, particularly if the costs of advertising are not adjusted for size of market. Almost everyone is likely to spend more in (say) New York and Los Angeles than in Des Moines and Lubbock. (Unless, maybe, you are selling agricultural equipment). Collinearity can do weird things to parameter estimates (although prediction is not affected).

Finally, be very careful about words like "effect" and "affect" and "cause". It's very easy to slip into using such terms. Even Shawn does it, despite explicitly warning against it and I am sure I have done it on occasion. "Associated with" is much better. But English lacks really good ways to express the results of regression without using causal language.