# correlation between independent variables in linear multiple regression

If we were to do simple linear regression with either TV or Online budget as the independent variable and sales and the outcome, we would see a statistically significant regression coefficient using either variable.

However if we fit a multiple regression:

$sale =B_0 + B_1TV + B_2online$

Where $B_1$ might be 0.3 and statistically significant (>0) and $B_2$ close to zero.

I understand that this can be due to the fact that online and TV ad spending may be correlated (more TV ad => more online ad). But the question is, is there a particular reason why it is $B_1$ that's large instead of $B_2$? My intuition is that you can't really tell the difference between which one is actually the main driver.

I apologize in advance for the really long answer, I just don't want to assume any level of familiarity with linear regression. Also, the answer touches on 2-3 different topics, so I wanted to cover all bases.

The answer to your question has to do with what are $B1$ and $B2$. Linear regression is trying to find the unique $\hat{\beta}$ that minimizes the least squares, i.e. in your case: $$\hat{B_0}, \hat{B_1}, \hat{B_2} = \arg\min_{B_0, B_1, B_2} \sum_{i=1}^n\left( sale - B_0 - B_1TV - B_2online\right)^2$$

or in vector notation:

$$\hat{\beta} = \arg\min_{\beta\in\mathbb{R}^3} || y - X\beta||^2_2$$ where $y$ is a $n\times1$ vector with your $sale$ data, and $X$ is a $n\times3$ matrix with your variables, i.e. the first column has all 1s, the second column has your $TV$ data, and the third has the $online$ data.

The way to solve this system of equations is: $$\hat{\beta} = (X^TX)^{-1}X^Ty$$

If your data satisfy the full rank assumption, then your $\hat{\beta}$ is unique, because $(X^TX)^{-1}$ is unique. So in the end, it's nothing more than just solving a system of equations.

Where does correlation come into play? If the correlation between $TV$ and $online$ is 1, then your data do not satisfy the full rank assumption, so the matrix above is not invertible, and there's not unique solution. Practically, this means that the algorithm doesn't know where to assign predictive/explanatory power. If it's close to 1 (e.g. > .9), the computer might have trouble finding the exact inverse, so be careful there. If it's less that that, but still high this will probably inflate your standard errors (multicollinearity).

Finally, why are both variables significant in the univariate cases, but not in the multiple regression? Exactly, because they're correlated, the each have a direct and an indirect effect (through the other variable) on $y$. By including only one of them, you're picking up both effects, but if you include both, you're picking up their direct effects (plus any indirect through other missing variables). Assuming that both variables have some effect on $y$ and they correlate with each other, you should include both of them in the regression, because otherwise you're introducing bias (omitted variable bias).

If you are asking which one is the main driver then it will be TV because spending on TV will result in statistically increased sales and online ad due to being close to zero represents that it has no effect on sales but I can't say that for sure because you haven't mentioned the p-value of online ad budget as it will tell us whether online ad effect is significantly different from zero or not. This whole thing tells you that TV ad is actually effecting the sales and online ad has little effect and the only reason you are seeing the significant coefficient of online ad is only due to the correlation between TV budget and online budget.

I think a simple example may help. Assume $sale = TV$ (yes, exactly) and $online = sale + \epsilon$, where $\epsilon \sim \mathcal{N}(0, 0.001)$ is some noise with small amplitude. If your regress $sale$ against $TV$ and $online$, the minimizer will always give all weight to $TV$ (because this gives an exact fit). If you regress $sale$ against $TV$ alone you will also get a perfect fit. But if you regress $sale$ against $online$ alone you will get pretty good results (because they are almost the same).

It looks like in your situation both variables are good predictors, it's just that one of them is better than the other.