Consider a linear regression model: $Y_i = \beta_1 A_i + \beta_2 B_i + u_i$ where all variables are assumed to have mean 0, and $A_{i}$ is distributed independently of both $B_{i}$ and $u_{i}$, but $B_{i}$ and $u_{i}$ might be correlated.

I want to show that the OLS estimator $\widehat{\beta}_1$ is consistent but $\widehat{\beta}_2$ might not be due to the correlation. So I'm deducing the formula for the OLS estimators and this is what I got: $$\widehat{\beta}_1 = \left(\sum A_i A_i'\right)^{-1} \left(\sum A_iY_i\right)$$ and $$\widehat{\beta}_2 = \left(\sum B_i B_i'\right)^{-1} \left(\sum B_i Y_i \right).$$ Is this formula correct? Can any one give me some hint on the problem?

Any help is appreciated, thanks in advance!


marked as duplicate by kjetil b halvorsen, mkt, Peter Flom regression Feb 3 at 13:18

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

  • 1
    $\begingroup$ I don't think this is a duplicate because that other question was about fitting only one parameter. A complication arises in this question because the tuples $\left( a_i : i=1,\ldots,n\right)$ and $\left( b_i : i=1,\ldots, n \right)$ may be correlated. And any answer to that other question would not say what to do about that. The answer proposed in this present posted question is wrong precisely because of that issue, which that other question cannot address. $\endgroup$ – Michael Hardy Feb 3 at 18:36
  • $\begingroup$ Thank you! That's exactly what I have in mind. Actually, in the original question, $b_{i}$ are correlated with $u_{i}$ but $a_{i}$ are uncorrelated with $b_{i}, u_{i}$ $\endgroup$ – Juan Feb 3 at 22:31
  • $\begingroup$ This question has a very large number of duplicates: look at any thread that discusses formulas for multiple regression. $\endgroup$ – whuber Feb 3 at 22:38
  • $\begingroup$ I've looked at a lot of threads but none of them discussed the potential correlation between $b_{i}$ and $u_{i}$...I might have missed some threads $\endgroup$ – Juan Feb 3 at 22:40
  • $\begingroup$ Your question doesn't posit or ask about "potential correlation." Could you be more specific about what you're looking for? If you are assuming $b$ and $u$ are random variables and you do not assume they are independent, then at the very least you need to stipulate how they fail to be independent, for otherwise there won't be any formula. $\endgroup$ – whuber Feb 3 at 22:42

Your notation has interesting assumptions. Considering $Y_i$ and $y_i$ are the same, $a_i,b_i$ should be real numbers representing your data points. But,in your notation, they're represented like vectors, e.g. you have $a_ia_i'$. If they were vectors, say $m\times 1$, you couldn't also have $a_i y_i$ because in order to have $y_i=\beta_1a_i+\beta_2b_i+u_i$, $y_i$ needs to have the same dimension with $a_i$, and the vector multiplication $a_iy_i$ becomes invalid. If you just have a typo, i.e. you have something like $\sum a_i y_i'$, which is algebraically valid, then you're either trying to solve a system of multiple outputs with more than one $\beta_1,\beta_2$ (i.e. $\beta_{1,2}$ are vectors, where you have separate $\beta_1,\beta_2$ for each element of $y_i$ vector, and you can solve the systems separately) or you constraint all the system to have the same scalar $\beta_1,\beta_2$ where you can actually add each of the outputs (i.e. $y_i=[y_{i1},y_{i2},...,y_{im}]$) as more data points, in the end having $nm$ data points. Either way, there is no need to solve something like that.

But, if we assume that these are just scalars, in which the problem translates into the usual multiple regression, then you can simply write your solution as $\beta_1=\frac{\sum a_i y_i}{\sum a_i^2}$, which is correct only if you don't have the second regressor. So, your answers are valid for the individual models $y_i=\beta_1 a_i+u_i$ and $y_i=\beta_2 b_i+u_i$. However, this doesn't account for the dependence between $\beta_1,\beta_2$. The coefficients tend to affect each other when other terms are incorporated.

The general formula for multiple regression is $\beta=(X^TX)^{-1}X^Ty$, where $X$ has $n\times p$, and $y$ has $n \times 1$, $\beta$ has $p \times 1$ dimensions (here $p=2$). Each row of $X$ corresponds to data points $(a_i,b_i)$ and rows of $\beta$ corresponds to $\beta_1$ and $\beta_2$. We can also come up with the same solution by differentiating $\text{SSE}=\sum(y_i-\hat{y_i})^2$ and solving for $\beta_1,\beta_2$. Correctly formulating the problem yields the following setup:

$$X=\begin{bmatrix} a_1 & \ldots & a_n \\ b_1 & \ldots & b_n\end{bmatrix}, y=\begin{bmatrix} y_1 \\ \vdots \\ y_n\end{bmatrix},\beta=\begin{bmatrix}\beta_1 \\ \beta_2\end{bmatrix}$$ Now, we just substitute all together into the equations: $$(X^TX)^{-1}=\left(\begin{bmatrix} a_1 & \ldots & a_n \\ b_1 & \ldots & b_n\end{bmatrix}\begin{bmatrix}a_1 & b_1 \\ \vdots&\vdots \\ a_n & b_n \end{bmatrix}\right)^{-1}=\begin{bmatrix}\sum a_i^2 & \sum a_ib_i \\ \sum a_ib_i & \sum b_i^2\end{bmatrix}^{-1}=\frac{1}{D}\begin{bmatrix}\sum b_i^2 & -\sum a_ib_i \\ -\sum a_ib_i & \sum a_i^2\end{bmatrix}$$

where $D=(\sum a_i^2)(\sum b_i^2)-(\sum a_ib_i)^2$, i.e. the determinant. And, we have $X^Ty=\begin{bmatrix}\sum a_iy_i \\ \sum b_iy_i\end{bmatrix}$.

Finally, we'll have $$\beta=\frac{1}{D}\begin{bmatrix}(\sum b_i^2)(\sum a_iy_i)-(\sum a_ib_i)(\sum b_iy_i) \\ (\sum a_i^2)(\sum b_iy_i)-(\sum a_ib_i)(\sum a_iy_i) \end{bmatrix}$$

First row is $\beta_1$, second row is $\beta_2$.

  • $\begingroup$ Thank you for the answer! However, I guess this would only work if we assume $a_{i}, b_{i}, u_{i}$ are all uncorrelated right? In the original question, $a_{i}$ is uncorrelated to others but $b_{i}$ and $u_{i}$ might be related, so $\hat{\beta}_{1}$ is consistent but $\hat{\beta}_{2}$ might not be, and I have problem with showing that... $\endgroup$ – Juan Feb 3 at 22:38
  • $\begingroup$ If you know $b$ and $u$ are correlated but don't attempt to model that correlation, you can still estimate $\hat{\beta}_2$ but it will be biased for $\beta_2$. If you attempt to model that correlation in your estimation, the model is not identified (i.e., there are more unknowns than knowns). So, there is no formula for an unbiased estimator of $\beta_2$ when $b$ and $u$ are correlated. $\endgroup$ – Noah Feb 4 at 1:01

Not the answer you're looking for? Browse other questions tagged or ask your own question.