So, I have been looking at this post, and others that are similar and know that the least square estimation of $\beta_1,\beta_2$ will be $(X^TX)^{-1}X^TY$, where the model is $Y_i = \beta_1x_{1i}+ \beta_2x_{2i} + \epsilon_i$. But what is the generally derived formula in this case for $\hat{\beta_1}$ and $\hat{\beta_2}$. How can represent them in a scalar multiplied out value/weight format rather than a matrix/vector format?
7
-
1$\begingroup$ Have you tried writing out the steps when you carry out the matrix arithmetic? What have you found? $\endgroup$– Sycorax ♦Jun 6, 2021 at 2:51
-
1$\begingroup$ Yes, I seem to get a symmetric matrix $X^T X$, but not sure how I would write its inverse and the entries in that? $\endgroup$– User_13Jun 6, 2021 at 2:54
-
$\begingroup$ LU factorization is one way to solve the linear system, and avoid the explicit inverse. What problem are you trying to solve, and why are you trying to avoid linear algebra in solving it? $\endgroup$– Sycorax ♦Jun 6, 2021 at 2:58
-
$\begingroup$ I want to use linear algebra but not sure how to derive the least squares estimators for the two regression weights here. I've set up the data generating model and objective function, solved the optimization problem by taking the partial derivatives with respect to each of the estimators and so forth, but I'm not clear on how I can show/represent the actual values for the estimated $\beta_1$ and $\beta_2$ from the derivation. $\endgroup$– User_13Jun 6, 2021 at 3:08
-
1$\begingroup$ It sounds like you're on the right track, writing out the linear algebra steps using summation notation. If you're stuck writing $(X^T X)^{-1}$, remember that in this problem, $X^T X$ is just a $2 \times 2$ matrix, which has a very simple inverse. $\endgroup$– Sycorax ♦Jun 6, 2021 at 3:23
|
Show 2 more comments
1 Answer
$\begingroup$
$\endgroup$
Based on the comments above (by @Sycorax):
$$X^TX = \begin{pmatrix} x_{11} & x_{12} & ... &x_{1n} \\ x_{21} & x_{22} & ... &x_{2n} \end{pmatrix} \begin{pmatrix} x_{11} & x_{21} \\ x_{12} & x_{22} \\ \vdots &\vdots\\ x_{1n} & x_{2n} \end{pmatrix} = \begin{pmatrix} x_{11}^2 + x_{12}^2 + \:...\: x_{1n}^2 & x_{11}x_{21} + x_{12}x_{22} + \:...\: +x_{1n}x_{2n} \\ x_{11}x_{21} + x_{12}x_{22} + \:...\: +x_{1n}x_{2n} & x_{21}^2 + x_{22}^2 + \:...\: x_{2n}^2 \\ \end{pmatrix} = \begin{pmatrix} \sum x_{1i}^2 & \sum x_{1i}x_{2i}\\ \sum x_{1i}x_{2i} & \sum x_{2i}^2 \\ \end{pmatrix}$$
$$(X^TX)^{-1} = \dfrac{1}{\sum x_{1i}^2 \sum x_{2i}^2 - (\sum x_{1i}x_{2i})^2} \begin{pmatrix} \sum x_{2i}^2 & - \sum x_{1i}x_{2i} \\ - \sum x_{1i}x_{2i} & \sum x_{1i}^2 \end{pmatrix}$$
$$X^T Y = \begin{pmatrix} x_{11} & x_{12} & ... &x_{1n} \\ x_{21} & x_{22} & ... &x_{2n} \end{pmatrix} \begin{pmatrix} y_{1} \\ y_{2} \\ \vdots \\ y_{n} \end{pmatrix} = \begin{pmatrix} x_{11}y_1 + x_{12}y_2 + \: ... \: + x_{1n}y_n \\ x_{21}y_1 + x_{22}y_2 + \: ... \: + x_{2n}y_n \end{pmatrix} = \begin{pmatrix} \sum x_{1i}y_i \\ \sum x_{2i}y_i \end{pmatrix} $$
$$\therefore \beta = (X^TX)^{-1}X^TY = \begin{pmatrix} \hat{\beta_1} \\ \hat{\beta_2} \end{pmatrix} = \dfrac{1}{\sum x_{1i}^2 \sum x_{2i}^2 - (\sum x_{1i}x_{2i})^2}\begin{pmatrix} \sum x_{2i}^2 \sum x_{1i}y_i - \sum x_{1i}x_{2i} \sum x_{2i}y_i \\ -\sum x_{1i}x_{2i} \sum x_{1i}y_i + \sum x_{1i}^2 \sum x_{2i}y_i \end{pmatrix} $$