2
$\begingroup$

So, I have been looking at this post, and others that are similar and know that the least square estimation of $\beta_1,\beta_2$ will be $(X^TX)^{-1}X^TY$, where the model is $Y_i = \beta_1x_{1i}+ \beta_2x_{2i} + \epsilon_i$. But what is the generally derived formula in this case for $\hat{\beta_1}$ and $\hat{\beta_2}$. How can represent them in a scalar multiplied out value/weight format rather than a matrix/vector format?

$\endgroup$
7
  • 1
    $\begingroup$ Have you tried writing out the steps when you carry out the matrix arithmetic? What have you found? $\endgroup$
    – Sycorax
    Commented Jun 6, 2021 at 2:51
  • 1
    $\begingroup$ Yes, I seem to get a symmetric matrix $X^T X$, but not sure how I would write its inverse and the entries in that? $\endgroup$
    – User_13
    Commented Jun 6, 2021 at 2:54
  • $\begingroup$ LU factorization is one way to solve the linear system, and avoid the explicit inverse. What problem are you trying to solve, and why are you trying to avoid linear algebra in solving it? $\endgroup$
    – Sycorax
    Commented Jun 6, 2021 at 2:58
  • $\begingroup$ I want to use linear algebra but not sure how to derive the least squares estimators for the two regression weights here. I've set up the data generating model and objective function, solved the optimization problem by taking the partial derivatives with respect to each of the estimators and so forth, but I'm not clear on how I can show/represent the actual values for the estimated $\beta_1$ and $\beta_2$ from the derivation. $\endgroup$
    – User_13
    Commented Jun 6, 2021 at 3:08
  • 1
    $\begingroup$ It sounds like you're on the right track, writing out the linear algebra steps using summation notation. If you're stuck writing $(X^T X)^{-1}$, remember that in this problem, $X^T X$ is just a $2 \times 2$ matrix, which has a very simple inverse. $\endgroup$
    – Sycorax
    Commented Jun 6, 2021 at 3:23

1 Answer 1

3
$\begingroup$

Based on the comments above (by @Sycorax):

$$X^TX = \begin{pmatrix} x_{11} & x_{12} & ... &x_{1n} \\ x_{21} & x_{22} & ... &x_{2n} \end{pmatrix} \begin{pmatrix} x_{11} & x_{21} \\ x_{12} & x_{22} \\ \vdots &\vdots\\ x_{1n} & x_{2n} \end{pmatrix} = \begin{pmatrix} x_{11}^2 + x_{12}^2 + \:...\: x_{1n}^2 & x_{11}x_{21} + x_{12}x_{22} + \:...\: +x_{1n}x_{2n} \\ x_{11}x_{21} + x_{12}x_{22} + \:...\: +x_{1n}x_{2n} & x_{21}^2 + x_{22}^2 + \:...\: x_{2n}^2 \\ \end{pmatrix} = \begin{pmatrix} \sum x_{1i}^2 & \sum x_{1i}x_{2i}\\ \sum x_{1i}x_{2i} & \sum x_{2i}^2 \\ \end{pmatrix}$$

$$(X^TX)^{-1} = \dfrac{1}{\sum x_{1i}^2 \sum x_{2i}^2 - (\sum x_{1i}x_{2i})^2} \begin{pmatrix} \sum x_{2i}^2 & - \sum x_{1i}x_{2i} \\ - \sum x_{1i}x_{2i} & \sum x_{1i}^2 \end{pmatrix}$$

$$X^T Y = \begin{pmatrix} x_{11} & x_{12} & ... &x_{1n} \\ x_{21} & x_{22} & ... &x_{2n} \end{pmatrix} \begin{pmatrix} y_{1} \\ y_{2} \\ \vdots \\ y_{n} \end{pmatrix} = \begin{pmatrix} x_{11}y_1 + x_{12}y_2 + \: ... \: + x_{1n}y_n \\ x_{21}y_1 + x_{22}y_2 + \: ... \: + x_{2n}y_n \end{pmatrix} = \begin{pmatrix} \sum x_{1i}y_i \\ \sum x_{2i}y_i \end{pmatrix} $$

$$\therefore \beta = (X^TX)^{-1}X^TY = \begin{pmatrix} \hat{\beta_1} \\ \hat{\beta_2} \end{pmatrix} = \dfrac{1}{\sum x_{1i}^2 \sum x_{2i}^2 - (\sum x_{1i}x_{2i})^2}\begin{pmatrix} \sum x_{2i}^2 \sum x_{1i}y_i - \sum x_{1i}x_{2i} \sum x_{2i}y_i \\ -\sum x_{1i}x_{2i} \sum x_{1i}y_i + \sum x_{1i}^2 \sum x_{2i}y_i \end{pmatrix} $$

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.