1
$\begingroup$

This work is all theoretical and for school, so we were only provided this information to work with, no actual y values. I have a simple linear model I have been asked to translate into a matrix, which is below:

enter image description here

I was also asked to find the XTX, which is also done below.

##    c1 c2
## c1  1  1
## c2  1  1

However, I need to find the normal equations AND the least sum of squares for the parameter estimators. I can get the normal equations below:

enter image description here

I've never had a scenario where the Beta0 and Beta1 normal equations were the same. This raises flags, but my bigger issue here is trying to get the estimators of the parameters. This is normally done by

$$ \widehat\beta = (X^TX)^{-1} X^TY $$

However, the inverse of the matrix does not exist. I have absolutely no idea where to go from here. There must be some form of solution to this, but my knowledge is not very advanced in this area, so I am pretty clueless on how to proceed? I know there has to be a solution, as there are follow-up questions, such as spelling out the variance-covariance matrix of the estimators and spelling out estimator of error variance. Can someone help?

Original assignment enter image description here

$\endgroup$
7
  • 3
    $\begingroup$ Since this is for school, presumably your course has included some topics related to rank-deficient regression, such as regularization methods (ridge-regression, lasso, elastic-net) or psuedo-inverses or related topics. Or else the questions that follow do not actually require a unique solution. Also, you can use math typesetting: math.meta.stackexchange.com/questions/5020/… $\endgroup$
    – Sycorax
    Commented Apr 11, 2022 at 13:38
  • 1
    $\begingroup$ The equation preceding $\beta_0=Y_1$ does not imply $\{$ $\beta_0=Y_1$ and $\beta_1=Y_1$ $\}$. You are probably not doing the matrix multiplication right. $\endgroup$ Commented Apr 11, 2022 at 14:18
  • 1
    $\begingroup$ Keep in mind that other than for didactic reasons, you should not use the inverse to compute it even if you could civilstat.com/2015/07/dont-invert-that-matrix-why-and-how $\endgroup$
    – Tim
    Commented Apr 11, 2022 at 14:35
  • $\begingroup$ My previous comment assumes that you've written down the information given in the problem correctly. But the second equation does not necessarily follow from the first equation unless you've already made some substitutions. $\endgroup$
    – Sycorax
    Commented Apr 11, 2022 at 14:45
  • 3
    $\begingroup$ This is a duplicate of stats.stackexchange.com/questions/571118/…. Reposting it made this community unaware of our previous conversation, so I will add here the comment I posted there: The matrix inverse is a convenient shorthand for "solve the system of simultaneous equations $(X^\prime X)\beta = X^\prime Y$ for $\beta.$" You can easily do this using elementary algebra.. $\endgroup$
    – whuber
    Commented Apr 11, 2022 at 16:37

2 Answers 2

1
$\begingroup$

I think your setup is incorrect.

Try something like this: $$ \beta = \begin{bmatrix} \beta_0 \\ \beta_1 \end{bmatrix}\quad,$$ $$ X = \begin{bmatrix} 1 & x_1 \\ 1 & x_2 \\ \vdots & \vdots \\ 1 & x_n \end{bmatrix}\quad,$$ $$ Y = \begin{bmatrix} y_1 \\ y_2 \\ \vdots \\ y_n \end{bmatrix} \quad,$$ $$ \epsilon = \begin{bmatrix} \epsilon_1 \\ \epsilon_2 \\ \vdots \\ \epsilon_n \end{bmatrix} \quad,$$ $$ Y = X \beta + \epsilon\quad.$$

Then: $$ \hat{\beta} = \left( X^\textrm{T}X \right)^{-1} X^\textrm{T} Y \quad,$$
$$ \hat{Y} = X \hat{\beta}\quad .$$

$\endgroup$
0
$\begingroup$

I believe Ridge regression was made at first to solve this problem: adding a small term $\lambda$ on the diagonal of $\boldsymbol{X}^\top\boldsymbol{X}$ solves imperfect collinearity problems. However if there are some column $X_j$ in the matrix which is perfectly correlated ($\rho(X_j,X_k)=1$) with another column $X_k$, you should drop one of those.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.