# How to obtain least squares when $X^TX$ cannot be inverted

This work is all theoretical and for school, so we were only provided this information to work with, no actual y values. I have a simple linear model I have been asked to translate into a matrix, which is below:

I was also asked to find the XTX, which is also done below.

##    c1 c2
## c1  1  1
## c2  1  1


However, I need to find the normal equations AND the least sum of squares for the parameter estimators. I can get the normal equations below:

I've never had a scenario where the Beta0 and Beta1 normal equations were the same. This raises flags, but my bigger issue here is trying to get the estimators of the parameters. This is normally done by

$$\widehat\beta = (X^TX)^{-1} X^TY$$

However, the inverse of the matrix does not exist. I have absolutely no idea where to go from here. There must be some form of solution to this, but my knowledge is not very advanced in this area, so I am pretty clueless on how to proceed? I know there has to be a solution, as there are follow-up questions, such as spelling out the variance-covariance matrix of the estimators and spelling out estimator of error variance. Can someone help?

Original assignment

• Since this is for school, presumably your course has included some topics related to rank-deficient regression, such as regularization methods (ridge-regression, lasso, elastic-net) or psuedo-inverses or related topics. Or else the questions that follow do not actually require a unique solution. Also, you can use math typesetting: math.meta.stackexchange.com/questions/5020/…
– Sycorax
Commented Apr 11, 2022 at 13:38
• The equation preceding $\beta_0=Y_1$ does not imply $\{$ $\beta_0=Y_1$ and $\beta_1=Y_1$ $\}$. You are probably not doing the matrix multiplication right. Commented Apr 11, 2022 at 14:18
• Keep in mind that other than for didactic reasons, you should not use the inverse to compute it even if you could civilstat.com/2015/07/dont-invert-that-matrix-why-and-how
– Tim
Commented Apr 11, 2022 at 14:35
• My previous comment assumes that you've written down the information given in the problem correctly. But the second equation does not necessarily follow from the first equation unless you've already made some substitutions.
– Sycorax
Commented Apr 11, 2022 at 14:45
• This is a duplicate of stats.stackexchange.com/questions/571118/…. Reposting it made this community unaware of our previous conversation, so I will add here the comment I posted there: The matrix inverse is a convenient shorthand for "solve the system of simultaneous equations $(X^\prime X)\beta = X^\prime Y$ for $\beta.$" You can easily do this using elementary algebra..
– whuber
Commented Apr 11, 2022 at 16:37

Try something like this: $$\beta = \begin{bmatrix} \beta_0 \\ \beta_1 \end{bmatrix}\quad,$$ $$X = \begin{bmatrix} 1 & x_1 \\ 1 & x_2 \\ \vdots & \vdots \\ 1 & x_n \end{bmatrix}\quad,$$ $$Y = \begin{bmatrix} y_1 \\ y_2 \\ \vdots \\ y_n \end{bmatrix} \quad,$$ $$\epsilon = \begin{bmatrix} \epsilon_1 \\ \epsilon_2 \\ \vdots \\ \epsilon_n \end{bmatrix} \quad,$$ $$Y = X \beta + \epsilon\quad.$$
Then: $$\hat{\beta} = \left( X^\textrm{T}X \right)^{-1} X^\textrm{T} Y \quad,$$
$$\hat{Y} = X \hat{\beta}\quad .$$
I believe Ridge regression was made at first to solve this problem: adding a small term $$\lambda$$ on the diagonal of $$\boldsymbol{X}^\top\boldsymbol{X}$$ solves imperfect collinearity problems. However if there are some column $$X_j$$ in the matrix which is perfectly correlated ($$\rho(X_j,X_k)=1$$) with another column $$X_k$$, you should drop one of those.