2
$\begingroup$

If $X^TX$ is not invertible, what is the set of solutions for the least squares estimator $\hat{\beta_1}$ in the below?

$Y_i = \beta_0 +\beta_1(x_i-\bar{x}) +\epsilon_i$

I got as far as writing out the normal equations, but not sure how i'd begin to solve the equations

$\endgroup$
2
$\begingroup$

You want all the possible solutions? Write the linear model in matrix form as $$ Y=X\beta + \epsilon, $$ and let the More-Penrose inverse of matrix $A$ be denoted by $A^+$. The normal equations for the least squares problem is $$ X^TX \beta= X^T Y$$, and if $X^TX$ is invertible then $\hat{\beta}=(X^TX)^{-1}X^TY$ is the unique solution. Otherwise, we can use the Moore-Penrose inverse to find the minimum norm solution $\beta^*=(X^TX)^+ X^TY$.

But in this case there are infinitely many other solutions. To find all we use the principle that all solutions of a linear system can be found as one particular solution of the inhomogeneous equation plus the general solution of the corresponding homogeneous equation. In this case that is $X^X \beta=0$, and the solutions (on unknown $\beta$) constitutes the null space or kernel of $X^TX$, which is known to coincide with the kernel of $X$.

So all solutions can be found by adding some vector in the null space of $X$ to $\beta^*$, which is our particular solution. The null space of $X$ can be described by using the singular value decomposition, SVD. If the rank of $X$ is $q$, this can be written as $X=U D V^T$ where $X$ is $n\times p$, $u$ is $n\times q$, $D$ is $q\times q$ diagonal with the positive singular values on the diagonal and $V$ is $p\times q$. $U, V$ are both column orthogonal. Then the null space of $X$ is $\{ \beta \colon U D V^T=0\}$, which can be reduced to $\{\beta \colon V^T\beta=0\}$. That is simply the orthogonal complement to the column space of $V$, which is a subspace in $\mathbb{R}^p$.

$\endgroup$
1
$\begingroup$

EDITED based on remarks by @AdamO

If $X^TX$ is not invertible:

  1. there is no unique solution for $\beta_1$
  2. it means not all columns are linearly independent, for example two columns might be proportional to each other (i.e. one is superfluous, just the same measurement but with a different unit of measure).
$\endgroup$
  • 4
    $\begingroup$ Not quite... 1: you can use a quasiinverse, solutions exist but are not unique. 2: one column can be expressed as a linear combination of the other columns. Removing one of the culprit columns is a way to obtain a full rank design matrix $\endgroup$ – AdamO May 12 '18 at 12:24
  • $\begingroup$ @AdamO yes, you are right. still, there is no use in having a not invertible design matrix. $\endgroup$ – fabiob May 12 '18 at 12:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.