3
$\begingroup$

Given $X\in\mathbb R^{n\times p}$ and $y\in \mathbb R^n$, the least square coefficients are: $\hat{\beta} = \text{argmin} \| X\beta - y\|^2_2$.

Is $\hat{\beta}$ unique in the case $\text{rank}(X)=p$?

$\endgroup$
  • $\begingroup$ Yes, providing that $n \ge p$. The key is that $\beta$ is unique if the columns of $X$ are linearly independent, which $rank(X) = p$ combined with $n \ge p$ ensures. $\endgroup$ – Mark L. Stone Apr 4 '16 at 0:22
  • $\begingroup$ can you write a proof ? $\endgroup$ – Donbeo Apr 4 '16 at 0:23
  • 6
    $\begingroup$ I could, but I leave that to you as an exercise. $\endgroup$ – Mark L. Stone Apr 4 '16 at 0:24
  • $\begingroup$ There's a algebraic-geometric proof if you know the theory of vector spaces. Are you familiar with vector spaces ? $\endgroup$ – Stéphane Laurent Apr 4 '16 at 10:40
  • $\begingroup$ Also see stats.stackexchange.com/questions/194460 $\endgroup$ – whuber Apr 4 '16 at 16:09
6
$\begingroup$

Assume $\text{rank}(X) = p$ and $n \geq p$ (in fact $\text{rank}(X) = p$ implies that $n \geq p$), the least square estimator has an explicit expression: $$\hat{\beta} = (X^TX)^{-1}X^Ty,$$ in which $X^TX$ is non-singular. Since the inverse is unique, $\hat{\beta}$ is unique.

So probably the only difficult part is to show that $X^TX$ is non-singular provided $\text{rank}(X) = p$. There are multiple ways to show that, perhaps the easiest way is to use $$\text{rank}(X^TX) = \text{rank}(X) \tag{$*$}$$ for any matrix $X$, also note that $X^TX$ is a $p \times p$ matrix, thus $X^TX$ has to be non-singular. Let me know if you need a proof for $(*)$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.