# Can an LS estimator be unbiased if there are infinite solutions?

Let

$$\mathbf{y} =\mathbf{A}\mathbf{x} + \mathbf{\eta}$$

where $\mathbf{\eta}$ is a vector of samples of white, zero-mean Gaussian noise. I want to estimate $\mathbf{x}$ (which is deterministic) given $\mathbf{y}$ using the least-squares method.

If the problem has a unique solution, then the estimator $\mathbf{\hat{x}} =(\mathbf{A}^T\mathbf{A})^{-1}\mathbf{A}^T\mathbf{y}$ is unbiased.

However, if $\mathbf{A}$ is doesn't have full rank, then there are infintely many solutions $\mathbf{\hat{x}}$. Are these (or some of them, or one of them) unbiased as well? Is there a proof regarding this topic that I could read?

If $\mathbf A$ doesn't have full rank, then the model is not identifiable, which implies that an unbiased estimator does not exist.
Here is what the proof looks like in this specific scenario. If $\mathbf A$ doesn't have full rank, then there exist two distinct vectors $\mathbf x^{(1)}$ and $\mathbf x^{(2)}$ such that $\mathbf A\mathbf x^{(1)} = \mathbf A\mathbf x^{(2)}$. In this case, the data $\mathbf y = \mathbf A\mathbf x + \eta$ has the same distribution under both $\mathbf x = \mathbf x^{(1)}$ and $\mathbf x = \mathbf x^{(2)}$. Suppose then that an unbiased estimator $T(\mathbf y)$ were to exist. Then by the definition of $T(\mathbf y)$ being unbiased, we must have $E_{\mathbf x^{(1)}}(T(\mathbf y)) = \mathbf x^{(1)}$, where $E_{\mathbf x^{(1)}}$ denotes the expected value taken under the scenario where $\mathbf x=\mathbf x^{(1)}$. Likewise $E_{\mathbf x^{(2)}}(T(\mathbf y)) = \mathbf x^{(2)}$. And yet, since $\mathbf y$ has the same distribution whether $\mathbf x = \mathbf x^{(1)}$ or $\mathbf x = \mathbf x^{(2)}$, it follows that $T(\mathbf y)$ also has the same distribution in both of these scenarios, which means that $E_{\mathbf x^{(1)}}(T(\mathbf y)) = E_{\mathbf x^{(2)}}(T(\mathbf y))$. Putting this together gives $\mathbf x^{(1)} = \mathbf x^{(2)}$, which is a contradiction since $\mathbf x^{(1)}$ and $\mathbf x^{(2)}$ were chosen to be distinct.
• I forgot to mention that $\mathbf{x}$ is deterministic (I just added it to the question). So, in that case, the expressions $E_{\mathbf x^{(1)}}(\cdot)$ and $E_{\mathbf x^{(2)}}(\cdot)$ don't make much sense. How would be the proof if $\mathbf{x}$ is not random? Aug 1 '17 at 12:16
• What do you mean by deterministic? Do you mean that $\mathbf x$ is known? If so, then there would be no point to estimating it; technically then you'd be dealing with the singleton parameter space $\{\mathbf x\}$ and an unbiased estimator would be the constant $\mathbf x$. In my answer, I did not assume $\mathbf x$ was random; I assumed it was an unknown constant, a parameter in the sense of classical statistics. Why do you say that $E_{\mathbf x^{(1)}}(\cdot)$ and $E_{\mathbf x^{(2)}}(\cdot)$ do not make sense? Aug 1 '17 at 16:07
• That's exactly what I meant: $\mathbf{x}$ is an unknown constant. I just misunderstood you, excuse me for that. I thought that when you wrote $E_{\mathbf x^{(1)}}(\cdot)$, you were referring to the expectation relative to $\mathbf{x^{(1)}}$, but I guess that was just notation. Great answer btw. Aug 1 '17 at 20:25