$$\mathbf{y} =\mathbf{A}\mathbf{x} + \mathbf{\eta} $$

where $\mathbf{\eta}$ is a vector of samples of white, zero-mean Gaussian noise. I want to estimate $\mathbf{x}$ (which is deterministic) given $\mathbf{y}$ using the least-squares method.

If the problem has a unique solution, then the estimator $\mathbf{\hat{x}} =(\mathbf{A}^T\mathbf{A})^{-1}\mathbf{A}^T\mathbf{y} $ is unbiased.

However, if $\mathbf{A}$ is doesn't have full rank, then there are infintely many solutions $\mathbf{\hat{x}}$. Are these (or some of them, or one of them) unbiased as well? Is there a proof regarding this topic that I could read?


If $\mathbf A$ doesn't have full rank, then the model is not identifiable, which implies that an unbiased estimator does not exist.

Here is what the proof looks like in this specific scenario. If $\mathbf A$ doesn't have full rank, then there exist two distinct vectors $\mathbf x^{(1)}$ and $\mathbf x^{(2)}$ such that $\mathbf A\mathbf x^{(1)} = \mathbf A\mathbf x^{(2)}$. In this case, the data $\mathbf y = \mathbf A\mathbf x + \eta$ has the same distribution under both $\mathbf x = \mathbf x^{(1)}$ and $\mathbf x = \mathbf x^{(2)}$. Suppose then that an unbiased estimator $T(\mathbf y)$ were to exist. Then by the definition of $T(\mathbf y)$ being unbiased, we must have $E_{\mathbf x^{(1)}}(T(\mathbf y)) = \mathbf x^{(1)}$, where $E_{\mathbf x^{(1)}}$ denotes the expected value taken under the scenario where $\mathbf x=\mathbf x^{(1)}$. Likewise $E_{\mathbf x^{(2)}}(T(\mathbf y)) = \mathbf x^{(2)}$. And yet, since $\mathbf y$ has the same distribution whether $\mathbf x = \mathbf x^{(1)}$ or $\mathbf x = \mathbf x^{(2)}$, it follows that $T(\mathbf y)$ also has the same distribution in both of these scenarios, which means that $E_{\mathbf x^{(1)}}(T(\mathbf y)) = E_{\mathbf x^{(2)}}(T(\mathbf y))$. Putting this together gives $\mathbf x^{(1)} = \mathbf x^{(2)}$, which is a contradiction since $\mathbf x^{(1)}$ and $\mathbf x^{(2)}$ were chosen to be distinct.

  • $\begingroup$ I forgot to mention that $\mathbf{x}$ is deterministic (I just added it to the question). So, in that case, the expressions $E_{\mathbf x^{(1)}}(\cdot)$ and $E_{\mathbf x^{(2)}}(\cdot)$ don't make much sense. How would be the proof if $\mathbf{x}$ is not random? $\endgroup$
    – Tendero
    Aug 1 '17 at 12:16
  • $\begingroup$ What do you mean by deterministic? Do you mean that $\mathbf x$ is known? If so, then there would be no point to estimating it; technically then you'd be dealing with the singleton parameter space $\{\mathbf x\}$ and an unbiased estimator would be the constant $\mathbf x$. In my answer, I did not assume $\mathbf x$ was random; I assumed it was an unknown constant, a parameter in the sense of classical statistics. Why do you say that $E_{\mathbf x^{(1)}}(\cdot)$ and $E_{\mathbf x^{(2)}}(\cdot)$ do not make sense? $\endgroup$ Aug 1 '17 at 16:07
  • $\begingroup$ That's exactly what I meant: $\mathbf{x}$ is an unknown constant. I just misunderstood you, excuse me for that. I thought that when you wrote $E_{\mathbf x^{(1)}}(\cdot)$, you were referring to the expectation relative to $\mathbf{x^{(1)}}$, but I guess that was just notation. Great answer btw. $\endgroup$
    – Tendero
    Aug 1 '17 at 20:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.