Estimator of two regression models

Question

Suppose I have two noisy versions of $\theta$, which is a parameter we want to estimate: $\theta_1=\theta+\epsilon_1$, and $\theta_2=\theta+\epsilon_2$. I want to use $\theta_1$ and $\theta_2$ to estimate $\theta$. And the covariance matrix of $(\epsilon_1, \epsilon_2)$ is known $\Sigma_{2\times2}$, but not necessarily to be independent (i.e., diagonal).

For an estimator $\hat{\theta}=f(\theta_1,\theta_2)$, I want to know that when, for example, only $\theta_1$ is useful (i.e. $\theta_1$ is a sufficient statistics of $\theta$)? We can assume linear unbiased estimators $a\theta_1+(1-a)\theta_2$ first, but not necessarily.

For example, over all the linear estimators, we can find the best one (minimal variance) by minimize the variance (of which the quadratic problem depends on the covariance of the error $\Sigma$). And sometimes $a=1$, which means only $\theta_1$ is useful in the best case. Is there any reason why $\theta_2$ is useless in such case?

Answer 1

Intuitively, you'd want to give a larger weight to the more reliable predictor. So in the extreme case where one of the predictors is perfectly reliable (noise $\epsilon=0$), that predictor should have a weight of 1. Assigning a non-zero weight to the other predictor will not improve your estimate in this case, so the other weight should be 0. Here's a formal explanation.

Let ${\bf \Theta} =\big( \theta_1 \quad \theta_2 \big)^T$ denote the predictors (that satisfy the relations in your question) and $\Sigma=\begin{pmatrix} \sigma_1^2 & \rho\sigma_1\sigma_2 \\\rho\sigma_1\sigma_2 & \sigma_2^2 \end{pmatrix}$ be the covariance between noise in the predictors. Let ${\bf a}= \big( a_1 \quad a_2 \big)^T$ be the vector of weights on the predictors such that $\hat\theta={\bf a}^T{\bf \Theta}$ is an unbiased linear estimator of $\theta$ . The variance of this estimator must satisfy the Cramer-Rao bound, which (assuming Gaussian noise) for your case is:

\begin{align} Var(\hat\theta) = {\bf a}^T\Sigma{\bf a} &\ge-\Big(E\Big[\frac{\partial^2lnP({\bf \Theta};\theta)}{\partial\theta^2}\Big]\Big)^{-1}\\ &\ge \Big(\frac{\partial\theta_1}{\partial\theta} \quad \frac{\partial\theta_2}{\partial\theta}\Big)\Sigma^{-1}\Big(\frac{\partial\theta_1}{\partial\theta} \quad \frac{\partial\theta_2}{\partial\theta}\Big)^T \\ & \ge \big(1 \quad 1\big)\Sigma^{-1}\big(1 \quad 1\big)^T \end{align}

You can verify that the optimal value of ${\bf a}$ that minimizes the variance (i.e., the case for which equality holds) is: ${\bf a}^*=\frac{1}{\sigma_1^2+\sigma_2^2-2\rho\sigma_1\sigma_2}\begin{pmatrix} \sigma_2^2 - \rho\sigma_1\sigma_2 \\ \sigma_1^2 - \rho\sigma_1\sigma_2 \end{pmatrix}$.

From the above equation, you can see that if, for example, $\sigma_1=0$, then ${\bf a}^*=(1 \quad 0)^T$. You can play with different values of $\sigma_1, \sigma_2, $ and $\rho$ to see how they affect the optimal weights ${\bf a}^*$ and the quality of your estimator.