Let us have two samples

$ X= (X_1,\dots X_n); \quad X_i \sim N(a,\sigma^2)$

$ Y= (Y_1,\dots Y_m); \quad Y_j \sim N(a,4\sigma^2)$.

How to find an unbiased efficient estimations for $a$ and $\sigma^2$?

Some ideas:

  • We know, that $\bar X$ and $S^2_{n-1}$ are both unbiased and efficient estimators.

  • We can't pool samples together, because we will obtain sample with unknown distribution.

  • It is unclear, why combination of estimations from separate samples e.g. $\bar X + \bar Y $ is efficient.

  • 1
    $\begingroup$ The solution is called Weighted (Ordinary) Least Squares. You can find explanations and examples by searching these terms. $\endgroup$ – whuber Jun 10 '18 at 16:43

Let us pool together samples $X$ and $Y$ to sample $Z=(X_1,\dots X_n,Y_1,\dots Y_m)=(Z_1\dots Z_k)$.

Consider linear model: $$Z=a\cdot I_k+\varepsilon,$$ where $I_k=(1,\dots 1)^T$ - $(k \times 1)$ vector and $\varepsilon$ - $(k \times 1)$ error vector.
In our case $\text{cov}(\varepsilon)= diag(\sigma^2,\dots,\sigma^2,4\sigma^2,\dots,4\sigma^2)$.

Let us scale our model by $\frac12$ for least $m$ components with help of $w=(1,\dots1,\frac14,\dots,\frac14)$ $$\tilde Z = \tilde a + \tilde \varepsilon, $$ where $\tilde Z = Z \cdot \sqrt w, \quad \tilde {I_k} = I_k \cdot \sqrt w, \quad \tilde \varepsilon= \varepsilon \cdot \sqrt w$.

This is standart linear regression model with $\text{cov}(\tilde \varepsilon) = \sigma^2I_{k\times k}$.
For this model we know unbiased estimates. Estimate for mean is:

$$\hat {\tilde a}=({\tilde I_k}^T{\tilde I_k})^{-1}{\tilde I_k ^T}\tilde Z =(\sqrt w^T I_k^T I_k \sqrt w )^{-1}\sqrt w ^T I_k^T Z \sqrt w = \\= \frac1k(\sqrt w^T \sqrt w)^{-1} \sqrt w ^T \sum Z \sqrt w = \frac1k \sum Z (\sqrt w^T \sqrt w)^{-1} (\sqrt w ^T \sqrt w) = \frac1{k} \sum Z_i$$ $$ \\{\bf \hat {\tilde a}=\frac1{k} \sum Z_i = \bar Z=\frac1{(n+m)}\left( \sum X_i + \sum Y_j \right)}$$ This estimate for mean is perfectly reasonable.

Estimate for variance is: $$\hat {\sigma}^2 = \frac {RSS}{k-1} = \frac{\Vert \tilde Z-\hat {\tilde {a}} \cdot \tilde {I_k} \Vert_2}{k-1} = \frac{\Vert Z \cdot \sqrt w- \bar Z \cdot I_k \sqrt w \Vert_2}{k-1} = \frac{1}{k-1} \Vert (Z-\bar Z I_k)\sqrt w \Vert_2 $$ $$ (Z \cdot - \bar Z \cdot I_k )\sqrt w= \left( \begin{pmatrix} X_1 \\ \vdots \\ X_n \\ Y_1 \\ \vdots\\ Y_m \end{pmatrix} - \begin{pmatrix} \bar Z\\ \vdots \\ \bar Z\\ \bar Z \\ \vdots\\ \bar Z \end{pmatrix} \right)\cdot (1,\dots,1,\frac12,\dots,\frac12)- = \begin{pmatrix} X_1-\bar Z\\ \vdots \\ X_n-\bar Z\\ \frac12 (Y_1 -\bar Z) \\ \vdots\\ \frac12 (Y_m-\bar Z) \end{pmatrix} $$ $${\bf \hat\sigma^2=\frac{1}{k-1} \left( \sum (X_i-\bar Z)^2 + \frac14 \sum (Y_j - \bar Z)^2 \right)}$$

Which is reasonable too.

| cite | improve this answer | |
  • $\begingroup$ +1--with a big thank you for sharing your solution. $\endgroup$ – whuber Jun 12 '18 at 13:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.