2
$\begingroup$

Let us have two samples

$ X= (X_1,\dots X_n); \quad X_i \sim N(a,\sigma^2)$

$ Y= (Y_1,\dots Y_m); \quad Y_j \sim N(a,4\sigma^2)$.

How to find an unbiased efficient estimations for $a$ and $\sigma^2$?

Some ideas:

  • We know, that $\bar X$ and $S^2_{n-1}$ are both unbiased and efficient estimators.

  • We can't pool samples together, because we will obtain sample with unknown distribution.

  • It is unclear, why combination of estimations from separate samples e.g. $\bar X + \bar Y $ is efficient.

$\endgroup$
1
  • 1
    $\begingroup$ The solution is called Weighted (Ordinary) Least Squares. You can find explanations and examples by searching these terms. $\endgroup$
    – whuber
    Jun 10, 2018 at 16:43

1 Answer 1

2
$\begingroup$

Let us pool together samples $X$ and $Y$ to sample $Z=(X_1,\dots X_n,Y_1,\dots Y_m)=(Z_1\dots Z_k)$.

Consider linear model: $$Z=a\cdot I_k+\varepsilon,$$ where $I_k=(1,\dots 1)^T$ - $(k \times 1)$ vector and $\varepsilon$ - $(k \times 1)$ error vector.
In our case $\text{cov}(\varepsilon)= diag(\sigma^2,\dots,\sigma^2,4\sigma^2,\dots,4\sigma^2)$.

Let us scale our model by $\frac12$ for least $m$ components with help of $w=(1,\dots1,\frac14,\dots,\frac14)$ $$\tilde Z = \tilde a + \tilde \varepsilon, $$ where $\tilde Z = Z \cdot \sqrt w, \quad \tilde {I_k} = I_k \cdot \sqrt w, \quad \tilde \varepsilon= \varepsilon \cdot \sqrt w$.

This is standart linear regression model with $\text{cov}(\tilde \varepsilon) = \sigma^2I_{k\times k}$.
For this model we know unbiased estimates. Estimate for mean is:

$$\hat {\tilde a}=({\tilde I_k}^T{\tilde I_k})^{-1}{\tilde I_k ^T}\tilde Z =(\sqrt w^T I_k^T I_k \sqrt w )^{-1}\sqrt w ^T I_k^T Z \sqrt w = \\= \frac1k(\sqrt w^T \sqrt w)^{-1} \sqrt w ^T \sum Z \sqrt w = \frac1k \sum Z (\sqrt w^T \sqrt w)^{-1} (\sqrt w ^T \sqrt w) = \frac1{k} \sum Z_i$$ $$ \\{\bf \hat {\tilde a}=\frac1{k} \sum Z_i = \bar Z=\frac1{(n+m)}\left( \sum X_i + \sum Y_j \right)}$$ This estimate for mean is perfectly reasonable.

Estimate for variance is: $$\hat {\sigma}^2 = \frac {RSS}{k-1} = \frac{\Vert \tilde Z-\hat {\tilde {a}} \cdot \tilde {I_k} \Vert_2}{k-1} = \frac{\Vert Z \cdot \sqrt w- \bar Z \cdot I_k \sqrt w \Vert_2}{k-1} = \frac{1}{k-1} \Vert (Z-\bar Z I_k)\sqrt w \Vert_2 $$ $$ (Z \cdot - \bar Z \cdot I_k )\sqrt w= \left( \begin{pmatrix} X_1 \\ \vdots \\ X_n \\ Y_1 \\ \vdots\\ Y_m \end{pmatrix} - \begin{pmatrix} \bar Z\\ \vdots \\ \bar Z\\ \bar Z \\ \vdots\\ \bar Z \end{pmatrix} \right)\cdot (1,\dots,1,\frac12,\dots,\frac12)- = \begin{pmatrix} X_1-\bar Z\\ \vdots \\ X_n-\bar Z\\ \frac12 (Y_1 -\bar Z) \\ \vdots\\ \frac12 (Y_m-\bar Z) \end{pmatrix} $$ $${\bf \hat\sigma^2=\frac{1}{k-1} \left( \sum (X_i-\bar Z)^2 + \frac14 \sum (Y_j - \bar Z)^2 \right)}$$

Which is reasonable too.

$\endgroup$
1
  • $\begingroup$ +1--with a big thank you for sharing your solution. $\endgroup$
    – whuber
    Jun 12, 2018 at 13:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.