Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
Let us have two samples
$ X= (X_1,\dots X_n); \quad X_i \sim N(a,\sigma^2)$
$ Y= (Y_1,\dots Y_m); \quad Y_j \sim N(a,4\sigma^2)$.
How to find an unbiased efficient estimations for $a$ and $\sigma^2$?
Some ideas:
We know, that $\bar X$ and $S^2_{n-1}$ are both unbiased and efficient estimators.
We can't pool samples together, because we will obtain sample with unknown distribution.
It is unclear, why combination of estimations from separate samples e.g. $\bar X + \bar Y $ is efficient.
Let us pool together samples $X$ and $Y$ to sample $Z=(X_1,\dots X_n,Y_1,\dots Y_m)=(Z_1\dots Z_k)$.
Consider linear model: $$Z=a\cdot I_k+\varepsilon,$$
where $I_k=(1,\dots 1)^T$ - $(k \times 1)$ vector and $\varepsilon$ - $(k \times 1)$ error vector.
In our case $\text{cov}(\varepsilon)= diag(\sigma^2,\dots,\sigma^2,4\sigma^2,\dots,4\sigma^2)$.
Let us scale our model by $\frac12$ for least $m$ components with help of $w=(1,\dots1,\frac14,\dots,\frac14)$ $$\tilde Z = \tilde a + \tilde \varepsilon, $$ where $\tilde Z = Z \cdot \sqrt w, \quad \tilde {I_k} = I_k \cdot \sqrt w, \quad \tilde \varepsilon= \varepsilon \cdot \sqrt w$.
This is standart linear regression model with $\text{cov}(\tilde \varepsilon) = \sigma^2I_{k\times k}$.
For this model we know unbiased estimates. Estimate for mean is:
$$\hat {\tilde a}=({\tilde I_k}^T{\tilde I_k})^{-1}{\tilde I_k ^T}\tilde Z =(\sqrt w^T I_k^T I_k \sqrt w )^{-1}\sqrt w ^T I_k^T Z \sqrt w = \\= \frac1k(\sqrt w^T \sqrt w)^{-1} \sqrt w ^T \sum Z \sqrt w = \frac1k \sum Z (\sqrt w^T \sqrt w)^{-1} (\sqrt w ^T \sqrt w) = \frac1{k} \sum Z_i$$ $$ \\{\bf \hat {\tilde a}=\frac1{k} \sum Z_i = \bar Z=\frac1{(n+m)}\left( \sum X_i + \sum Y_j \right)}$$ This estimate for mean is perfectly reasonable.
Estimate for variance is: $$\hat {\sigma}^2 = \frac {RSS}{k-1} = \frac{\Vert \tilde Z-\hat {\tilde {a}} \cdot \tilde {I_k} \Vert_2}{k-1} = \frac{\Vert Z \cdot \sqrt w- \bar Z \cdot I_k \sqrt w \Vert_2}{k-1} = \frac{1}{k-1} \Vert (Z-\bar Z I_k)\sqrt w \Vert_2 $$ $$ (Z \cdot - \bar Z \cdot I_k )\sqrt w= \left( \begin{pmatrix} X_1 \\ \vdots \\ X_n \\ Y_1 \\ \vdots\\ Y_m \end{pmatrix} - \begin{pmatrix} \bar Z\\ \vdots \\ \bar Z\\ \bar Z \\ \vdots\\ \bar Z \end{pmatrix} \right)\cdot (1,\dots,1,\frac12,\dots,\frac12)- = \begin{pmatrix} X_1-\bar Z\\ \vdots \\ X_n-\bar Z\\ \frac12 (Y_1 -\bar Z) \\ \vdots\\ \frac12 (Y_m-\bar Z) \end{pmatrix} $$ $${\bf \hat\sigma^2=\frac{1}{k-1} \left( \sum (X_i-\bar Z)^2 + \frac14 \sum (Y_j - \bar Z)^2 \right)}$$
Which is reasonable too.
