Let us pool together samples $X$ and $Y$ to sample $Z=(X_1,\dots X_n,Y_1,\dots Y_m)=(Z_1\dots Z_k)$.
Consider linear model: $$Z=a\cdot I_k+\varepsilon,$$
where $I_k=(1,\dots 1)^T$ - $(k \times 1)$ vector and $\varepsilon$ - $(k \times 1)$ error vector.
In our case $\text{cov}(\varepsilon)= diag(\sigma^2,\dots,\sigma^2,4\sigma^2,\dots,4\sigma^2)$.
Let us scale our model by $\frac12$ for least $m$ components with help of $w=(1,\dots1,\frac14,\dots,\frac14)$
$$\tilde Z = \tilde a + \tilde \varepsilon, $$ where
$\tilde Z = Z \cdot \sqrt w, \quad \tilde {I_k} = I_k \cdot \sqrt w, \quad \tilde \varepsilon= \varepsilon \cdot \sqrt w$.
This is standart linear regression model with $\text{cov}(\tilde \varepsilon) = \sigma^2I_{k\times k}$.
For this model we know unbiased estimates. Estimate for mean is:
$$\hat {\tilde a}=({\tilde I_k}^T{\tilde I_k})^{-1}{\tilde I_k ^T}\tilde Z
=(\sqrt w^T I_k^T I_k \sqrt w )^{-1}\sqrt w ^T I_k^T Z \sqrt w = \\=
\frac1k(\sqrt w^T \sqrt w)^{-1} \sqrt w ^T \sum Z \sqrt w = \frac1k \sum Z (\sqrt w^T \sqrt w)^{-1} (\sqrt w ^T \sqrt w) = \frac1{k} \sum Z_i$$ $$ \\{\bf \hat {\tilde a}=\frac1{k} \sum Z_i = \bar Z=\frac1{(n+m)}\left( \sum X_i + \sum Y_j \right)}$$
This estimate for mean is perfectly reasonable.
Estimate for variance is:
$$\hat {\sigma}^2 = \frac {RSS}{k-1} = \frac{\Vert \tilde Z-\hat {\tilde {a}} \cdot \tilde {I_k} \Vert_2}{k-1} = \frac{\Vert Z \cdot \sqrt w- \bar Z \cdot I_k \sqrt w \Vert_2}{k-1} = \frac{1}{k-1} \Vert (Z-\bar Z I_k)\sqrt w \Vert_2 $$
$$
(Z \cdot - \bar Z \cdot I_k )\sqrt w=
\left(
\begin{pmatrix}
X_1 \\
\vdots \\
X_n \\
Y_1 \\
\vdots\\
Y_m
\end{pmatrix}
-
\begin{pmatrix}
\bar Z\\
\vdots \\
\bar Z\\
\bar Z \\
\vdots\\
\bar Z
\end{pmatrix}
\right)\cdot
(1,\dots,1,\frac12,\dots,\frac12)-
=
\begin{pmatrix}
X_1-\bar Z\\
\vdots \\
X_n-\bar Z\\
\frac12 (Y_1 -\bar Z) \\
\vdots\\
\frac12 (Y_m-\bar Z)
\end{pmatrix}
$$
$${\bf \hat\sigma^2=\frac{1}{k-1} \left( \sum (X_i-\bar Z)^2 + \frac14 \sum (Y_j - \bar Z)^2 \right)}$$
Which is reasonable too.