3
$\begingroup$

Take the following situation. We believe that $X$ is drawn from a normal distribution with unknown mean and variance; we wish to estimate the population mean.

Previously, several samples were drawn from the population of $X$, say $n$ of them. They were drawn appropriately and independently, and we have their sample means and sample variances.

If we had their sample sizes as well, we could essentially do a weighted average of all the sample means and get a good estimate of the population mean (right?).

However, in my context (essentially an online learning algorithm) we do not have the sample sizes. So how would we go about estimating the population mean?

I thought about maximum-likelihood estimators, but it seems really hard - it seems like we need to estimate the population variance at the same time, and the result is a lot of calculus that (a) I can't solve analytically, and (b) doesn't look like it has a unique solution, due to an apparently lack of concavity.

Is this a classical problem with a classical solution?

$\endgroup$

2 Answers 2

1
$\begingroup$

If the goal is to only estimate the population mean, and not the standard error, then you can just take the mean of the means. Here is why this works.

Lets consider two samples, one of size $n_1$ consisting of iid draws from $N(\mu, \sigma^2)$. Let the sample mean of this sample be $\bar{X}_1$. Similarly the second sample consists of $n_2$ iid draws from $N(\mu, \sigma^2)$ with sample mean $\bar{X}_2$. Then $$\bar{X}_1 \sim N\left( \mu, \dfrac{\sigma^2}{n_1}\right) \quad \text{ and } \quad\bar{X}_2 \sim N\left(\mu, \dfrac{\sigma^2}{n_2} \right). $$

Thus, essentially you have two realizations from two normals with different variances but the same mean. If you take the mean of these means, i.e. $$\bar{X} = \dfrac{\bar{X}_1 + \bar{X}_2}{2}, $$

then $$E(\bar{X}) = \mu. $$

So your estimator will be unbiased and if the number of samples $n$ is large, then it might even be a decent estimator. Only thing is you won't know what the variance of this estimator is, which is far from ideal.

$\endgroup$
3
  • 1
    $\begingroup$ I agree that's unbiased. However, there are other unbiased estimators; any convex combination of the two estimates in your simpler situation would be unbiased. Wouldn't it make more sense to give more weight to the one with more elements (for example)? Perhaps giving weight to one with the lower sample variance? $\endgroup$ Aug 5, 2016 at 2:11
  • $\begingroup$ You could look at more convex combinations, and weigh as you suggested. This estimator I present will probably have a large variance. It seemed from the question that you just wanted an estimator, so I wrote up the answer. $\endgroup$ Aug 5, 2016 at 2:19
  • $\begingroup$ Well, I appreciate it. I am hoping to see a good way to take advantage of the sample variances, though, if possible. $\endgroup$ Aug 5, 2016 at 3:02
0
$\begingroup$

Yes. There is a statistical solution to this. You were right about using the variances to estimate the population mean.

Consider two samples for the same random variable having sample means x and y and their sample variances $\sigma_x^2$ and $\sigma^2_y$.

The way these two samples are fused is:

$$\hat{x}^+ = \frac{\hat{x}^-*\sigma_y^2 + y*\sigma_{x^-}^2}{\sigma_{x^-}^2 + \sigma_y^2}$$

Where:

$\hat{x}^-$ : Sample Mean Prediction from Process Model.

$\sigma_{x^-}^2$ : Sample Variance of Predictions.

$y$ : Sample Mean Measurement from Sensor.

$\sigma_y^2$ : Sample Variance of Measurement.

$\hat{x}^+$ : Fused Sample Mean. Or Optimal Estimate in Kalman Filtering.

This is hence a weighted mean of the two sample means depending on their respective variances.

I do not remember the Statistical Term for this, but this fusion method is the foundational principle of Kalman Filtering.

Kalman Filter Sample Fusion. Mathworks

In Kalman Filtering, the noisy measurement sample is fused with process predictions which also have some error. This is the way to mitigate the error and get the best estimate.

Hope it helps.

$\endgroup$
3
  • 1
    $\begingroup$ Welcome to cv, kartiks77! It is a really good idea to use the variances for weighting - I wonder why this has not been suggested before. I would have upvoted but then I looked closer at your formula: shouldn't the mean that is more precise get a larger weight? Please check your formula again and try to wite it in LaTeX instead of pasting a copied image. Images can be unreadable for vision impaired readers. :-) $\endgroup$
    – Ute
    Aug 27, 2023 at 13:46
  • $\begingroup$ @Ute, you are totally correct. I put up a formula that I wrote incorrectly in my report. It should be: x_hat_plus = (xsigma_y^2 + ysigma_x^2)/(sigma_x^2 + sigma_y^2) I shall correct it now. $\endgroup$
    – kartiks77
    Aug 28, 2023 at 11:27
  • $\begingroup$ :-) - can you translate this into the language of the original question? The technique is called inverse variance weighting in statistics, and you can find a Wikipedia page on that. If you have time, you can perfectionate your answer $\endgroup$
    – Ute
    Aug 28, 2023 at 12:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.