Suppose I have a number of samples drawn from a normal distribution $x_i \sim \mathcal{N}(\mu,C)$ with $i = 1 \dots n$. I can make observations $z_i = x_i + e_i$ for those samples which are perturbed by noise $e_i$ of known characteristic $e_i \sim \mathcal{N}(0,\Sigma_i)$. Note that the distribution of the observation noise is different for each $z_i$.

I would like to estimate $\mu$ and $C$ given the $z_i$.

Since $e_i$ is unbiased $\mu$ should just be $\mathbb{E}[z_i]$. In case $\Sigma_i = D$ for all $i$, $C = \operatorname{Var}(z_i) - D$ assuming independence between the noise. Would it be safe to say that $C = \operatorname{Var}(z_i) - \mathbb{E}[\Sigma_i]$ in the general case?

  • $\begingroup$ (1) What does $\mathbb{E}[\Sigma_i]$ mean when it is stipulated that $\Sigma_i$ is a "known characteristic"? Are you suggesting there is some prior distribution involved? (2) Because there is no guarantee that $\text{Var}(z_i) - \mathbb{E}[\Sigma_i]$ is positive definite, you might want to worry a bit about the potential inadmissibility or unrealism of your estimator. $\endgroup$ – whuber Feb 19 '13 at 21:28
  • $\begingroup$ (1) $\mathbb{E}[\Sigma_i] = 1/n \sum_{i=1..n} \Sigma_i$ (2) The same would be true for when $\Sigma_i = D$ for all $i$ as well. How do I get around this problem? And yes, I worry about the admissibility of the estimator, hence my question :) $\endgroup$ – Jakob Feb 19 '13 at 21:51
  • $\begingroup$ If $\Sigma_i$ is known, and $\Sigma_i = L_i L_i'$, if you let $x_i^* = L_i^{-1} x$ (and similarly for $z$ and $e$), then depending on how you tend to look at things that may help a bit. $\endgroup$ – Glen_b Feb 19 '13 at 22:06
  • $\begingroup$ @whuber I've looked at your answer to a similar problem, basically performing a weighting of the samples for the estimators. Which I guess is similar to what Glen_b is suggesting, right? I can't seem to derive an good estimator for $C$ based on the samples though. $\endgroup$ – Jakob Feb 20 '13 at 11:30
  • 1
    $\begingroup$ @whuber Ah sorry. Currently I am happy with $x_i$ being univariate, but I think I might have to extend it to the multivariate case later. I did an update to the notation to make it more clear, but reverted it again since then its out of line with the comments. $\endgroup$ – Jakob Feb 20 '13 at 19:46

I think I've figured it out now. As stated in the comments, I think there is indeed a problem with the estimators. I found another answer on using a linear estimator for a similar problem. If I didn't make any mistakes, the estimator for $\mu$ in my case should be $$\hat{\mu} = \frac{ \sum_{i=1..n} z_i/(C+\Sigma_i)}{ \sum_{i=1..n} 1/(C+\Sigma_i)}$$ Since I don't actually know $C$, I guess I could use the estimate $\hat{C}$ based on the samples. Based on the Wikipedia Article on weighted mean and then subtracting the covariance influence from the $\Sigma_i$, I can estimate the covariance as $$\hat{C} = \left(\frac{\sum w_i}{(\sum w_i)^2 - \sum w_i^2} \sum w_i(z_i - \hat{\mu}) \right) - \frac{n}{\sum w_i}$$ where $w_i = 1/\Sigma_i$.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.