1
$\begingroup$

I think I am confused with this thing.

If we have a 3 dimension Gaussian then the MLE estimate for $\mu$ is a vector with 3 element

$$\mu(1)' = \frac{1}{n}\sum_{j = 1} ^ n x_j\text{ and so on }\mu(2)\text{ for }y\text{ and }\mu(3)\text{ for }z$$

also $\sigma$

$$ \sigma(1)' = \sqrt{\frac{1}{n}\sum_{j = 1} ^ n(x_j - \mu')}\text{ and so on }\sigma(2)\text{ for }y\text{ and }\sigma(3)\text{ for }z$$

When we know that the $x,y,z$ are not independent, in which case the covarience matrix looks like this: $\sigma\cdot I$ where $\sigma$ is a number and $I$ an $3\times 3$ identity matrix?

How can we compute $\sigma$?

Thanks for the answers an

$\endgroup$
7
  • $\begingroup$ You seem to ignoring the covariances between the components. $\endgroup$
    – whuber
    Feb 14, 2013 at 19:23
  • $\begingroup$ yeah, this is what I can't understand, and why $\sigma%$ is a number, why not 3 length vector? thanks for your comment $\endgroup$
    – flatronka
    Feb 14, 2013 at 19:27
  • 1
    $\begingroup$ en.wikipedia.org/wiki/Multivariate_normal_distribution $\endgroup$
    – whuber
    Feb 14, 2013 at 19:37
  • 1
    $\begingroup$ That's right, because it's not: covariances among three random variables form a matrix. $\endgroup$
    – whuber
    Feb 14, 2013 at 19:48
  • 1
    $\begingroup$ $I\sigma$ where $I$ is an identity matrix and $\sigma$ is a standard deviation is not right; it should be $I\sigma^2$ since the diagonal entries in a covariance matrix are variances not standard deviations. More to the point, $I\sigma^2$ is the covariance matrix of three uncorrelated random variables with identical variances $\sigma^2$, and in this case, since you (seem to be) assuming joint normality, the random variables are independent as well. You "... can't see covariance like this..." because that is not the covariance matrix of dependent jointly normal random variables. $\endgroup$ Feb 14, 2013 at 20:27

1 Answer 1

1
$\begingroup$

There's lots of ways we can calculate sigma :) . I think you want the maximum likelihood one though, as that's how you've calculated the mean.

It's going to be a bit easier to look at this in terms of vectors. Let's concatenate our data into a set of vectors $v_1$, $v_2$, etc, up to $v_n$ where

$$ v_1 = [x_1, y_1, z_1]^T $$

i.e. we're just taking your three dimensional data and stacking it. Similarly, we have the mean $\mu' = [\mu(1)', \mu(2)', \mu(3)']^T$. In that case the log likelihood $L$ of your data is

$$ L = \sum_{i=1}^n \log \Bigg(\frac{1}{(2\pi\sigma^2)^{3/2}}e^{\frac{|| v_i - \mu' ||^2}{-2\sigma^2}} \Bigg) $$

If you're not sure why this is, look up multivariate gaussians, set the covariance matrix equal to $\sigma^2 I$ and the number of dimensions to $3$.

NB - the standard way to define an isotropic covariance matrix is by setting it equal to $\sigma^2 I$, and not $\sigma I$ as you have. Doing that will confuse anyone who reads your work at a later date, as it looks misleadingly like the usual notation. As such, you're going to have to be careful when you look at the result of this derivation, and make sure you're clear what the term $\sigma$ refers to in it before you start using it. Otherwise you may end up squaring / square rooting / etc incorrectly.

Okay. So, to make it easier to work with I'm going to expand the logarithm term. It's just some algebra, nothing fancy.

$$ L = -\frac{3n}{2}\log(2\pi\sigma^2) - \sum_{i=1}^n\frac{|| v_i - \mu' ||^2}{2\sigma^2}. $$

Now, to find the maximum likelihood result we differentiate by $\sigma$ and set the whole thing to zero, giving us this,

$$ 0 = -\frac{3n}{2}\frac{4\pi\sigma}{2\pi\sigma^2} + \sum_{i=1}^n\frac{|| v_i - \mu' ||^2}{\sigma^3}. $$

There's a whole bunch of stuff which cancels, and once we isolate the terms in $\sigma$ we're left with

$$ \sigma^2 = \frac{\sum_{i=1}^n|| v_i - \mu' ||^2}{3n} $$

which you'll notice is equal to

$$ \sigma^2 = \frac{1}{3}\big(\sigma(1)'^2 + \sigma(2)'^2 + \sigma(3)'^2\big) $$

i.e. the overall variance is equal to the mean variance in each dimension. Which makes a nice, intuitive sense.

That's the maximum likelihood result. However, like I said right at the start, it's not the only thing we can do. For example, this result is slightly biased - not a problem if $n$ is large, but if it's small you may want to investigate these. Also, you could find a bayesian result, which uses priors and will probably be different as a result. But for now I think this ought to give you what you're looking for.

$\endgroup$
1
  • $\begingroup$ Wow thanks for the answers. this is what I looking for. I think I have a new question, but I will ask that in a new topic. $\endgroup$
    – flatronka
    Feb 15, 2013 at 15:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.