Skip to main content
67 votes
Accepted

Maximum Likelihood Estimators - Multivariate Gaussian

Deriving the Maximum Likelihood Estimators Assume that we have $m$ random vectors, each of size $p$: $\mathbf{X^{(1)}, X^{(2)}, \dotsc, X^{(m)}}$ where each random vectors can be interpreted as an ...
Xavier Bourret Sicotte's user avatar
30 votes

Deriving the conditional distributions of a multivariate normal distribution

The answer by Macro is great, but here is an even simpler way that does not require you to use any outside theorem asserting the conditional distribution. It involves writing the Mahalanobis distance ...
Ben's user avatar
  • 129k
18 votes

Understanding degenerate multivariate normal distribution

A joint density function, say of two random variables $X$ and $Y$, is $f_{X,Y}(x,y)$ is an ordinary function of two real variables and the meaning that we ascribe to it is that if $\mathcal B$ is a ...
Dilip Sarwate's user avatar
16 votes
Accepted

Deriving the normalizing constant for the multivariate Gaussian

The reason that $\int_{\mathcal{R}^d} e^{-\frac{1}{2} \sum_i \lambda_i y_i^2} dy$ is easier to integrate is that it can be expressed as a product of univariate integrals: \begin{align} \int_{\mathcal{...
Elizabeth Santorella's user avatar
14 votes
Accepted

Quadratic form and Chi-squared distribution

In general, the quadratic form is a weighted sum of $\chi_1^2$ It is not true in general that $\mathbf{z}^\text{T} \mathbf{\Sigma} \mathbf{z} \sim \chi^2_p$ for any symmetric positive-definite (...
Ben's user avatar
  • 129k
14 votes
Accepted

How to calculate the probability of a data point belonging to a multivariate normal distribution?

Yeah, that sounds right. If you have parameters $\mu$ and $\Sigma$ and data point $x$, then the set of all data points that are less likely than $x$ are the ones that have less density, or in other ...
Taylor's user avatar
  • 21.2k
14 votes

Maximum Likelihood Estimators - Multivariate Gaussian

An alternate proof for $\widehat{\Sigma}$ that takes the derivative with respect to $\Sigma$ directly: Picking up with the log-likelihood as above: \begin{eqnarray} \ell(\mu, \Sigma) &=& ...
Eric Kightley's user avatar
13 votes
Accepted

Why are contours of a multivariate Gaussian distribution elliptical?

You can understand the shape of the ellipsoid better if you look at the spectral/eigen decomposition of the precision matrix (inverse of the covariance matrix). You want to look at the eigenvalues of ...
Taylor's user avatar
  • 21.2k
13 votes

Generalized Chi-Squared Distribution PDF

Your question is really a special case of https://math.stackexchange.com/questions/442472/sum-of-squares-of-dependent-gaussian-random-variables/442916#442916 (with $A=I$). But nevertheless: $X$ is ...
kjetil b halvorsen's user avatar
13 votes
Accepted

What is the geometric relationship between the covariance matrix and the inverse of the covariance matrix?

Before I answer your questions, allow me to share how I think about covariance and precision matrices. Covariance matrices have a special structure: they are positive semi-definite (PSD), which means ...
PAF's user avatar
  • 605
12 votes
Accepted

Why does the resulting matrix from Cholesky decomposition of a covariance matrix when multiplied by its transpose not give back the covariance matrix?

As explained in my comment, the inconvenient truth is that the Cholesky decomposition while usually defined as $K=LL^T$ where $L$ is lower triangular, is equally valid as $K=U^TU$ where $U$ is upper ...
usεr11852's user avatar
12 votes
Accepted

How to determine if a sample is within a standard deviation of a multivariate normal distribution

The usual metric to use here is the scaled Mahalanobis distance: $$S(\mathbf{x}) \equiv \frac{D(\mathbf{x})}{\sqrt{n}} = \sqrt{\frac{(\mathbf{x} - \boldsymbol{\mu})^\text{T} \mathbf{\Sigma}^{-1} (\...
Ben's user avatar
  • 129k
10 votes

Generate normally distributed random numbers with non positive-definite covariance matrix

Solution Method A: If C is not symmetric, then symmetrize it. D <-- $0.5(C + C^T)$ Add a multiple of the Identity matrix to the symmetrized C sufficient to make it positive definite with whatever ...
Mark L. Stone's user avatar
10 votes
Accepted

Finding $\mathbb E(Y_1^2Y_2^2)$ when $(Y_1,Y_2)$ is normal

Since all Gaussian variables have a kurtosis of $3,$ any Gaussian variable with variance $\sigma^2$ has a central fourth moment of $3\sigma^4.$ In particular, when $Z$ is a zero-mean Gaussian, $$E[Z^...
whuber's user avatar
  • 329k
9 votes
Accepted

Sufficient statistic for bivariate or multivariate normal

As W. Huber tried to lead you to conclude, the sufficiency is a simple consequence of looking at the likelihood: \begin{align*} f(\mathbf{x}_1,\ldots,\mathbf{x}_n|\boldsymbol{\mu},\boldsymbol{\Sigma}) ...
Xi'an's user avatar
  • 107k
9 votes

Generate normally distributed random numbers with non positive-definite covariance matrix

The question concerns how to generate random variates from a multivariate Normal distribution with a (possibly) singular covariance matrix $\mathbb{C}$. This answer explains one way that will work ...
whuber's user avatar
  • 329k
9 votes
Accepted

MLE of multivariate normal distribution with diagonal covariance matrix

This is not an answer to your question, but a hint that should make the answer fairly obvious. If $\Sigma$ is diagonal, then the likelihood function factors into a product of $p$ densities (one could ...
dsaxton's user avatar
  • 12.2k
9 votes
Accepted

If $X \sim N(0,1)$, does $(X, X)^\prime$ have a bivariate normal distribution?

Two copies of the same normal variable stacked up in a vector yield a degenerate bivariate normal. See wikipedia on the degenerate case of the multivariate normal While it is a special case of the ...
Glen_b's user avatar
  • 286k
9 votes

Why is Gaussian Copula's Tail Dependence Zero?

For a non-technical, intuitive view of what the tail index is telling you, we can look at simulation and compute sample estimates of the quantity $P[F(Y) > q | F(X) > q]$ as $q$ increases. Here ...
Glen_b's user avatar
  • 286k
9 votes
Accepted

Need help to understand Maximum Likelihood Estimation for multivariate normal distribution?

It's just linear algebra. Let's take a concrete example where $n=3$. Then ${\bf \mu} \in \mathbb{R}^{3 \times 1}$. $$\Sigma_{MLE} = \dfrac{1}{3}\sum_{i=1}^3(x_i-{\bf \mu})(x_i-{\bf \mu})^T = \frac13 ...
ilanman's user avatar
  • 4,787
9 votes
Accepted

Prove that $\mathrm{Cov}(x^TAx,x^TBx) = 2 \mathrm{Tr}(A \Sigma B \Sigma) + 4 \mu^TA \Sigma B \mu$

We have \begin{align} \operatorname{Var}(x^T(A+B)x)&=\operatorname{Var}(x^TAx+x^TBx) \\&=\operatorname{Var}(x^TAx)+\operatorname{Var}(x^TBx)+2\operatorname{Cov}(x^TAx,x^TBx) \end{align} So, $$\...
StubbornAtom's user avatar
  • 11.4k
9 votes
Accepted

Expected value of the outer product of normalized, non-centered Gaussian vector

After some work I found the solution to the problem. The solution is the following: Eq (1). \begin{equation} \mathbb{E}\left[\frac{XX^T}{X^TX}\right] = \frac{1}{n}{}_{1}F_1\left(1; \frac{n}{2}+1; \...
dherrera's user avatar
  • 1,268
9 votes

Variance Among Coordinates of Multivariate Normal

Based on your comment, you question can be clarified as follows: If $(X_1, \ldots, X_n) \sim N_n(0, \Sigma)$, what is the expected value of the sample variance (before bias correction) \begin{align} ...
Zhanxiong's user avatar
  • 20.8k
9 votes

Finding $\mathbb E(Y_1^2Y_2^2)$ when $(Y_1,Y_2)$ is normal

You can use the conditional distributions, which are univariate normal. For example, $Y_2$ given $Y_1$ is normal with mean $$E\left[Y_2\mid Y_1\right]=\frac{\sigma_{12}}{\sigma_{11}} Y_1$$ and ...
StubbornAtom's user avatar
  • 11.4k
9 votes

Finding $\mathbb E(Y_1^2Y_2^2)$ when $(Y_1,Y_2)$ is normal

Use the Characteristic function. It is well known that, $$\varphi_Y(s) := \mathbb E\left[e^{is^\intercal Y}\right] = e^{-\frac12s^\intercal \Sigma s}.$$ So \begin{align} \frac{\partial^2 \varphi_Y}{\...
Kroki's user avatar
  • 280
8 votes

Is Gaussian Process just a Multivariate Gaussian Distribution?

The multivariate Gaussian distribution is a distribution that describes the behaviour of a finite (or at least countable) random vector. Contrarily, a Gaussian process is a stochastic process defined ...
Ben's user avatar
  • 129k
8 votes
Accepted

Does assumption of normality of each mixture components implies that each margins is normal

Multivariate Gaussian mixtures are not themselves multivariate Gaussian, their components are. Statements that apply to the components of a mixture don't generally apply to the mixture (this would ...
Glen_b's user avatar
  • 286k
8 votes
Accepted

Transform general multivariate normal to standard multivariate normal

With the help of the content here, I was able to work this out and thought I'd post the resulting code and verification: ...
Mike Lawrence's user avatar
8 votes
Accepted

Multivariate normal distribution vs. sampling multiple times from univariate normal distribution

What you propose ignores correlations. When marginals have correlation (say positive for now), then when one is high, the others will tend to be high. Example Marginals are $X,Y\sim N(0,1)$. Draw from ...
Dave's user avatar
  • 65.1k
7 votes

Conceptual proof that conditional of a multivariate Gaussian is multivariate Gaussian

The device used in the answer that you cite will also get you the conditional distribution. Here is a self-contained derivation with a slight change in notation. Partition the column vector $X:=(X_1, ...
grand_chat's user avatar
  • 2,862

Only top scored, non community-wiki answers of a minimum length are eligible