# Ridge regression in multivariate Gaussian distribution

When implementing GMM (Gaussian Mixture Model) in practice, the covariance matrix ${\Sigma}_{D\times D}$ is often singular. The reason is that we have to estimate $\frac{D(D+1)}{2}$ parameters in $\Sigma$, which often causes overfitting.

The common solution is adding $\lambda I$ to $\widehat{\Sigma}$ (i.e.$\widehat{\Sigma}=\frac{1}{n}{\Sigma}_{i=1}^{n}((x_i-\mu)(x_i-\mu)^T)+\lambda I$), which comes from ridge regression. However, I failed to deduce ridge regression in estimation of covariance matrix of multivariate Gaussian distribution

I have tried below:

For linear models, we introduce least square function to estimate parameters, in case of overfitting (i.e. matrix singular), we adding constraint of $w$ (use its $l_{2}$ norm) :

$f(w)=\frac{1}{2}(Y-Xw)(Y-Xw)^T+\frac{\lambda}{2}w^Tw$

taking derivation of $w$ on both sides, we get $\widehat{w}=(X^TX+\lambda I)^{-1}X^TY$

However, for multivariate Gaussian distribution, we use MLE(maximum likelihood estimation) to estimate parameters. The cost function is the log-maximum likelihood:

$lnP(X,Y|\Sigma,\mu)= \frac{n}{2}ln|{\Sigma}^{-1}|-\frac{1}{2}{\Sigma}_{i=1}^{n}((x_i-\mu)^T{\Sigma}^{-1}(x_i-\mu))+Garbage$

($Garbage$ doesn't contain $\Sigma$)

Follow the constraint of $w$ in linear models, I add constraint of $\Sigma$, which I assumes to be $\frac{1}{2}tr(\Sigma ^T\Sigma)$, then the MLE becomes:

$lnP(X,Y|\Sigma,\mu)= \frac{n}{2}ln|{\Sigma}^{-1}|-\frac{1}{2}{\Sigma}_{i=1}^{n}((x_i-\mu)^T{\Sigma}^{-1}(x_i-\mu))+\frac{1}{2}tr(\Sigma ^T\Sigma)+Garbage$

However, after taking derivation of $\Sigma ^{-1}$ on both sides, I failed to get the expected form of $\widehat{\Sigma}=\frac{1}{n}{\Sigma}_{i=1}^{n}((x_i-\mu)(x_i-\mu)^T)+\lambda I$

Could someone give me some help?

• Thanks Alex. I've read the slides in detail and I found that Gaussian Distributions mentioned in the slides refers to the distribution of the error of linear model (i.e. $e=Y-Xw$ is the Gaussian noise). It is different in GMM that we assume data points are pre-sampled from potential Gaussian Distributions. However, the slides provide a probabilistic view of ridge regression, which is very interesting. I'll read the other two documents you provide and see if they can help. Many thanks! – zodiac Mar 23 '15 at 3:27