6
$\begingroup$

When implementing GMM (Gaussian Mixture Model) in practice, the covariance matrix ${\Sigma}_{D\times D}$ is often singular. The reason is that we have to estimate $\frac{D(D+1)}{2}$ parameters in $\Sigma$, which often causes overfitting.

The common solution is adding $\lambda I$ to $\widehat{\Sigma}$ (i.e.$\widehat{\Sigma}=\frac{1}{n}{\Sigma}_{i=1}^{n}((x_i-\mu)(x_i-\mu)^T)+\lambda I$), which comes from ridge regression. However, I failed to deduce ridge regression in estimation of covariance matrix of multivariate Gaussian distribution

I have tried below:

For linear models, we introduce least square function to estimate parameters, in case of overfitting (i.e. matrix singular), we adding constraint of $w$ (use its $l_{2}$ norm) :

$f(w)=\frac{1}{2}(Y-Xw)(Y-Xw)^T+\frac{\lambda}{2}w^Tw$

taking derivation of $w$ on both sides, we get $\widehat{w}=(X^TX+\lambda I)^{-1}X^TY$

However, for multivariate Gaussian distribution, we use MLE(maximum likelihood estimation) to estimate parameters. The cost function is the log-maximum likelihood:

$lnP(X,Y|\Sigma,\mu)= \frac{n}{2}ln|{\Sigma}^{-1}|-\frac{1}{2}{\Sigma}_{i=1}^{n}((x_i-\mu)^T{\Sigma}^{-1}(x_i-\mu))+Garbage$

($Garbage$ doesn't contain $\Sigma$)

Follow the constraint of $w$ in linear models, I add constraint of $\Sigma$, which I assumes to be $\frac{1}{2}tr(\Sigma ^T\Sigma)$, then the MLE becomes:

$lnP(X,Y|\Sigma,\mu)= \frac{n}{2}ln|{\Sigma}^{-1}|-\frac{1}{2}{\Sigma}_{i=1}^{n}((x_i-\mu)^T{\Sigma}^{-1}(x_i-\mu))+\frac{1}{2}tr(\Sigma ^T\Sigma)+Garbage$

However, after taking derivation of $\Sigma ^{-1}$ on both sides, I failed to get the expected form of $\widehat{\Sigma}=\frac{1}{n}{\Sigma}_{i=1}^{n}((x_i-\mu)(x_i-\mu)^T)+\lambda I$

Could someone give me some help?

$\endgroup$
1
$\begingroup$

It is beyond my current knowledge to present a detailed answer. However, I hope that the following resources will help you figure out the solution. Check this set of presentation slides. Take a look at page 9 of this document: see references to Thisted (1976) and to Brown and Zidek (1980) before equation 18. Finally, this more recent document on ridge regression might provide additional ideas.

$\endgroup$
  • 1
    $\begingroup$ Thanks Alex. I've read the slides in detail and I found that Gaussian Distributions mentioned in the slides refers to the distribution of the error of linear model (i.e. $e=Y-Xw$ is the Gaussian noise). It is different in GMM that we assume data points are pre-sampled from potential Gaussian Distributions. However, the slides provide a probabilistic view of ridge regression, which is very interesting. I'll read the other two documents you provide and see if they can help. Many thanks! $\endgroup$ – zodiac Mar 23 '15 at 3:27
  • $\begingroup$ @zodiac: You're very welcome! Thank you for the kind words. I'm glad that you found my answer helpful. I'm quite impressed with your mastery of math - the question is nicely formulated (+1). $\endgroup$ – Aleksandr Blekh Mar 23 '15 at 5:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.