# Estimating parameters in multivariate classification resulting zero determinant sample covariance matrix

Newbie here typesetting my question, so excuse me if this don't work.

I am trying to give a bayesian classifier for a multivariate classification problem where input is assumed to have multivariate normal distribution. I choose to use a discriminant function defined as $\log (likelihood \times prior)$.

However, from the distribution,

$f(x \mid\mu,\Sigma) = (2\pi)^{-N \times d/2}\det(\Sigma)^{-N/2}exp[(-1/2)(x-\mu)^{\top}\Sigma^{-1}(x-\mu)]$

i encounter a term $-\log(det(S_i))$, where $S_i$ is my sample covariance matrix for a specific class i. Since my input actually represents a square image data, my $S_i$ discovers quite some correlation and resulting in $det(S_i)$ being zero. Then my discriminant function all turn $\infty$, which is disastrous for me.

I know there must be a lot of things go wrong here, anyone willling to help me out?

Replace $\log|S_i|$ by $\sum_{j=1}^{J} \log\lambda_j$ where $\lambda_1>\ldots>\lambda_p$ are the $p>J$ eigenvector of $S_i$ in decreasing order ($S_i$ is a $p$ by $p$ square matrix) and $\lambda_{J+1}$ is the first eigenvalue smaller than a numerical 0 threshold (say 1e-9).

As for the second term (e.g. --$(x-\mu)^{\top}\Sigma^{-1}(x-\mu)$--), replace $S_i$ by

$$\pmb P_{J}^{}\pmb D_J\pmb P_J'$$

where

$$\pmb D=\text{diag}(\lambda_1,\ldots,\lambda_{p})$$

and

$$S_i=\pmb P^{}\pmb D\pmb P'$$

is the eigenvector decomposition of $S_i$ (ordered in decreasing order of the $\lambda_j$'s) and $\pmb P_{J}$ is the matrix formed of the first $J$ columns of $\pmb P$ and $\pmb D_J$ is the diagonal matrix formed of the first $J$ rows and columns of $\pmb D$.

Note that for that $i$-th class, your estimates will no longer be affine equivariant (if you rescale the $\pmb X$ matrix the membership probabilities will change).

• How is it related to taking PCA transformation of $J$ most important components? Commented Sep 29, 2013 at 12:35
• Somewhat! (but $J$ is fixed by the data and will be typically larger than the number of most important component) Commented Sep 29, 2013 at 12:36
• thanks a looooot dude! @user603 modifying the code now :D
– Ray
Commented Sep 29, 2013 at 12:51
• @user603 I am still confused some how about the $P$. Do mean $A = x - \mu$ and take the first $J$ colums of $A$? That may perform like choosing $J$ features from the input. And for $D$ you give, is it the diagonal matrix of first $J$ largest eigenvalues?
– Ray
Commented Sep 29, 2013 at 13:35
• I'm not sure what is $A$. If $S_i$ is a covariance matrix, it is square symmetric and can be decomposed as $S_i=\pmb P\pmb D\pmb P'$ where $\pmb D$ is diagonal and $\pmb P$ is orthogonal. Commented Sep 29, 2013 at 14:14