Newbie here typesetting my question, so excuse me if this don't work.

I am trying to give a bayesian classifier for a multivariate classification problem where input is assumed to have multivariate normal distribution. I choose to use a discriminant function defined as $\log (likelihood \times prior)$.

However, from the distribution,

$f(x \mid\mu,\Sigma) = (2\pi)^{-N \times d/2}\det(\Sigma)^{-N/2}exp[(-1/2)(x-\mu)^{\top}\Sigma^{-1}(x-\mu)]$

i encounter a term $-\log(det(S_i))$, where $S_i$ is my sample covariance matrix for a specific class i. Since my input actually represents a square image data, my $S_i$ discovers quite some correlation and resulting in $det(S_i)$ being zero. Then my discriminant function all turn $\infty$, which is disastrous for me.

I know there must be a lot of things go wrong here, anyone willling to help me out?


Replace $\log|S_i|$ by $\sum_{j=1}^{J} \log\lambda_j$ where $\lambda_1>\ldots>\lambda_p$ are the $p>J$ eigenvector of $S_i$ in decreasing order ($S_i$ is a $p$ by $p$ square matrix) and $\lambda_{J+1}$ is the first eigenvalue smaller than a numerical 0 threshold (say 1e-9).

As for the second term (e.g. --$(x-\mu)^{\top}\Sigma^{-1}(x-\mu)$--), replace $S_i$ by

$$\pmb P_{J}^{}\pmb D_J\pmb P_J'$$


$$\pmb D=\text{diag}(\lambda_1,\ldots,\lambda_{p})$$


$$S_i=\pmb P^{}\pmb D\pmb P'$$

is the eigenvector decomposition of $S_i$ (ordered in decreasing order of the $\lambda_j$'s) and $\pmb P_{J}$ is the matrix formed of the first $J$ columns of $\pmb P$ and $\pmb D_J$ is the diagonal matrix formed of the first $J$ rows and columns of $\pmb D$.

Note that for that $i$-th class, your estimates will no longer be affine equivariant (if you rescale the $\pmb X$ matrix the membership probabilities will change).

  • 1
    $\begingroup$ How is it related to taking PCA transformation of $J$ most important components? $\endgroup$ – lejlot Sep 29 '13 at 12:35
  • $\begingroup$ Somewhat! (but $J$ is fixed by the data and will be typically larger than the number of most important component) $\endgroup$ – user603 Sep 29 '13 at 12:36
  • $\begingroup$ thanks a looooot dude! @user603 modifying the code now :D $\endgroup$ – Ray Sep 29 '13 at 12:51
  • $\begingroup$ @user603 I am still confused some how about the $P$. Do mean $A = x - \mu$ and take the first $J$ colums of $A$? That may perform like choosing $J$ features from the input. And for $D$ you give, is it the diagonal matrix of first $J$ largest eigenvalues? $\endgroup$ – Ray Sep 29 '13 at 13:35
  • $\begingroup$ I'm not sure what is $A$. If $S_i$ is a covariance matrix, it is square symmetric and can be decomposed as $S_i=\pmb P\pmb D\pmb P'$ where $\pmb D$ is diagonal and $\pmb P$ is orthogonal. $\endgroup$ – user603 Sep 29 '13 at 14:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.