3
$\begingroup$

I am studying Linear Discriminant Analysis (LDA). According to the formula for LDA, we are supposed to get the inverse of within group covariance. However, if $p\gg n$ (i.e., the dimension is much larger than the number of samples), what should I do?

$\endgroup$
3
$\begingroup$

One possibility is to regularize your estimate of the covariance matrix. See Regularization parameter to generate inverse covariance matrix.

Regularization uses assumed prior-information to turn the covariance inversion into a well-posed problem (this is also mentioned in the wikipedia article)

In practice, you can use a shrinkage estimator of the covariance matrix, such that:

$$\Sigma \rightarrow (\lambda-1)\Sigma+\lambda I_m$$

Where $\lambda$ is a hyperparameter that must be set in advance (and possibly optimized).

$\endgroup$
  • $\begingroup$ Thank you! That is a good strategy to get the inverse of a matrix. But in LDA, do people also use this method to solve this problem? $\endgroup$ – coolcat Oct 5 '18 at 17:50
  • $\begingroup$ @coolcat Yes, it's even mentioned in the article linked. There are other ways to achieve regularization though. I only presented an alternative. $\endgroup$ – Firebug Oct 5 '18 at 18:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.