1
$\begingroup$

First of all sorry but I am a little bit of a newbie to advanced stats hence why my question may sound silly.

I am trying to perform a classification task where I assume that my data is generated by a multivariate Gaussian distribution. For that purpose I estimate the covariance matrices from my sample data, but for some of the variables I am getting a non positive definite matrix. Could anyone provide me with a sound explanation to this (I understand I have some non linearly independent variables in my data probably?) and maybe a workaround so I can proceed?

$\endgroup$
8
  • 2
    $\begingroup$ Could you explain what you mean by "proceed"? What exactly is the matter with obtaining a singular covariance matrix? $\endgroup$
    – whuber
    Oct 11, 2016 at 16:24
  • $\begingroup$ Hi @whuber, in order to "proceed" with my classification task I need to estimate probabilities that depend on these estimated covariance matrices. What I mean is I am using a multivariate Gaussian to calculate the probabilities of an observation given each class and then just using the argmax of that to classify, hence why I need those covariance matrices to work for the probability density function... Any idea how I could work around my data to avoid singular covariance matrices? $\endgroup$
    – PL-RL
    Oct 11, 2016 at 23:16
  • $\begingroup$ It's unclear how the singularity of a covariance matrix could prevent you from estimating a probability. $\endgroup$
    – whuber
    Oct 12, 2016 at 12:50
  • $\begingroup$ Any data generated by a multivariate Gaussian function must have a positive definite covariance matrix. I am reversing that, and estimating the covariance matrices from my data set, having as a starting point the assumption that my data is Gaussian. If my covariance matrix is not positive definite, I cannot use it to calculate probabilities with a multivariate Gaussian. But maybe I am missing something in this approach...? $\endgroup$
    – PL-RL
    Oct 12, 2016 at 14:06
  • 1
    $\begingroup$ Any data generated by any distribution--including Gaussians--only need have a positive semidefinite covariance. You definitely can use your estimate to compute probabilities. After all, your covariance estimate (along with an estimated multivariate mean) completely determines the distribution! $\endgroup$
    – whuber
    Oct 12, 2016 at 14:14

1 Answer 1

1
$\begingroup$

You might have a singular multivariate normal distribution. See Multivariate normal with singular covariance. Such a singular distribution do not have a density (in the full space ... ) Can a multivariate distribution with a singular covariance matrix have a density function? But we can still work with it, for instance using the Moore-Penrose generalized inverse of the covariance matrix.

But, maybe the software you are using do not do so. If you tell us with more details what you want to do, we can tell you ...

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.