I'm working on a problem where I need to find the probability of a given data sample belonging to a give class using the Bayes Theorem for classification.
From everything I've been able to find so far the way to do this would be to first calculate the conditional probabilities using multivariate Gaussians (shown below), then apply Bayes Theorem to get the desired posterior probability.
$$P(x|k)=\frac{1}{\sqrt{(2\pi)^{p/2}|\boldsymbol\Sigma|^{1/2}}} \exp\left(-\frac{1}{2}({x}-{\mu_k})^T{\boldsymbol\Sigma}^{-1}({x}-{\mu_k}) \right)$$
The problem I'm having is that the Mahalanobis distance in the exponent term is consistently coming to extremely large values, making the conditionally probabilities far too small to work with programmatically. I've determined this is almost entirely due to the covariance matrix, which is populated with values that are incredibly small. Furthermore, when I just ignore the covariance matrix and solve using the Euclidean distance instead, I arrive at reasonable probability estimates with distances that appear to more accurately reflect what I would expect in classification.
So I was wondering:
- If there was a way to calculate these conditional probabilities or the overall posterior probability without direct exposure to the exponent, so they become feasible to work with programmatically.
- If there is any statistical justification for using the Euclidean distances over the Mahalanobis distance in this case.
[Edit] Here are a few more details regarding the derivation of the covariance matrix and the dataset:
The dataset I'm working with has been transformed into a lower dimensional space using the discriminant analysis (ULDA) technique described in this paper. In this form of discriminant analysis there is a shared pooled within-class covariance matrix corresponding to each class conditional probability (this being calculated over the reduced dimensional space). Interestingly, with this technique, if the total scatter matrix is used as the shared covariance matrix then the covariance matrix in the reduced space will be the identity matrix. This technique optimizes with respect to the total scatter matrix rather than the within-class scatter matrix, so maybe I'm misunderstanding what the covariance matrix should be in this case. Also, as mentioned in the paper, the ULDA technique produces features in the reduced space are uncorrelated to each other. I suppose another possibility would be that this particular dimensionality reduction reduces the usefulness of the within-class scatter matrix to the point that it gives worse results than just using the plain identity matrix as the covariance.
Any help or advice would be greatly appreciated, thanks in advance.