0
$\begingroup$

I use Gaussian mixture model (GMM) to infer probability density of multidimensional data written as: $p(x) = \sum_{j=1}^{K}\pi_j*N(x|\bf \mu_j, \Sigma_j)$, where $K$ is a number of mixtures, $\pi_j$ are their mixing coefficients and $\bf \mu_j, \Sigma_j$ are corresponding mean vectors and covariance matrices. These parameters are calculated as a result of GMM.

Then I want to check similarity between a new testing data batch and the training data. I thought, that using fitted GMM, I could define some probability density threshold. Note that this would of course result in some outliers in the training data, but these would be the false ones since GMM assumes no outliers (in other words, all the data is generated by underlying Gaussian constituents). Then again using fitted GMM, one can compute probability density for all points in a testing data batch. If it has too many points (how many - one has to define yet another threshold) falling below the density threshold then this data batch can be flagged.

My problem is that I don't know how to define the probability density threshold. One can do this based on 5% percentile for an individual multivariate Gaussian, but this seems to be a challenge for the mixture of Gaussians. Is there a way to do it or alternatively any other strategy to approach the task?

$\endgroup$
6
  • $\begingroup$ I don't understand how outlier identification is related to "check similarity between a new testing data batch and the training data". Are you saying you want to find out whether a new observation is an outlier with respect to the training data? What would be the consequences if you declare an observation "outlier"? Also note that generally there is no objective rule how to choose thresholds. For example whether you want 5% or 1% or 0.1% or whatever is entirely your choice. $\endgroup$ Nov 15, 2023 at 17:36
  • $\begingroup$ Obviously you can compute the density value of the new observation and compare it to low percentiles of the training data. 5% seems a lot though; 5% training data wrongly declared as outliers looks a lot to me; a "5% outlier" isn't really very surprising. $\endgroup$ Nov 15, 2023 at 17:38
  • $\begingroup$ @ChristianHennig, regarding your first comment: well, the heuristic here is that if unexpectedly large number of testing data points has low probability density then the generating function responsible for testing data is not similar to the one of the training data. $\endgroup$
    – baronett
    Nov 15, 2023 at 19:10
  • $\begingroup$ @ChristianHennig, regarding your second reply: I doubt that defining the density threshold based on the percentile of the training data is a good criteria. If the training points are not sampled densely enough then you would overestimate the density threshold. In other words, percentile for the sampled data is not the same as percentile for the probability density $\endgroup$
    – baronett
    Nov 15, 2023 at 19:13
  • $\begingroup$ Why wouldn't you look for general methods comparing/testing the difference between two multivariate distributions? These can use more information than just an outlier classification (which, as already elaborated, will depend on tuning decisions). $\endgroup$ Nov 15, 2023 at 23:08

1 Answer 1

0
$\begingroup$

The fact that all the data is generated by underlying components does NOT mean there will be no outliers. It means that outliers will be surprising. But that's what outliers are. If you run any generating function you will get outliers and, if you define outliers by an arbitrary threshold, then you can give a good estimate of how many outliers you will get.

Even in a univariate normal distribution, if you arbitrarily say that some values are outliers and delete them, you then have a truncated normal. It's worse in multivariate cases, even if all the variables are normally distributed and independent. This is because the volume of a p-dimensional unit hypercube grows much faster than a p-dimensional unit hypersphere, as p increases.

$\endgroup$
2
  • $\begingroup$ I think there is a miscommunication. I don't say there will be no outliers, rather GMM fit is assumed to describe the underlying processes well. Otherwise, we indeed have too many surprising points and GMM doesn't make sense. Regarding the answer may I clarify: are you suggesting just to run the generating function to densely sample the space and then claculate the density threshold based on the percentile of generated data? $\endgroup$
    – baronett
    Nov 15, 2023 at 12:48
  • $\begingroup$ Then I am confused. Maybe someone else will give a more helpful answer. But I suggest that, especially, in a high dimensional space, it is extremely hard to identify outliers. $\endgroup$
    – Peter Flom
    Nov 15, 2023 at 14:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.