I am new to the topic, and I am trying to understand how it is possible to perform anomaly detection by using gaussian mixture models. Could you please give me some hints about literature on the topic?
Gaussian Mixture Models allow assigning a probability to each datapoint of beeing created by one of k gaussian distributions.
These are normalized to sum up to one, allowing interpretation as "Which cluster is most probably responsible for this datapoint?"
If you do not normalize, you have absolute probabilities which estimate how probable a point is - given a specific gaussian mixture model.
Then you can simply define an outlier such as: If p < 0.05 for each cluster, then the point is an outlier.
Yet be warned, the expectation maximization algorithm for gaussian mixture models - which you will need to get best parameters for your gaussian mixture model - is not very robust and tends to find suboptimal solution.
For reading more - especially unterstanding more - I recommend Bishop: Pattern Recognition and Machine Learning