My core problem is to set a cutoff to my one dimension data between normal with abnormal. I think this is a 'anomaly detection' problem.
My data is one dimension, consists with below:
mdifferent gaussian distributions with small mu,
m= 1~3 in most case. Fig.1 shows two normal gaussian distributions as red color.
ndifferent unknown distributions(mainly gaussian i think) with large mu,
nis unknown. They can't see in Fig.1 because sometimes they are too small, maybe only few points.
- (Abnormal) Noise points.
I want to set a cutoff (on x axis) to separate normal and abnormal.
After searching around and asking around, my solutions comes below:
Find all normal data gaussian distributions and use the max
mu + 3 sigma value as cutoff(3 sigma rules).
- Firstly, i use some outlier detection methods to remove most abnormal points, then the rest data is mainly normal.
- Then use KDE recognize how many peaks the rest data has. And use this value as the number of gaussian distributions in normal data.
- At last, use GMM to fit the rest data, and get the max
mu + 3 sigmavalue.
My problems mainly at step 1. It doesn't work well. I have tried
a) LOF (not stable, sometime it recognizes 99% points as outlier).
b) DBSCAN (hardly to cluster normal data as few clusters).
c) GMM, of course (use 2 gaussian distributions to separate normal and abnormal. It works well when
m = 1, but other cases failed).