Anomaly Detection in a set of points

Question

I have a set of points in a matrix of size 100 x 100(total 10000 points). I know that there are roughly 500 anomaly points in it. There is a corresponding truth file which contains the true anomalous points which is not available while building the algorithm. The goal is to maximize the F-measure of the anomalies returned. How do I approach this problem?

The dataset of all points

Answer 1

To find the outliers you could use an outlier detection algorithm like Local Outlier Factor. This algorithm computes a score for each data point, so that you could treat the 500 objects with the highest score as an outlier.

Answer 2

You can approach this as a binary classification problem (since you have class labels available). Use a ensemble classifier like RandomForest to build your model. You can use N folds cross-validation to check if your model is good. You might want to use techniques like boosting to improve accuracy

You need to extract some meaningful features on which normal and anomaly points differ and use them to train your model. This is the most important aspect.

If you don't get good F measure for anomalies, then try to bring the ratio of normal to anomaly points to around 5:1 using some resampling techniques (SMOTE, stratifiedSampling etc) and repeat the above process.