Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

I have a set of points in a matrix of size 100 x 100(total 10000 points). I know that there are roughly 500 anomaly points in it. There is a corresponding truth file which contains the true anomalous points which is not available while building the algorithm. The goal is to maximize the F-measure of the anomalies returned. How do I approach this problem?

The dataset of all points

share|improve this question

2 Answers 2

up vote 1 down vote accepted

To find the outliers you could use an outlier detection algorithm like Local Outlier Factor. This algorithm computes a score for each data point, so that you could treat the 500 objects with the highest score as an outlier.

share|improve this answer

You can approach this as a binary classification problem (since you have class labels available). Use a ensemble classifier like RandomForest to build your model. You can use N folds cross-validation to check if your model is good. You might want to use techniques like boosting to improve accuracy

You need to extract some meaningful features on which normal and anomaly points differ and use them to train your model. This is the most important aspect.

If you don't get good F measure for anomalies, then try to bring the ratio of normal to anomaly points to around 5:1 using some resampling techniques (SMOTE, stratifiedSampling etc) and repeat the above process.

share|improve this answer
    
i dont have the truth file or labels. It is going to be used for evaluating the performance of my algorithm. So, I only have the data points that I have linked. The only thing I know is there are about 500 anomalous points. –  Pankaj More Nov 22 '12 at 8:58

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.