I have a system able to produce a ranking of some operations according to their anomalous score. I have N anomalous operations inside the dataset, which I expect to be in the first N positions (best case). This score is computed by a formula which has some parameters. According to the parameters configurations obviously the ranking changes. How can I measure the performance quality of the system? Currently, I am measuring the true positive rate, but I would prefer some measure that shows how much the anomalous operations are concentrated in the first positions. What I mean is that I can have the same TPR but in one case the FNs can be in N+1, ..., N+10 position, right after the Nth position, and another case where the FNs are at the end of the ranking. I definitely prefer the first case.
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
2
What you want is average precision, that you will compute on each element in the dataset, and then you will obtain mean average precision (MAP)
-
$\begingroup$ Thank you, I will try to use it. It seems to be exactly I was looking for! Instead, what do you think about DCG? Can it be also a good measure if I found a way to define the "relevance" variable? $\endgroup$ Commented Jan 5, 2015 at 9:44
-
$\begingroup$ Well, it could be, but really for your problem, the standard is MAP. $\endgroup$– jplCommented Jan 5, 2015 at 10:10