0
$\begingroup$

I have a system able to produce a ranking of some operations according to their anomalous score. I have N anomalous operations inside the dataset, which I expect to be in the first N positions (best case). This score is computed by a formula which has some parameters. According to the parameters configurations obviously the ranking changes. How can I measure the performance quality of the system? Currently, I am measuring the true positive rate, but I would prefer some measure that shows how much the anomalous operations are concentrated in the first positions. What I mean is that I can have the same TPR but in one case the FNs can be in N+1, ..., N+10 position, right after the Nth position, and another case where the FNs are at the end of the ranking. I definitely prefer the first case.

$\endgroup$
0
$\begingroup$

What you want is average precision, that you will compute on each element in the dataset, and then you will obtain mean average precision (MAP)

$\endgroup$
2
  • $\begingroup$ Thank you, I will try to use it. It seems to be exactly I was looking for! Instead, what do you think about DCG? Can it be also a good measure if I found a way to define the "relevance" variable? $\endgroup$ Jan 5 '15 at 9:44
  • $\begingroup$ Well, it could be, but really for your problem, the standard is MAP. $\endgroup$
    – jpl
    Jan 5 '15 at 10:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.