5
$\begingroup$

I will be running a machine learning anomaly detection algorithm on some data. It's a binary classification problem: each data point is either anomalous or normal. I have some known anomalous data that I can use in testing the features I select.

I will be using the F1 score for testing the features I've selected for this algorithm. What would be considered a good F1 score?

$\endgroup$
10
$\begingroup$

You may have guessed it already, but it depends on ...

  1. what the data allows you to achieve at maximum. Regarding this, please see Expected best performance possible on a data set
  2. the cost of a wrong prediction and the benefit of a detected anomaly. I recommend The Foundations of Cost-Sensitive Learning by Charles Elkan for creation of such a cost-benefit-matrix.

So in summary, as long as the F1-score is significantly better than a random classifier (or any other dummy approach) and the cost-benefit-calculation based upon the model allows the conclusion that it is useful in practice, the corresponding F1-score can be considered as good.

Edit: What is the F1-score of a random classifier ?

Let's say we have 2 classes $c_1,c_2$.

Let's denote the a priori class probabilties as $p(c_1),p(c_2)$, $p(c_1)+p(c_2)=1$, where $p(c_k)$=number of instances with class $c_k$ divdided by number of all instances.

If the random classifier is presented an instance, it selects a class with probablitiy $p_{random}(c_k)$. Two major options exist:

  • $p_{random}(c_k)=p(c_k)$. This classifier will maximize the overall accuracy.
  • $p_{random}(c_k)=\frac{1}{2}$. This classifier will maximize recall for a minor class (as it is in case of anomaly detection) for the cost that more instances of the major class are not caught (i.e. the recall of the major class decreases). That's why a cost-maxtrix is needed.

Since the random classifier assigns the class independently of the presented instance, the prior class probabilities and hence the precision remain the same. So the precision for class k is just $precision(c_k)=p(c_k)$

Using the same independence argument, Recall, which is ratio of items of class k which have been correctly classified, is just $p_{random}(c_k)$.

Then, the F1-score for class k is (reference)

$F_1(c_k)=2\frac{precision*recall}{precision + recall}$

=$2\frac{p(c_k)p_{random}(c_k)}{p(c_k)+p_{random}(c_k)}$

Example:

Let $p(anomaly)=0.01,p(\neg anomaly)=0.99$. Then

If $p_{random}(c_k)=p(c_k)$:

precision(anomaly)=0.01, recall(anomaly)=0.01, F1(anomaly)~0.01 (accuracy=0.9892)

If $p_{random}(c_k)=\frac{1}{2}$:

precision(anomaly)=0.01, recall(anomaly)=0.5, F1(anomaly)~0.01961 (accuracy=0.5)

I have "verified" the calculation of precision and recall using the following MC-simulation (sorry for the quality, my R is a little bit rusty).

size <- 10000
trials <- 10000

anomalyCount <- size*0.01
noAnomalyCount <- size-size*0.01
dat <- data.frame("id"=1:size,"class"=as.character(c(rep("anomaly",anomalyCount),rep("no_anomaly",noAnomalyCount))))

prandom <- 0.01

precisionValues <- rep(NA,trials)
recallValues <- rep(NA,trials)

for(i in 1:trials){
    predictions <- sample(c("anomaly","no_anomaly"),size,replace=T,prob=c(prandom,1-prandom))
    anomalyInds <- which(predictions =="anomaly")   

    hits <- sum(dat$class[anomalyInds] == predictions[anomalyInds])
    precisionValues[i] <- hits / length(anomalyInds)
    recallValues[i] <- hits/anomalyCount
}

which delivers for $p_{random}(anomaly)=0.5$

[1] "average precision 0.00998538701407172 with error +/- 1.00354527409206e-07"
[1] "average recall 0.499318 with error +/- 1.00354527409206e-07"

and for $p_{random}(anomaly)=p(anomaly)=0.01$

[1] "average precision 0.0100008820210962 with error +/- 9.88994082273052e-07"
[1] "average recall 0.009979 with error +/- 9.88994082273052e-07"
$\endgroup$
  • $\begingroup$ Steffen - What is the F-1 score of a random classifier supposed to be? (e.g. for AUC-ROC the performance of a random classifier should is supposed to be 0.5) $\endgroup$ – Josh Nov 17 '13 at 3:57
  • $\begingroup$ @Josh I have updated my answer. $\endgroup$ – steffen Nov 18 '13 at 13:26
  • $\begingroup$ Thanks @steffen. For $p_\text{random}(c_k) = 0.5$ , shouldn't precision(anomaly) be 0.5 and not 0.05? I presume that would also make F1 = 0.5, correct? $\endgroup$ – Josh Nov 18 '13 at 20:38
  • $\begingroup$ @Josh No. On the other hand, I have realized I have made a stupid mistake. The random classifier assigns a class randomly and independently of the presented instance. Independently means, that the average a priori class probabilities will remain the same. So in both cases the precision is the same. $\endgroup$ – steffen Nov 18 '13 at 22:56
  • 1
    $\begingroup$ I will run a Monte Carlo - Simulation tomorrow to verify that (boy, I am tired ;)). $\endgroup$ – steffen Nov 18 '13 at 23:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.