2
$\begingroup$

There are two popular measures that aggregate $Precision$ and $Recall$, there are $F1$ and $\text{Precision Recall Breakeven point}$.

$F1$ can be calculated easily by formula, but how to calculate $\text{breakeven point}$?

I have some experiment and get all four values: true positive, false positive, true negative, false negative, so now I can calculate Precision and Recall but how to find breakeven point in this case.

$\endgroup$

2 Answers 2

3
$\begingroup$

There is an excellent post (Obtaining predicted values (Y=1 or 0) from a logistic regression model fit) about the break-even point of precision (or sensitivity) and specificity. The latter is not the same as recall, but it should be easy to generalize from there.

If you look at the plot you will see a point where the metrics cross, this is your optimal cutoff point.

EDIT I have updated the code to include precision, recall, and F1

perf = function(cut, mod, y)
{
     yhat = (mod$fit>cut)
     w = which(y==1)
     sensitivity = mean( yhat[w] == 1 ) 
     specificity = mean( yhat[-w] == 0 ) 
     c.rate = mean( y==yhat ) 
     d = cbind(sensitivity,specificity)-c(1,1)
     d = sqrt( d[1]^2 + d[2]^2 ) 

     # F-score
     retrieved <- sum(yhat)
     precision <- sum(yhat & y) / retrieved
     recall <- sum(yhat & y) / sum(y)
     Fmeasure <- 2 * precision * recall / (precision + recall)
     out = t(as.matrix(c(sensitivity, specificity, c.rate,d, Fmeasure)))
     colnames(out) = c("sensitivity", "specificity", "c.rate", "distance", "F-score")
     return(out)
} 

y3.mod <- glm(y3 ~ x1 + x2 + x3 + x4 + x5 + x6, family=binomial()) 

par(mfrow=c(1,1))
s = seq(.01,.99,length=100)
OUT = matrix(0,100,5)
for(i in 1:100) OUT[i,]=perf(s[i],y3.mod,y3)   
      plot(s,OUT[,1],xlab="Cutoff",ylab="Value",cex.lab=1.5,cex.axis=1.5,ylim=c(0,1),type="l",lwd=2,axes=FALSE,col=2)
axis(1,seq(0,1,length=5),seq(0,1,length=5),cex.lab=1.)
axis(2,seq(0,1,length=5),seq(0,1,length=5),cex.lab=1.)
lines(s,OUT[,2],col="darkgreen",lwd=2)
lines(s,OUT[,3],col=4,lwd=2)
lines(s,OUT[,4],col="darkred",lwd=2)
lines(s,OUT[,5],col="black",lwd=2)
grid()
box()
legend("topleft",col=c(2,"darkgreen",4,"darkred","black"),lwd=c(2,2,2,2,2),c("Sensitivity","Specificity","Classification Rate","Distance","F-score"))
$\endgroup$
3
  • $\begingroup$ Thank you very much for your answer, the problem is I had an experiment and got Precision, Recall, I have no idea what to do with the data in order to get a lot of Precision Recall value and find where there are equal. $\endgroup$
    – user16168
    Jun 12, 2014 at 17:54
  • $\begingroup$ If you only have 1 set of outputs then you will only have 1 precision, recall, and F1 score. The idea is that for each model you would compare these scores to determine which model is the best. $\endgroup$
    – mike1886
    Jun 12, 2014 at 18:24
  • 1
    $\begingroup$ In addition to comparing models, you can use these measures to compare decision thresholds applied to a single model in order to choose a threshold that balances errors based on their costs and the specifics of your problem. Maybe that is what the questioner wants to do? $\endgroup$
    – MattBagg
    Jun 12, 2014 at 22:26
0
$\begingroup$

Precision and recall measure the performance of a set of items which are predicted to be positive.

The break-even point measures the performance of a ranking of items which puts the items most likely to be positive at the top. You can take the top k items from this ranking and calculate the precision and recall of that set. Different values of k will give you different values of precision and recall. It is a fact that for some value of k, precision will equal recall, and this value of precision (and recall) is the break-even point.

So you need more information. You can't get the answer from just the precision and recall of the set of items which are predicted positive. You need to know the order of the items.

Once you have this, the break-even point is in fact equivalent to the measure called R-precision. This is the precision of the top R items in the ranking, where R is the number of relevant items. That's the way to calculate it.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.