Suppose I'm building a logistic regression classifier that predicts whether someone is married or single. (1 = married, 0 = single) I want to choose a point on the precision-recall curve that gives me at least 75% precision, so I want to choose thresholds $t_1$ and $t_2$, so that:
- If the output of my classifier is greater than $t_1$, I output "married".
- If the output is below $t_2$, I output "single".
- If the output is in between, I output "I don't know".
A couple questions:
- I think under the standard definition of precision, precision will be measuring the precision of the married class alone (i.e., precision = # times I correctly predict married / total # times I predict married). However, what I really want to do is measure the overall precision (i.e., the total # times I correctly predict married or single / total # times I predict married or single). Is this an okay thing to do? If not, what should I be doing?
- Is there a way to calculate this "overall" precision/recall curve in R (e.g., using the ROCR package or some other library)? I'm currently using the ROCR package, but it seems to only give me the single-class-at-a-time precision/recall.