I am trying to make a classifier that would predict whether an item is a "buy" (positive) or "not a buy" (negative). My dataset is ~65% positive examples, and ~35% negatives. It is more important for my model to keep false positives down (at the cost of reducing true positives). Let's say that the acceptable level of false positives is 5%. I am OK with my model missing a lot of "buy" opportunities, but when it recommends me to buy, I want to have a great degree of confidence (>95%) it actually is a buy.

I train my classifier, draw a ROC curve, and select the threshold such that false positives are less than 5% (which gives me true positive rate of 35%).

Now, am I correct in saying that a dummy classifier (i.e. one that ignores the features when making its prediction, but only looks at the distribution of positive and negative examples) will never be able to get true positive rate better than 5% if false positive rate is to be kept less than 5%? That is, its precision can never be better than 0.5, regardless of whether the dataset is balanced or unbalanced?

  • $\begingroup$ I think you mean 0.05 in the title not 0.5? $\endgroup$ – Aaron Mar 31 '17 at 17:34
  • $\begingroup$ No, 0.5 is correct - I am talking about precision there, which is TP / (TP + FP). Since the "dummy" ROC curve is just a straight diagonal line from (0, 0) to (1, 1), precision would supposedly be 0.5 everywhere. $\endgroup$ – rinspy Mar 31 '17 at 18:11
  • $\begingroup$ I'm just struggling to get the intuition here. The "threshold" for our dummy classifier would be the probability of classifying an example as positive, right? So why is it that for any threshold we choose, true positive rate will equal false positive rate - or is my understanding incorrect, and they will not necessarily be equal? $\endgroup$ – rinspy Mar 31 '17 at 18:14
  • $\begingroup$ Wouldn't the null model (aka dummy classifier, if I am reading you correctly) always output the prior probability (i.e. the fraction of positive classes in the training data)? $\endgroup$ – Adrian Mar 31 '17 at 18:53
  • $\begingroup$ Depends on how you define it. What I mean by a dummy classifier is one that does not look at the features, so it can output any prior probability. $\endgroup$ – rinspy Apr 1 '17 at 11:23

A naive classifier will classify each case as "positive" at random, with a constant probability $q$. For instance, it could classify everything as "positive" ($q=1$), or classify everything as "negative" ($q=0$), or anything in between.

Let's call your true prevalence of positives $p$. In your example, $p=0.65$. If we now classify using the naive classifier as above, we will get:

  • $TP=qp$ true positives
  • $TN=(1-q)(1-p)$ true negatives
  • $FP=q(1-p)$ false positives
  • $FN=(1-q)p$ false negatives

For instance, the $TP$ comes from the fact that the proportion of actual positives is $p$, and we classify a proportion of $q$ of these as "positive". The others are calculated similarly.

The precision then is


i.e., the true positive prevalence. In your case, any naive classifier will have a precision of $p=0.65$, more than $0.5$.

This assumes that we can cancel $q$, that is, that $q>0$. Happily enough, the false positive rate under this naive classifier is

$$ FPR = \frac{FP}{N}=\frac{q(1-p)}{1-p}=q, $$

so choosing any $q\leq 0.05$ will satisfy your criterion of no more than 5% false positive rate. Incidentally, the true positive rate is also

$$ TPR = \frac{TP}{P}=\frac{qp}{p}=q. $$

You may be interested in this: Why is accuracy not the best measure for assessing classification models?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.