# Exercise: Should we trust our naive bayes or a human "guru"?

This is from an exercise about the cost of missclassifications using a Naive Bayes classifier. We know that for some example which has three binary features $x_1,x_2,x_3$ the posterior is $P(Y = 1 | X_1 = x_1, X_2 = x_2, X_3 = x_3) = p$. There are two classes, y $\in$ {0,1}.The cost for misclassication is $c_m$. For each prediction we can either trust our naive bayes classifier, or ask a human expert with cost $c_h$ < $c_m$, who is 100% accurate.

The question is which of the two procedures would minimize the expected cost?

Edit: I think part of the confusion we are having is this: We have two possible outcomes. So normally, if y = 1 is true, but we decide on state y = 0, i.e. a = 0, the cost should be $p*c_m$. If in contrast y = 0 is true, but we decide on state y = 1, i.e. a = 1, the cost should be $(p-1)*c_m$. Would choosing the optimal action a* not imply by itself then that we are deciding based on the higher posterior?

BUT, the the answer below suggests that we already should take into account that the classifier goes with the maximal posterior before searching for optimal action a*: In this scenario, the cost would always be $c_m*min(1-p,p)$. For instance, if p = 0.45, the Naive Bayes decides on y = 0 (because of the higher posterior for y=0), so the cost is $0.45*c_m$, but even if p = 0.55, the cost would be $0.45*c_m$ since now Naive Bayes is choosing y =1.

• Can you explain your reasoning behind the expected cost for naive Bayes? Commented Aug 4, 2017 at 17:25
• The expected cost of the naive Bayes classifier should be the cost of misclassification times the probability of misclassification, which will expand out into the cost of misclassification times the sum of the probability of a false positive and the probability of a false negative. So two things: 1. These are not $p$ and $1-p$ in the equation above. 2. If instead we call the probability of a false positive $f$, it is not necessarily true that the probability of a false negative is $1-f$. So you won't necessarily get the canceling-out that you're seeing. Commented Aug 4, 2017 at 17:34
• Ahh..so you mean $p(y|x)((c_m)*(1-p)+p*(c_m))$? And there are only two classes y in this task! Then it would be true? If these are correct, I will edit shortly
– user24544
Commented Aug 4, 2017 at 17:37
• No. From your question's first paragraph, it seems like you are using $p$ to define the model's probability output for a given observation. This is not really related to the probability of misclassification, which is what should be in these expected cost equations. edit: of course, $p$ is related to the probability of misclassification insofar as the value of $p$ (combined with a cutoff threshold to make the model a classifier) determines the prediction of the model. But I think you're confusing the model's output with the predictive accuracy of that output. Commented Aug 4, 2017 at 17:40
• Ah I think I understand now: So you say that p is just the likelihood for a certain event, and not the probability for the posterior?
– user24544
Commented Aug 4, 2017 at 17:55

From my reading of the original post, the question boils down to "what is the relationship between $E[Cost_H]$ and $E[Cost_M]$?" i.e. the "expected cost of consulting the human" and the "expected cost of trusting the machine."

The first part of this is easy: the expected cost of consulting the human is the term $c_h$ given in the question. More formally:

\begin{align}E[Cost_H] &= (Cost_H|Correct)\times P(Correct) + (Cost_H|Incorrect)\times P(Incorrect) \\ &=(c_h \times 1) + (c_h \times 0) \\ &=c_h \end{align}

The second part is a bit more involved to derive, but is easiest to think of if we look at the confusion matrix which considers the true class, $Y$, against the predicted class, $y$.

Again we can derive an expression for the expected cost of trusting the machine (using the terms $a$, $b$, $c$, and $d$ from the confusion matrix to declutter):

\begin{align}E[Cost_M] &= (Cost_M|Correct)\times P(Correct) + (Cost_M|Incorrect) \times P(Incorrect) \\ &= 0 \times P(Correct) + c_m \times P(Incorrect) \\ &=c_m \times(P(y=1|Y=0)P(Y=0)+P(y=0|Y=1)P(Y=1))\\ &=c_m \times ((c\times (1-p))+(b \times p))\end{align}

Finally, as asked in our question, we compare the two expected costs. We can say that the trade off point between the two methods occurs when $E[Cost_H] = E[Cost_M]$:

\begin{align}E[Cost_H] &= E[Cost_M] \\ c_h &=c_m \times ((c\times (1-p))+(b \times p)) \\ \frac{c_h}{c_m} &= c\times (1-p)+(b \times p) \end{align}

So determining which approach will minimize the expected cost depends on $b$, the probability that the machine predicts class 0 given that the true class is 1, $c$, the probability that the machine predicts class 1 given that the true class is 0, and $p$, the probability of class 1 occurring. For example:

$\text{For the case where }p = (1-p) = 0.5 \text{ we have:}\\\text{If }\frac{c_h}{c_m} > \frac{c+b}{2} \text{, then trusting the machine minimizes the expected cost;}\\\text{If } \frac{c_h}{c_m} = \frac{c+b}{2} \text{, then the approaches have the same expected cost; and}\\\text{If }\frac{c_h}{c_m} < \frac{c+b}{2} \text{, then consulting the human minimizes the expected cost.}$

NB: If $p \neq (1-p)$ then the comparison is of the same form with the terms $c$ and $b$ appropriately weighted.

• Thanks for the beautiful answer. Why however do you include b and c? "the probability that the machine predicts class 0 given that the true class is 1" -- what is the reason you would not set that to p?
– user24544
Commented Aug 6, 2017 at 18:58
• @TestGuest Simply put, $b$ and $c$ are included as separate terms because they are different concepts than $p$. As a quick illustration, consider the case at the end of my answer where $p=(1-p)=0.5$. If $b$ and $c$ were forced to be equal to $p$, then this would mean that your classifier would be wrong half the time; it would have no discriminating power. A Naïve Bayes classifier is not equivalent to flipping a coin simply because the likelihoods of the two classes is the same, so the equivalencies $b=p$ and $c=p$ are "absurd" in the style of reductio ad absurdum. Commented Aug 6, 2017 at 19:54
• I was thinking along these terms at first, and I believe it makes sense if we can somehow know b and c. For instance, we could get estimates of b and c if we had a separate holdout test set against which to measure our model's performance. However, the question seems to want the answer only in terms of p. Given that p is our best estimate of the probability that Y = 1, it in itself gives us the model's confidence in the prediction. In other words, p = 0.75 implies that we have a 0.25 chance of a [c] error. Commented Aug 6, 2017 at 22:05

What we arrived at in chat: we assume the naive Bayes classifier goes with the maximum a posteriori class, which gives us a p = 0.5 cutoff in the two class case.

Suppose p > 0.5. Our model thus predicts class 1, and we expect to be wrong with probability 1 - p. So, expected cost of going with the classifier is the cost of misclassification times 1 - p. Similarly, the expected cost of going with the classifier when p < 0.5 is the cost of misclassification (same for either type of error in this problem) times p.

If the expected cost thus calculated is less than the cost of the guru, then go with the model's prediction. Otherwise, consult the guru.

There is a simple equation which can express our decision rule no matter what p is. I'll leave this unanswered. Hint: p being further from 0.5 in either direction means we should be less likely to consult the guru.