Exercise: Should we trust our naive bayes or a human "guru"? This is from an exercise about the cost of missclassifications using a Naive Bayes classifier. We know that for some example which has three binary features $x_1,x_2,x_3$ the posterior is $P(Y = 1 | X_1 = x_1, X_2 = x_2, X_3 = x_3) = p$. There are two classes, y $\in$ {0,1}.The cost for misclassication is $c_m$. For each prediction we can either trust our naive bayes classifier, or ask a human expert with cost $c_h $ < $c_m$, who is 100% accurate. 
The question is which of the two procedures would  minimize the expected cost?
Edit: I think part of the confusion we are having is this: We have two possible outcomes. So normally, if y = 1 is true, but we decide on state y = 0, i.e. a = 0, the cost should be $p*c_m$. If in contrast  y = 0 is true, but we decide on state y = 1, i.e. a = 1, the cost should be $(p-1)*c_m$. Would choosing the optimal action a* not imply by itself then that we are deciding based on the higher posterior?
BUT, the the answer below suggests that we already should take into account that the classifier goes with the maximal posterior before searching for optimal action a*: In this scenario, the cost would always be $c_m*min(1-p,p)$. For instance, if p = 0.45, the Naive Bayes decides on y = 0 (because of the higher posterior for y=0), so the cost is $0.45*c_m$, but even if p = 0.55, the cost would be $0.45*c_m$ since now Naive Bayes is choosing y =1.  
 A: From my reading of the original post, the question boils down to "what is the relationship between $E[Cost_H]$ and $E[Cost_M]$?" i.e. the "expected cost of consulting the human" and the "expected cost of trusting the machine."
The first part of this is easy: the expected cost of consulting the human is the term $c_h$ given in the question. More formally:
$\begin{align}E[Cost_H] &= (Cost_H|Correct)\times P(Correct) + (Cost_H|Incorrect)\times P(Incorrect) \\ &=(c_h \times 1) + (c_h \times 0) \\ &=c_h \end{align}$
The second part is a bit more involved to derive, but is easiest to think of if we look at the confusion matrix which considers the true class, $Y$, against the predicted class, $y$.

Again we can derive an expression for the expected cost of trusting the machine (using the terms $a$, $b$, $c$, and $d$ from the confusion matrix to declutter):
$\begin{align}E[Cost_M] &= (Cost_M|Correct)\times P(Correct) + (Cost_M|Incorrect) \times P(Incorrect) \\ &= 0 \times P(Correct) + c_m \times P(Incorrect) \\ &=c_m \times(P(y=1|Y=0)P(Y=0)+P(y=0|Y=1)P(Y=1))\\ &=c_m \times ((c\times (1-p))+(b \times p))\end{align}$
Finally, as asked in our question, we compare the two expected costs. We can say that the trade off point between the two methods occurs when $E[Cost_H] = E[Cost_M]$:
$\begin{align}E[Cost_H] &= E[Cost_M] \\ c_h &=c_m \times ((c\times (1-p))+(b \times p)) \\ \frac{c_h}{c_m} &= c\times (1-p)+(b \times p) \end{align}$
So determining which approach will minimize the expected cost depends on $b$, the probability that the machine predicts class 0 given that the true class is 1, $c$, the probability that the machine predicts class 1 given that the true class is 0, and $p$, the probability of class 1 occurring. For example:
$\text{For the case where }p = (1-p) = 0.5 \text{ we have:}\\\text{If }\frac{c_h}{c_m} > \frac{c+b}{2} \text{, then trusting the machine minimizes the expected cost;}\\\text{If } \frac{c_h}{c_m} = \frac{c+b}{2} \text{, then the approaches have the same expected cost; and}\\\text{If }\frac{c_h}{c_m} < \frac{c+b}{2} \text{, then consulting the human minimizes the expected cost.}$
NB: If $p \neq (1-p)$ then the comparison is of the same form with the terms $c$ and $b$ appropriately weighted.
A: What we arrived at in chat: we assume the naive Bayes classifier goes with the maximum a posteriori class, which gives us a p = 0.5 cutoff in the two class case.
Suppose p > 0.5. Our model thus predicts class 1, and we expect to be wrong with probability 1 - p. So, expected cost of going with the classifier is the cost of misclassification times 1 - p. Similarly, the expected cost of going with the classifier when p < 0.5 is the cost of misclassification (same for either type of error in this problem) times p.
If the expected cost thus calculated is less than the cost of the guru, then go with the model's prediction. Otherwise, consult the guru. 
There is a simple equation which can express our decision rule no matter what p is. I'll leave this unanswered. Hint: p being further from 0.5 in either direction means we should be less likely to consult the guru. 
