Minimize mis-classification - 0 - 1 output

I am studying logistic regression from the book Advanced Data Analysis from an Elementary Point of View which states the following on page 280:

“We minimize the mis-classification rate by predicting whichever class is more likely”:

Let $\hat Y(x)$ be our predicted class, either $0$ or $1$. Our error rate is then $P(Y \neq \hat Y)$ . Show that $P(Y \neq \hat Y) = E[(Y −\hat Y)^2]$ .

Further show that $E[(Y− \hat Y)^2|X=x] = Pr(Y=1|X=x)(1−2\hat Y(x))+\hat Y^2(x)$.

Conclude by showing that if $Pr(Y = 1|X = x ) > 0.5$, the risk of mis-classification is mini- mized by taking $\hat Y = 1$, that if $Pr(Y = 1|X = x) < 0.5$ the risk is minimized by taking $\hat Y = 0$, and that when $Pr(Y = 1|X = x) = 0.5$ both predictions are equally risky.

I am a bit lost by this approach, as I have seen the derivations for the Bayes classifier using $L_2$ loss in the continuous case, and using 0-1 loss in the discrete case in ESL (page 18), but nothing of this sort before.

How should one approach this problem ?

• You should start by expanding out the expectation on the right-hand side of the first equation as an explicit sum over all the possible values of $(Y-\hat{Y})^2$. (Hint: this sum only has two terms in it.) Jun 25 '18 at 22:25
• isn't that for squared loss ? I am confused by the fact that this seems to be a 0 - 1 output variable, can we apply squared loss in such a case ? Jun 26 '18 at 12:23
• Indeed that is part of the purpose of this exercise: to show that in the case of a binary target variable, mean squared error is equivalent to the misclassification rate. Jun 26 '18 at 20:12
• @tddevlin Thanks for your hints it helped a lot, is this correct ? Jun 27 '18 at 11:17

An attempt following hints by @tddevlin. Since $Y, \hat Y$ can only take values $\{ 0,1\}$ the algebra is considerably simplified

\begin{aligned} E[(Y - \hat Y)^2] &= \sum (Y - \hat Y)^2 Pr((Y-\hat Y)^2) \\ & = 1 \times Pr( \ (Y-\hat Y)^2 = 1) + 0 \times Pr( \ (Y-\hat Y)^2 = 0) \\ & = Pr( \ (Y = 0, \hat Y = 1) \ \cup \ (Y = 1, \hat Y = 0) \ ) \\ & = Pr(Y \neq \hat Y) \end{aligned}

\begin{aligned} E[(Y - \hat Y(x))^2 | X = x] &= E[Y^2 - 2Y \hat Y(x) + \hat Y(x)^2 | X = x] \\ & = E[Y^2|X=x] - 2\hat Y(x) E[Y|X=x] + \hat Y(x)^2 \\ & = 1 \times Pr(Y^2 = 1|X=x) - 2\hat Y(x) Pr(Y = 1|X=x) + \hat Y(x)^2 \\ & = Pr(Y = 1|X=x)[1 -2\hat Y(x) ] + \hat Y(x)^2 \end{aligned}

Since $X = x$, the term $\hat Y(x)$ is not a r.v. and $Pr(Y^2 = 1) = Pr(Y = 1)$

Minimizing by equating the derivative w.r.t. $\hat Y(x)$ to zero \begin{aligned} \frac{\partial}{\partial \hat Y(x)} &= -2 Pr(Y = 1|X=x) + 2 \hat Y(x) = 0 \\ \hat Y(x) &= Pr(Y = 1|X=x) \end{aligned}

Incorrect approach

Since $\hat Y(x) \in \{0,1\}$ we must round to the nearest value giving

• $\hat Y(x) = 1$ if $Pr(Y = 1|X=x) > 0.5$
• $\hat Y(x) = 0$ if $Pr(Y = 1|X=x) < 0.5$

From comments below: Implicitly using the claim that if we want to maximize a function $f$ over a discrete set $\{x_1,…,x_n\}$, we need simply choose the $x_i$ which lies closest to the function's maximum $x^∗=\arg \max f(x)$ over its entire domain. This claim does not hold

Correct approach

By inspection of the discrete cases. Consider as an example the case where $Pr(Y = 1|X=x) = 0.7$

The probability of missclassification is $Pr(Y \neq \hat Y | X = x) = 0.7 - 1.4 \hat Y + \hat Y^2$

• If we choose $\hat Y = 0$ then $Pr(Y \neq \hat Y | X = x) = 0.7$
• If we choose $\hat Y = 1$ then $Pr(Y \neq \hat Y | X = x) = 0.3$

Hence in this case we minimize the missclassification error by choosing $\hat Y = 1$. Perform similar calculations for the other case.

• Your answer is correct except for the last part about rounding. You are implicitly using the claim that if we want to maximize a function $f$ over a discrete set $\{x_1, \dots, x_n\}$, we need simply choose the $x_i$ which lies closest to the function's maximum $x^* = \text{argmax } f(x)$ over its entire domain. This claim does not hold. Again, since the target variable is binary, you can just enumerate the possible cases. Jun 27 '18 at 16:24
• Excellent ! I have corrected the approach - it is an interesting question and an interesting mistake I made. Jun 29 '18 at 7:26