2
$\begingroup$

I am studying logistic regression from the book Advanced Data Analysis from an Elementary Point of View which states the following on page 280:

“We minimize the mis-classification rate by predicting whichever class is more likely”:

Let $\hat Y(x)$ be our predicted class, either $0$ or $1$. Our error rate is then $P(Y \neq \hat Y)$ . Show that $P(Y \neq \hat Y) = E[(Y −\hat Y)^2]$ .

Further show that $E[(Y− \hat Y)^2|X=x] = Pr(Y=1|X=x)(1−2\hat Y(x))+\hat Y^2(x)$.

Conclude by showing that if $Pr(Y = 1|X = x ) > 0.5$, the risk of mis-classification is mini- mized by taking $\hat Y = 1$, that if $Pr(Y = 1|X = x) < 0.5$ the risk is minimized by taking $\hat Y = 0$, and that when $Pr(Y = 1|X = x) = 0.5$ both predictions are equally risky.

I am a bit lost by this approach, as I have seen the derivations for the Bayes classifier using $L_2$ loss in the continuous case, and using 0-1 loss in the discrete case in ESL (page 18), but nothing of this sort before.

How should one approach this problem ?

$\endgroup$
4
  • $\begingroup$ You should start by expanding out the expectation on the right-hand side of the first equation as an explicit sum over all the possible values of $(Y-\hat{Y})^2$. (Hint: this sum only has two terms in it.) $\endgroup$
    – tddevlin
    Jun 25 '18 at 22:25
  • $\begingroup$ isn't that for squared loss ? I am confused by the fact that this seems to be a 0 - 1 output variable, can we apply squared loss in such a case ? $\endgroup$ Jun 26 '18 at 12:23
  • 1
    $\begingroup$ Indeed that is part of the purpose of this exercise: to show that in the case of a binary target variable, mean squared error is equivalent to the misclassification rate. $\endgroup$
    – tddevlin
    Jun 26 '18 at 20:12
  • $\begingroup$ @tddevlin Thanks for your hints it helped a lot, is this correct ? $\endgroup$ Jun 27 '18 at 11:17
1
$\begingroup$

An attempt following hints by @tddevlin. Since $Y, \hat Y$ can only take values $ \{ 0,1\}$ the algebra is considerably simplified

\begin{aligned} E[(Y - \hat Y)^2] &= \sum (Y - \hat Y)^2 Pr((Y-\hat Y)^2) \\ & = 1 \times Pr( \ (Y-\hat Y)^2 = 1) + 0 \times Pr( \ (Y-\hat Y)^2 = 0) \\ & = Pr( \ (Y = 0, \hat Y = 1) \ \cup \ (Y = 1, \hat Y = 0) \ ) \\ & = Pr(Y \neq \hat Y) \end{aligned}

\begin{aligned} E[(Y - \hat Y(x))^2 | X = x] &= E[Y^2 - 2Y \hat Y(x) + \hat Y(x)^2 | X = x] \\ & = E[Y^2|X=x] - 2\hat Y(x) E[Y|X=x] + \hat Y(x)^2 \\ & = 1 \times Pr(Y^2 = 1|X=x) - 2\hat Y(x) Pr(Y = 1|X=x) + \hat Y(x)^2 \\ & = Pr(Y = 1|X=x)[1 -2\hat Y(x) ] + \hat Y(x)^2 \end{aligned}

Since $X = x$, the term $\hat Y(x)$ is not a r.v. and $Pr(Y^2 = 1) = Pr(Y = 1)$

Minimizing by equating the derivative w.r.t. $\hat Y(x)$ to zero \begin{aligned} \frac{\partial}{\partial \hat Y(x)} &= -2 Pr(Y = 1|X=x) + 2 \hat Y(x) = 0 \\ \hat Y(x) &= Pr(Y = 1|X=x) \end{aligned}

Incorrect approach

Since $\hat Y(x) \in \{0,1\}$ we must round to the nearest value giving

  • $ \hat Y(x) = 1$ if $Pr(Y = 1|X=x) > 0.5$
  • $ \hat Y(x) = 0$ if $Pr(Y = 1|X=x) < 0.5$

From comments below: Implicitly using the claim that if we want to maximize a function $f$ over a discrete set $\{x_1,…,x_n\}$, we need simply choose the $x_i$ which lies closest to the function's maximum $x^∗=\arg \max f(x)$ over its entire domain. This claim does not hold

Correct approach

By inspection of the discrete cases. Consider as an example the case where $Pr(Y = 1|X=x) = 0.7$

The probability of missclassification is $Pr(Y \neq \hat Y | X = x) = 0.7 - 1.4 \hat Y + \hat Y^2$

  • If we choose $\hat Y = 0$ then $Pr(Y \neq \hat Y | X = x) = 0.7$
  • If we choose $\hat Y = 1$ then $Pr(Y \neq \hat Y | X = x) = 0.3$

Hence in this case we minimize the missclassification error by choosing $\hat Y = 1$. Perform similar calculations for the other case.

$\endgroup$
2
  • $\begingroup$ Your answer is correct except for the last part about rounding. You are implicitly using the claim that if we want to maximize a function $f$ over a discrete set $\{x_1, \dots, x_n\}$, we need simply choose the $x_i$ which lies closest to the function's maximum $x^* = \text{argmax } f(x)$ over its entire domain. This claim does not hold. Again, since the target variable is binary, you can just enumerate the possible cases. $\endgroup$
    – tddevlin
    Jun 27 '18 at 16:24
  • $\begingroup$ Excellent ! I have corrected the approach - it is an interesting question and an interesting mistake I made. $\endgroup$ Jun 29 '18 at 7:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.