Bayes optimal decision for logistic regression: Self-study exercise

Question

We want to find the Bayes optimal decision for logistic regression. That means that the goal is to find the actions, which minimize our expected loss (also often called expected cost or risk). Here are the the characteristics:

• $ \begin{equation} \text{Conditional distribution $P(y|x)$} = \begin{cases} \sigma (\mathbf{w}^T\mathbf{x}), & \text{if}\ y=1 \\ 1-\sigma (\mathbf{w}^T\mathbf{x}), & \text{if y= -1} \end{cases} \end{equation}$

• Action set = $\{-1,1,D\}$

• $ \begin{equation} \text{Cost-function $C(y,a)$} = \begin{cases} (\mathbf{1}_{y \neq a}), & \text{if a $\in$ {1,-1}}\ \\ c < 0.5, & \text{if a = D} \end{cases} \end{equation}$

Where $\sigma$ is the Sigmoid-function used in the context of logistic regression. D means that "we don't know". The cost function is written down as follows in our solutions:

$(*) \mathbb{E}[C(a,y)|x] = \int C(y,a)P(y|x)dy = \sum_y (\mathbb{1}[y \neq a]\mathbb{1}[a \neq D]+c*\mathbb{1}[D=a])P(y|x)$

My question

The way (*) is written, it reads to me as:

$ (\mathbb{1}[y_1 \neq a]\mathbb{1}[a \neq D]P(y|x)+c*\mathbb{1}[D=a]P(y|x) + (\mathbb{1}[y_{-1} \neq a]\mathbb{1}[a \neq D]P(y|x)+c*\mathbb{1}[D=a]P(y|x)$

So we are summing up the cost for D twice - is this not wrong?

Answer 1

I think it is likely to be less confusing if you first write out the posterior expected loss for each action separately, rather than as one complicated function. It is actually quite simple.

$$E[C(+1,y)|x]=P(y=-1|x)$$ $$E[C(-1,y)|x]=P(y=+1|x)$$ $$E[C(D,y)|x]=c$$

The expected cost for "don't know" doesn't depend on the posterior probabilities, because the cost doesn't depend on $y$. This is why you see it appear "twice" (once for each outcome).

Answer 2

the summation is over $y$ ; in your case $-1$ and $1$. The $P(y|x)$ term will be once for $y=-1$ and once for $y=1$

\begin{align}&(\mathbb1[y_1\ne a]\mathbb1[a\ne D]P(y_1|x)+c\cdot \mathbb1[D=a]P(y_1|x)\\&+(\mathbb1[y_{−1}\ne a]\mathbb1[a\ne D]P(y_{-1}|x)+c\cdot 1[D=a]P(y_{-1}|x)\end{align}