# Bayes optimal decision for logistic regression: Self-study exercise

We want to find the Bayes optimal decision for logistic regression. That means that the goal is to find the actions, which minimize our expected loss (also often called expected cost or risk). Here are the the characteristics:

• $\begin{equation} \text{Conditional distribution$P(y|x)$} = \begin{cases} \sigma (\mathbf{w}^T\mathbf{x}), & \text{if}\ y=1 \\ 1-\sigma (\mathbf{w}^T\mathbf{x}), & \text{if y= -1} \end{cases} \end{equation}$

• Action set = $\{-1,1,D\}$

• $\begin{equation} \text{Cost-function$C(y,a)$} = \begin{cases} (\mathbf{1}_{y \neq a}), & \text{if a$\in${1,-1}}\ \\ c < 0.5, & \text{if a = D} \end{cases} \end{equation}$

Where $\sigma$ is the Sigmoid-function used in the context of logistic regression. D means that "we don't know". The cost function is written down as follows in our solutions:

$(*) \mathbb{E}[C(a,y)|x] = \int C(y,a)P(y|x)dy = \sum_y (\mathbb{1}[y \neq a]\mathbb{1}[a \neq D]+c*\mathbb{1}[D=a])P(y|x)$

My question

The way (*) is written, it reads to me as:

$(\mathbb{1}[y_1 \neq a]\mathbb{1}[a \neq D]P(y|x)+c*\mathbb{1}[D=a]P(y|x) + (\mathbb{1}[y_{-1} \neq a]\mathbb{1}[a \neq D]P(y|x)+c*\mathbb{1}[D=a]P(y|x)$

So we are summing up the cost for D twice - is this not wrong?

• Are you sure this is the posterior distribution? It seems the posterior would more look like $p(w|y,x)$ Aug 4 '17 at 10:23
• It's quite different. What is called the conditional distribution here is a model assumption which will lead to the posterior distribution. Aug 4 '17 at 12:10
• No it's a model assumption. You assume the data are conditionally Binomial with $y|x \sim Bin(n, \sigma(w^Tx))$, I suppose. Aug 4 '17 at 14:05
• This seems to be a strange statement. The posterior is $p(w|y,x)$ and its not estimated but simulated from or approximated Aug 4 '17 at 14:08
• Ah I think I have seen this before. However, imho the posterior probability is defined as the probability of the parameters given the data. See here: en.wikipedia.org/wiki/Posterior_probability Aug 4 '17 at 14:20

$$E[C(+1,y)|x]=P(y=-1|x)$$ $$E[C(-1,y)|x]=P(y=+1|x)$$ $$E[C(D,y)|x]=c$$
The expected cost for "don't know" doesn't depend on the posterior probabilities, because the cost doesn't depend on $$y$$. This is why you see it appear "twice" (once for each outcome).
the summation is over $$y$$ ; in your case $$-1$$ and $$1$$. The $$P(y|x)$$ term will be once for $$y=-1$$ and once for $$y=1$$
\begin{align}&(\mathbb1[y_1\ne a]\mathbb1[a\ne D]P(y_1|x)+c\cdot \mathbb1[D=a]P(y_1|x)\\&+(\mathbb1[y_{−1}\ne a]\mathbb1[a\ne D]P(y_{-1}|x)+c\cdot 1[D=a]P(y_{-1}|x)\end{align}