# Bayes optimal decision for logistic regression: Self-study exercise

We want to find the Bayes optimal decision for logistic regression. That means that the goal is to find the actions, which minimize our expected loss (also often called expected cost or risk). Here are the the characteristics:

• $$$\text{Conditional distribution P(y|x)} = \begin{cases} \sigma (\mathbf{w}^T\mathbf{x}), & \text{if}\ y=1 \\ 1-\sigma (\mathbf{w}^T\mathbf{x}), & \text{if y= -1} \end{cases}$$$

• Action set = $\{-1,1,D\}$

• $$$\text{Cost-function C(y,a)} = \begin{cases} (\mathbf{1}_{y \neq a}), & \text{if a \in {1,-1}}\ \\ c < 0.5, & \text{if a = D} \end{cases}$$$

Where $\sigma$ is the Sigmoid-function used in the context of logistic regression. D means that "we don't know". The cost function is written down as follows in our solutions:

$(*) \mathbb{E}[C(a,y)|x] = \int C(y,a)P(y|x)dy = \sum_y (\mathbb{1}[y \neq a]\mathbb{1}[a \neq D]+c*\mathbb{1}[D=a])P(y|x)$

My question

The way (*) is written, it reads to me as:

$(\mathbb{1}[y_1 \neq a]\mathbb{1}[a \neq D]P(y|x)+c*\mathbb{1}[D=a]P(y|x) + (\mathbb{1}[y_{-1} \neq a]\mathbb{1}[a \neq D]P(y|x)+c*\mathbb{1}[D=a]P(y|x)$

So we are summing up the cost for D twice - is this not wrong?

• Are you sure this is the posterior distribution? It seems the posterior would more look like $p(w|y,x)$ Aug 4, 2017 at 10:23
• It's quite different. What is called the conditional distribution here is a model assumption which will lead to the posterior distribution. Aug 4, 2017 at 12:10
• No it's a model assumption. You assume the data are conditionally Binomial with $y|x \sim Bin(n, \sigma(w^Tx))$, I suppose. Aug 4, 2017 at 14:05
• This seems to be a strange statement. The posterior is $p(w|y,x)$ and its not estimated but simulated from or approximated Aug 4, 2017 at 14:08
• Ah I think I have seen this before. However, imho the posterior probability is defined as the probability of the parameters given the data. See here: en.wikipedia.org/wiki/Posterior_probability Aug 4, 2017 at 14:20

I think it is likely to be less confusing if you first write out the posterior expected loss for each action separately, rather than as one complicated function. It is actually quite simple.

$$E[C(+1,y)|x]=P(y=-1|x)$$ $$E[C(-1,y)|x]=P(y=+1|x)$$ $$E[C(D,y)|x]=c$$

The expected cost for "don't know" doesn't depend on the posterior probabilities, because the cost doesn't depend on $$y$$. This is why you see it appear "twice" (once for each outcome).

the summation is over $$y$$ ; in your case $$-1$$ and $$1$$. The $$P(y|x)$$ term will be once for $$y=-1$$ and once for $$y=1$$

\begin{align}&(\mathbb1[y_1\ne a]\mathbb1[a\ne D]P(y_1|x)+c\cdot \mathbb1[D=a]P(y_1|x)\\&+(\mathbb1[y_{−1}\ne a]\mathbb1[a\ne D]P(y_{-1}|x)+c\cdot 1[D=a]P(y_{-1}|x)\end{align}