3
$\begingroup$

We want to find the Bayes optimal decision for logistic regression. That means that the goal is to find the actions, which minimize our expected loss (also often called expected cost or risk). Here are the the characteristics:

  • $ \begin{equation} \text{Conditional distribution $P(y|x)$} = \begin{cases} \sigma (\mathbf{w}^T\mathbf{x}), & \text{if}\ y=1 \\ 1-\sigma (\mathbf{w}^T\mathbf{x}), & \text{if y= -1} \end{cases} \end{equation}$

  • Action set = $\{-1,1,D\}$

  • $ \begin{equation} \text{Cost-function $C(y,a)$} = \begin{cases} (\mathbf{1}_{y \neq a}), & \text{if a $\in$ {1,-1}}\ \\ c < 0.5, & \text{if a = D} \end{cases} \end{equation}$

Where $\sigma$ is the Sigmoid-function used in the context of logistic regression. D means that "we don't know". The cost function is written down as follows in our solutions:

$(*) \mathbb{E}[C(a,y)|x] = \int C(y,a)P(y|x)dy = \sum_y (\mathbb{1}[y \neq a]\mathbb{1}[a \neq D]+c*\mathbb{1}[D=a])P(y|x)$

My question

The way (*) is written, it reads to me as:

$ (\mathbb{1}[y_1 \neq a]\mathbb{1}[a \neq D]P(y|x)+c*\mathbb{1}[D=a]P(y|x) + (\mathbb{1}[y_{-1} \neq a]\mathbb{1}[a \neq D]P(y|x)+c*\mathbb{1}[D=a]P(y|x)$

So we are summing up the cost for D twice - is this not wrong?

$\endgroup$
12
  • $\begingroup$ Are you sure this is the posterior distribution? It seems the posterior would more look like $p(w|y,x)$ $\endgroup$
    – tomka
    Aug 4 '17 at 10:23
  • 1
    $\begingroup$ It's quite different. What is called the conditional distribution here is a model assumption which will lead to the posterior distribution. $\endgroup$
    – tomka
    Aug 4 '17 at 12:10
  • 1
    $\begingroup$ No it's a model assumption. You assume the data are conditionally Binomial with $y|x \sim Bin(n, \sigma(w^Tx))$, I suppose. $\endgroup$
    – tomka
    Aug 4 '17 at 14:05
  • 1
    $\begingroup$ This seems to be a strange statement. The posterior is $p(w|y,x)$ and its not estimated but simulated from or approximated $\endgroup$
    – tomka
    Aug 4 '17 at 14:08
  • 1
    $\begingroup$ Ah I think I have seen this before. However, imho the posterior probability is defined as the probability of the parameters given the data. See here: en.wikipedia.org/wiki/Posterior_probability $\endgroup$
    – tomka
    Aug 4 '17 at 14:20
1
$\begingroup$

I think it is likely to be less confusing if you first write out the posterior expected loss for each action separately, rather than as one complicated function. It is actually quite simple.

$$E[C(+1,y)|x]=P(y=-1|x)$$ $$E[C(-1,y)|x]=P(y=+1|x)$$ $$E[C(D,y)|x]=c$$

The expected cost for "don't know" doesn't depend on the posterior probabilities, because the cost doesn't depend on $y$. This is why you see it appear "twice" (once for each outcome).

$\endgroup$
0
$\begingroup$

the summation is over $y$ ; in your case $-1$ and $1$. The $P(y|x)$ term will be once for $y=-1$ and once for $y=1$

\begin{align}&(\mathbb1[y_1\ne a]\mathbb1[a\ne D]P(y_1|x)+c\cdot \mathbb1[D=a]P(y_1|x)\\&+(\mathbb1[y_{−1}\ne a]\mathbb1[a\ne D]P(y_{-1}|x)+c\cdot 1[D=a]P(y_{-1}|x)\end{align}

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.