32
$\begingroup$

The canonical probabilistic interpretation of linear regression is that $y$ is equal to $\theta^Tx$, plus a Gaussian noise random variable $\epsilon$.

However, in standard logistic regression, we don't consider noise (e.g. random bit flips with probability $p$) of the label $y$. Why is that?

$\endgroup$
6

2 Answers 2

38
$\begingroup$

Short answer: we do, just implicitly.


A possibly more enlightening way of looking at things is the following.

In Ordinary Least Squares, we can consider that we do not model the errors or noise as $N(0,\sigma^2)$ distributed, but we model the observations as $N(x\beta,\sigma^2)$ distributed.

(Of course, this is precisely the same thing, just looking at it in two different ways.)

Now the analogous statement for logistic regression becomes clear: here, we model the observations as Bernoulli distributed with parameter $p(x)=\frac{1}{1+e^{-x\beta}}$.

We can flip this last way of thinking around if we want: we can indeed say that we are modeling the errors in logistic regression. Namely, we are modeling them as "the difference between a Bernoulli distributed variable with parameter $p(x)$ and $p(x)$ itself".

This is just very unwieldy, and this distribution does not have a name, plus the error here depends on our independent variables $x$ (in contrast to the homoskedasticity assumption in OLS, where the error is independent of $x$), so this way of looking at things is just not used as often.

$\endgroup$
4
  • 9
    $\begingroup$ (+1) My understanding of regression models improved drastically once I stopped thinking about modelling the "distribution + residuals" and started thinking about modelling the conditional distribution (your second paragraph). $\endgroup$ Aug 4, 2020 at 7:00
  • 8
    $\begingroup$ @Stephan Kolassa: I got here from some other more recent thread and just wanted to comment that that's a TREMENDOUS explanation. I've always been confused with respect to where the noise was and now I'm not. Thanks. $\endgroup$
    – mlofton
    Aug 4, 2020 at 11:59
  • $\begingroup$ With GLM we actually do flip it around and make it unwieldy by thinking of the difference between the observed value and modeled value. But indeed. it does not make so much physical sense to see it that way (it is not like the noise is some additive mechanism and the starting point is not like that), and it is more of a mathematical trick to minimize the likelihood by minimizing the squares of weighted residuals (which has a simple algebraic solution). $\endgroup$ Aug 5, 2020 at 11:53
  • 1
    $\begingroup$ Stephan Kolassa, bra-VO! And thank you!. @SextusEmpiricus You make a very good point. We should be careful reifying our mathematical tools—my old mentor would use the example of the fast Fourier transform: just because you can perfectly represent any sound as a time series of infinite sums of single-frequency sine waves, does not mean that there are an infinite number of discrete sine-wave generators in, say, one's throat. :) $\endgroup$
    – Alexis
    Jan 6, 2022 at 5:10
5
$\begingroup$

To supplement Stephan's answer, similar to how in linear regression the target $y$ is generated by a ''systematic'' component involving $x$ and an independent ''noise'' component, in logistic regression (and softmax regression more generally) you can actually also think of the target $y$ as computed by the following operation involving $x$ and some noise $\epsilon$:

$$ y = \arg \max_{i \in \{0, 1\}} (\alpha_i + \epsilon_i)$$

where $\alpha_0 = 0, \alpha_1 = \theta^T x$, and $\epsilon_0, \epsilon_1$ are both independent "noise" variables following $\text{Gumbel}(0,1)$ distribution; you can check that this way $y$ follows Bernoulli with $\mathbb{P}(y=1|x)= 1/(1+e^{-\theta^T x})$ as desired.

This way of sampling from a categorical (in this case Bernoulli) distribution is widely known as the Gumbel-max trick in machine learning: https://lips.cs.princeton.edu/the-gumbel-max-trick-for-discrete-distributions/ (The basic idea comes from the reparameterization trick. There's also a closely related Gumbel-softmax trick that essentially turns the above $\arg \max$ operation of Gumbel-max differentiable).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.