33
$\begingroup$

The canonical probabilistic interpretation of linear regression is that $y$ is equal to $\theta^Tx$, plus a Gaussian noise random variable $\epsilon$.

However, in standard logistic regression, we don't consider noise (e.g. random bit flips with probability $p$) of the label $y$. Why is that?

$\endgroup$
6
  • 1
    $\begingroup$ Does this answer your question? Is it possible to simulate logistic regression without randomness? $\endgroup$
    – Haitao Du
    Commented Aug 4, 2020 at 5:33
  • $\begingroup$ @HaitaoDu Thanks, the two answers there are quite related, but they don't have an explicit answer to my question (at least not one that I can understand). $\endgroup$
    – kennysong
    Commented Aug 4, 2020 at 5:41
  • $\begingroup$ Another related question here on utility foundations of binary choice models. $\endgroup$
    – dimitriy
    Commented Aug 4, 2020 at 5:48
  • 1
    $\begingroup$ And another: Logistic Regression - Error Term and its Distribution $\endgroup$
    – Scortchi
    Commented Aug 4, 2020 at 13:32
  • $\begingroup$ As a side note, if you had something along the lines of heteroskedasticity in logistic regression (e.g., a positive value of an indicator variable introduced so much extra noise that most predictions would be close to the baseline regardless of other independent variables), you would need to use an interaction effect that could zero-out those variable's effects. For more complicated interactions, I imagine you would need to use a different kind of model. $\endgroup$ Commented Aug 5, 2020 at 18:09

2 Answers 2

40
$\begingroup$

Short answer: we do, just implicitly.


A possibly more enlightening way of looking at things is the following.

In Ordinary Least Squares, we can consider that we do not model the errors or noise as $N(0,\sigma^2)$ distributed, but we model the observations as $N(x\beta,\sigma^2)$ distributed.

(Of course, this is precisely the same thing, just looking at it in two different ways.)

Now the analogous statement for logistic regression becomes clear: here, we model the observations as Bernoulli distributed with parameter $p(x)=\frac{1}{1+e^{-x\beta}}$.

We can flip this last way of thinking around if we want: we can indeed say that we are modeling the errors in logistic regression. Namely, we are modeling them as "the difference between a Bernoulli distributed variable with parameter $p(x)$ and $p(x)$ itself".

This is just very unwieldy, and this distribution does not have a name, plus the error here depends on our independent variables $x$ (in contrast to the homoskedasticity assumption in OLS, where the error is independent of $x$), so this way of looking at things is just not used as often.

$\endgroup$
4
  • 10
    $\begingroup$ (+1) My understanding of regression models improved drastically once I stopped thinking about modelling the "distribution + residuals" and started thinking about modelling the conditional distribution (your second paragraph). $\endgroup$ Commented Aug 4, 2020 at 7:00
  • 8
    $\begingroup$ @Stephan Kolassa: I got here from some other more recent thread and just wanted to comment that that's a TREMENDOUS explanation. I've always been confused with respect to where the noise was and now I'm not. Thanks. $\endgroup$
    – mlofton
    Commented Aug 4, 2020 at 11:59
  • $\begingroup$ With GLM we actually do flip it around and make it unwieldy by thinking of the difference between the observed value and modeled value. But indeed. it does not make so much physical sense to see it that way (it is not like the noise is some additive mechanism and the starting point is not like that), and it is more of a mathematical trick to minimize the likelihood by minimizing the squares of weighted residuals (which has a simple algebraic solution). $\endgroup$ Commented Aug 5, 2020 at 11:53
  • 2
    $\begingroup$ Stephan Kolassa, bra-VO! And thank you!. @SextusEmpiricus You make a very good point. We should be careful reifying our mathematical tools—my old mentor would use the example of the fast Fourier transform: just because you can perfectly represent any sound as a time series of infinite sums of single-frequency sine waves, does not mean that there are an infinite number of discrete sine-wave generators in, say, one's throat. :) $\endgroup$
    – Alexis
    Commented Jan 6, 2022 at 5:10
6
$\begingroup$

To supplement Stephan's answer, similar to how in linear regression the target $y$ is generated by a ''systematic'' component involving $x$ and an independent ''noise'' component, in logistic regression (and softmax regression more generally) you can actually also think of the target $y$ as computed by the following operation involving $x$ and some noise $\epsilon$:

$$ y = \arg \max_{i \in \{0, 1\}} (\alpha_i + \epsilon_i)$$

where $\alpha_0 = 0, \alpha_1 = \theta^T x$, and $\epsilon_0, \epsilon_1$ are both independent "noise" variables following $\text{Gumbel}(0,1)$ distribution; you can check that this way $y$ follows Bernoulli with $\mathbb{P}(y=1|x)= 1/(1+e^{-\theta^T x})$ as desired.

This way of sampling from a categorical (in this case Bernoulli) distribution is widely known as the Gumbel-max trick in machine learning: https://lips.cs.princeton.edu/the-gumbel-max-trick-for-discrete-distributions/ (The basic idea comes from the reparameterization trick. There's also a closely related Gumbel-softmax trick that essentially turns the above $\arg \max$ operation of Gumbel-max differentiable).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.