I understand that logistic regression has no analytical solution, but I was investigating the marginal effect in a simple logistic regression and followed this train of thought to the final line. Here it seems we have an analytical solution for $\beta$, so I must have done something wrong?

enter image description here

  • 3
    $\begingroup$ How would you compute the conditional probabilities in question if you do not already have a fit logistic regression? $\endgroup$ – Matthew Drury Aug 25 '17 at 1:40
  • $\begingroup$ estimated from the sample proportions ? $\endgroup$ – user2879934 Aug 25 '17 at 1:51
  • $\begingroup$ If $x$ is a continuous feature, it's very likely you will not have observed any data at both some value $t$ and $t+1$. Even if you have, different choices of reference point will give different answers. $\endgroup$ – Matthew Drury Aug 25 '17 at 2:05
  • $\begingroup$ but still, what is mathematically wrong with this statement? $\endgroup$ – user2879934 Aug 25 '17 at 10:00
  • $\begingroup$ or rather, I am missing some part of logistic regression that makes this obviously false, but I am not sure which part that is $\endgroup$ – user2879934 Aug 25 '17 at 10:01

what is mathematically wrong with this statement?

There's nothing mathematically wrong with your derivation, but the symbols must be interpreted in a particular way.

The conditional probabilities in your statement must mean the model predictions. Said differently, the first equation you write already assumes you have a fit logistic regression models, as the conditional probabilities in your model are the predictions from the model.

It no longer works if you interpret those probabilities in any other way, in particular, as empirical sample probabilities. For example:

1) $P(Y = 1 \mid x = 0)$ may not even be approximatable from your sample, you may not observe any data points at $x=0$.

2) If your $x$ feature is continuous, you may not observe any more than one data point at any value of the feature.

3) There may be no value of $t$ for which you have observed multiple data at both $x = t$ and $x = t + 1$. Even if you use other values for your separation, the same may be true.

4) Even if all the above are not a problem, you've observed many data points at values of $x$ that are spaced by some separation, using different reference points to compute your sample probabilities may lead to different answers, i.e. different values of $\beta$.

The magic of the regression model is that it gets around all these problems. It uses all your pairs of data, with all their spacings, to estimate one value of $\beta$ that best describes them all at once. There really is no way to do this in a simpler manner.

| cite | improve this answer | |
  • $\begingroup$ Okay I really like this answer except for one thing. I get how the lack of data in the real world could be a problem, but assume you have infinite data so the problems in 1,2,3 are gone. Now I claim that the above is a solution for beta....it just isn't the solution that maximizes the log likelihood of your data.UNLESS your predictor x is binary , then the above seems like a closed form because we replace x with 0 and x+1 with 1 $\endgroup$ – user2879934 Aug 25 '17 at 14:16
  • 1
    $\begingroup$ That, I believe, is true. In the binary case it is the beta that maximizes the log-likelihood. In any other case, yes, it gives you a beta, but you can't call the process logistic regression. $\endgroup$ – Matthew Drury Aug 25 '17 at 14:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.