1
$\begingroup$

I read logistic regression formula i.e. $ \log\frac{P(C_1|X)}{P(C_2|X)}=w^Tx+w_o$ but this equation is true if we have $P(X|C_1)$ and $P(X|C_2)$ sampled from two Gaussian with the same covariance matrices ($\sum_1=\sum_2$); furthermore, we can derive the sigmoid transformation $P(C_1|X) = \frac{1}{1+ \exp{(-(w^Tx+w_o))}}$ but the linearity is given by our assumption on $P(X|C_1)$ and $P(X|C_2)$; the question is: I have a strong assumption on the data generation $P(X|C_i)$ (i.e. Gaussian with same covariance matrix and then same "shape") but in logistic regression we throw away all these assumption and use this model without taking care the original data generation (I understand this from Machine Learning: A Bayesian and Optimization Perspective , page 291); isn't this a very strange thing? We are using something build on an assumption and then we throw away this assumption and use this model however? It seems to me a very bias model, because seems to be true just for a class of data (generated with Gaussians with same covariace) but not for all others dataset; where is my mistake?

EDIT Here it said the same Logistic Regression cite: We make little assumptions on P(x|y), e.g. it could be Gaussian or Multinomial. Ultimately it doesn't matter, because we estimate the vector w and b directly with MLE or MAP. so, it seems that an assumption on Gaussian is done, but, in my opinion, the question remains, because I still make an MLE on a model that is based on data made by two gaussian with same covariances ...

$\endgroup$
4
  • 5
    $\begingroup$ There is no such Normality assumption in "standard" logistic regression. To my understanding, this book (which is known to be a great book), deals with logistic regression in the context of Bayesian inference. Quoting the author: "when the distributions underlying the data are Gaussians with a common covariance matrix, then the log ratio of the posteriors is a linear function". The assumption refers to posterior distributions, a concept which doesn't exist in a classic GLM. $\endgroup$
    – Digio
    Dec 23, 2017 at 10:37
  • 3
    $\begingroup$ In addition, the models between the two schools are not at all similar. In the basic scenario of a single categorical regressor $z_{ij}$ with two levels $j=1, ~j=2$, with a non-Bayesian logistic model you fit: $logit(\pi_{ij}) = log{\pi_{ij} \over 1-\pi_{ij}} = log{(~Odds_{ij})} = b_0 + b_1z_{ij}$. As you can see, the classic model regresses the log of the Odds and not the ratio of the posteriors and therefore there is an assumption for linearity but not for Normality. $\endgroup$
    – Digio
    Dec 23, 2017 at 11:01
  • $\begingroup$ but here it said the same cs.cornell.edu/courses/cs4780/2015fa/web/lecturenotes/… cite: ** We make little assumptions on P(x|y), e.g. it could be Gaussian or Multinomial. Ultimately it doesn't matter, because we estimate the vector w and b directly with MLE or MAP. ** so, it seems that an assumption on Gaussian is done, but, in my opinion, is very strong.... $\endgroup$
    – volperossa
    Dec 23, 2017 at 13:01
  • 3
    $\begingroup$ This is also about logistic regression in a Bayesian framework where you estimate the posterior distribution $p(y|x)$ and not a point estimate $\hat y$ as in the classic framework. In classic logistic regression the data are assumed by construction to follow the Binomial distribution; if they follow Multinomial or Gaussian then they're another type of GLM. I find that this answers explain the differences well. $\endgroup$
    – Digio
    Dec 23, 2017 at 13:30

1 Answer 1

1
$\begingroup$

In a frequentist generalized linear model like logistic regression the Bernoulli likelihood function and logit link function leads to the maximum likelihood estimator for the parameters. When it comes time to perform inference we could derive the sampling distribution of the maximum likelihood estimator and use its CDF to construct p-values and confidence intervals. A convenient way to approximate this sampling distribution is with a normal approximation, i.e. a Wald test on the log odds scale. This is not an assumption about the data generative process, it is a convenient approximation to the sampling distribution of our estimator (a function of the complete sufficient statistic). Because the logit function is a monotonic transformation we can back transform inference on the log odds to the odds scale or the Bernoulli proportion scale.

This is analogous to using a logit link function in a Bayesian model with a normal prior distribution on the parameters so that the posterior is also a normal distribution. This posterior can be back transformed using the inverse link function to produce a posterior in the odds scale or the Bernoulli proportion scale.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.