3
$\begingroup$

I am taking this short example from wiki:https://en.wikipedia.org/wiki/Logistic_regression

Hours   |  Pass
--------|-------
0.50    |   0
0.75    |   0
1.00    |   0
1.25    |   0
1.50    |   0
1.75    |   0
1.75    |   1
2.00    |   0
2.25    |   1
2.50    |   0
2.75    |   1
3.00    |   0
3.25    |   1
3.50    |   0
4.00    |   1
4.25    |   1
4.50    |   1
4.75    |   1
5.00    |   1
5.50    |   1

I am trying to manually calculate the intercept and coefficient.

How does the algorithm calculates the log of odds at each point to fit linear regression? If i try to calculate log of odds, I get into situation where it is log(0) or log(infinity). Can somebody please help here?

$\endgroup$
6
$\begingroup$

You are right that although you should be able to calculate the OLS coefficient estimate in logit space, you can't do it directly because the logit, $g(y) = \log \frac{p}{1-p}$, goes either to $-\infty$ for $y=0$ or $\infty$ for $y=1$. An added difficulty is that the variance in this model depends on $x$.

The likelihood for logistic regression is optimized by an algorithm called iteratively reweighted least squares (IRLS). There is a nice breakdown of this in Shalizi's Advanced Data Analysis from an Elementary Point of View, from which I have the details below:

  • To deal with the infinite logit problem, make a first-order Taylor approximation to $g(y)$ around the point $p$ such that $g(y) \approx g(p) + (y − p)g'(p)$. Since $g(p)$ is by definition $\beta_0 + \beta x$, put that in there instead of $g(p)$ and say that your effective response is $z = \beta_0 + \beta x + (y-p)g'(p)$.
  • Calculate the variance $V[Z|X=x] = V[(Y-p)g'(p) | X=x]=g'(p)^2V(p)$. Use this to weight your samples so that you can simply do a weighted regression of $z$ on $x$.

At this point you might ask yourself how you can use the regression coefficients you're trying to estimate to calculate your effective response, $z$. Of course you can't. And what is $p$ anyway? That's where the iterative part of IRLS comes in: you start with some guess at the $\beta$s, for instance to set them all to 0. From this you can first calculate the fitted probabilities $p$, and second use these fitted probabilities and your current coefficient estimates to calculate $z$.

All this and you get a new estimate for your $\beta$s, and it should be closer to the right one, but probably not the right one. So you iterate: use the new coefficients to calculate new fitted probabilities, calculate new effective responses, new weights, and go again. Sooner or later, unless you're unlucky, the $\beta$s will converge to a nice estimate. Says Shalizi:

The treatment above is rather heuristic, but it turns out to be equivalent to using Newton’s method, only with the expected second derivative of the log likelihood, instead of its actual value.

So in summary: you never use the logit directly because, as you point out, it's impractical. You can certainly calculate the logistic regression coefficients by hand, but it won't be fun.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.