5
$\begingroup$

If I want to perform a logistic regression analysis, can I use the linear regression model to compute the values of the parameters and then apply the formula:

$$\frac{1} {1 + \exp-(\beta_0+x_1\beta_1+\ldots +x_i\beta_i)}?$$

$\endgroup$
  • $\begingroup$ That would be a bad fit. Why would you ever do that? $\endgroup$ – Memming Sep 27 '13 at 15:12
  • $\begingroup$ why a bad fit ? $\endgroup$ – T-student Sep 27 '13 at 15:16
  • 1
    $\begingroup$ I think what @Memming means to say is that linear regression already supplies some kind of fit in the form of $\beta_0 + x_1\beta_1 + \cdots + x_i\beta_i.$ Applying the logistic function to these values is really going to screw up that fit, likely (but not invariably) making it worse. You can find out more by searching our site for (high-voted) answers about logistic regression and its interpretation: there are lots of them that should give you the insight you seek. $\endgroup$ – whuber Sep 27 '13 at 15:16
  • $\begingroup$ can you give me a link? please $\endgroup$ – T-student Sep 27 '13 at 15:18
  • 2
    $\begingroup$ I think there's a minus sign missing in the exponent, because the inverse logit function (a.k.a. the logistic function) is $1/(1+e^{-x})$. $\endgroup$ – Macro Sep 27 '13 at 15:45
1
$\begingroup$

This idea is a bit like Platt Scaling (see this previous question and the answers given), which is used to get probabilities from support vector machines and other machine learning methods. You could use this method (although you would need to add a scaling and an offset to the output of the linear regression to make it work as well as possible), however for a linear model, there isn't really much benefit in doing so as it is straight-forward to construct a logistic regression model with similar computational expense, and that will probably give better estimates of probability.

Essentially for most problems it is best to use the statistical method that gives the most direct answer to the question posed, rather than use a simpler (or more complicated) method and post-process the result.

See also Appendix D.2 of Tipping's paper on the Relevance Vector Machine, which explains why Platt scaling may not give as good a result as building a model that estimates the probability of class membership more directly.

| cite | improve this answer | |
$\endgroup$
5
$\begingroup$

A linear regression relates proportions $p$ linearly to predictors $x_1, x_2, \dots$: $$ p = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + \dots $$ while a logistic regression relates log-odds linearly to $x_1, x_2, \dots$: $$ \log (p/(1-p)) = \alpha_0 + \alpha_1 x_1 + \alpha_2 x_2 + \dots $$ Now, if you want to predict log-odds using linear regression (which is quite unnatural), you would clearly need to apply the logit function on predictions $\hat p$ from the linear regression, not the inverse logit as you propose. So my clear answer is "no, you should not use your formula".

| cite | improve this answer | |
$\endgroup$
0
$\begingroup$

You need to estimate a logistic regression by maximum likelihood. The following link shows the likelihood function along with empirical maximization using Newton's method.

http://www.stat.cmu.edu/~cshalizi/uADA/12/lectures/ch12.pdf

| cite | improve this answer | |
$\endgroup$
  • 8
    $\begingroup$ Just because ML is almost always used to estimate coefficients in a logistic regression does not mean it's the only way! We need to bear in mind that the probability model for our data and the statistical procedure we use to reason about our data with that model are different things. Logistic regression is, fundamentally, a model. Thus, Frank's question still stands unanswered: what is not right about the procedure he proposes? $\endgroup$ – whuber Sep 27 '13 at 15:32
  • $\begingroup$ @whuber this page says that i have to minimize the sum of the squares archives.math.utk.edu/ICTCM/VOL13/C013/paper.html but i see a lot of papers where it say to maximize the likelihood, them are both right ??? (here to says to minimize holehouse.org/mlclass/06_Logistic_Regression.html) $\endgroup$ – T-student Sep 27 '13 at 15:59
  • 1
    $\begingroup$ (BTW, I don't think this answer deserves downvotes: it was offered in a helpful spirit and is not incorrect.) $\endgroup$ – whuber Sep 27 '13 at 15:59
  • $\begingroup$ Frank, they are indeed both correct for the most common form of regression where (among other things) the possible error of each observation has the same size no matter what value the observation may have. That is almost never the case for binary observations. Minimizing the sum of squares it not a good way to carry out logistic regression. It can be done (as a crude initial estimate), but when it is you don't change the answer! You just use the prediction that is output by the least squares algorithm. Incidentally, as a general rule no statistical reference based on Excel is reliable. $\endgroup$ – whuber Sep 27 '13 at 16:02
  • 1
    $\begingroup$ I downvoted this because it doesn't answer the question. $\endgroup$ – Macro Sep 27 '13 at 17:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.