Do you do linear regression in logistic regression?

Question

I just studied single & multiple linear regression 2 days ago now I'm reading about logistics regression and I want to implement it from scratch, I just want to know If my understanding about logistic regression is correct.

You see I was actually quite confused about logistic regression, In my understanding in logistics regression we still use the equation of linear regression

$$\hat{y} = \hat\beta_0+\hat\beta_1x_1+\cdots+\hat\beta_nx_n$$

so what I think is happening here is:

• first find the best-fit-line, and for that to happen we need to find the beta coefficient just like multiple linear regression
• after finding the $\hat\beta$ coefficients, we plug them in the sigmoid function along with our independent variables to get the prediction, $$P(x) = \frac{1}{1+e^{-(b_0+b_1x_1+...+b_nx_n)}}$$ and then that's it.

Is that how it's done? or was that wrong? or was there some steps we need to do first or after?

Answer 1

The simplest way of likening logistic regression to standard linear regression is using the latent variable interpretation. The logistic regression model can be described by considering the observable responses:

$$Y_i = \mathbb{I}(Y_i^* > 0),$$

based on the unobservable latent variables:

$$Y_i^* = \beta_0 + \beta_1 x_{i,1} + \cdots + \beta_m x_{i,m} + \varepsilon_i \quad \quad \quad \varepsilon_1, ... \varepsilon_n \sim \text{IID Logistic}(0, s).$$

As you can see, in this model form the latent response variable follows a linear regression model with an error term that has a logistic distribution. We observe whether the latent response variable is above zero or not, but we do not observe its actual value. For estimation purposes, this manifests in a substantial difference in estimating the coefficients in the model. We do not estimate using OLS estimation, so this is not "just like multiple linear regression". Nevertheless, we do indeed get estimated coefficients and this does get us an estimated function for the conditional probability of the response outcome, so there are certainly similarities.

Answer 2

No, it is not done like this. Quoting my other answer

Logistic regression can be described as a linear combination

$$ \eta = \beta_0 + \beta_1 X_1 + ... + \beta_k X_k $$

that is passed through the link function $g$:

$$ g(E(Y)) = \eta $$

where the link function is a logit function

$$ E(Y|X,\beta) = p = \text{logit}^{-1}( \eta ) $$

As you can see, the linear predictor $\eta = \mathbf{X}\boldsymbol{\beta}$ is not equal to the conditional mean of $y$, but you need to transform it first using the inverse of the logit link function $g^{-1}(\eta)$. If you just ran the linear regression, you would be ignoring the fact that the transformation of the linear predictor happens.

You can easily verify this yourself, run linear and logistic regression on the same data. If using linear regression would be enough, you should get the same regression parameters. As you can see from the example below, that's not the case.

> lm(vs~mpg+cyl, data=mtcars)

Call:
lm(formula = vs ~ mpg + cyl, data = mtcars)

Coefficients:
(Intercept)          mpg          cyl  
   2.164638    -0.008217    -0.252454 

> glm(vs~mpg+cyl, family=binomial, data=mtcars)

Call:  glm(formula = vs ~ mpg + cyl, family = binomial, data = mtcars)

Coefficients:
(Intercept)          mpg          cyl  
    15.9714      -0.1633      -2.1482  

Degrees of Freedom: 31 Total (i.e. Null);  29 Residual
Null Deviance:      43.86 
Residual Deviance: 17.49    AIC: 23.49

Logistic regression is fitted by using an optimization algorithm that maximizes the likelihood function. The likelihood function is defined in terms of Bernoulli distribution:

$$ L(\boldsymbol{\beta}|y;\mathbf{X}) = \prod_i\, g^{-1}(\mathbf{X}_i\boldsymbol{\beta})^{y_i} \, (1 - g^{-1}(\mathbf{X}_i\boldsymbol{\beta}))^{1-y_i} $$

Commonly IRLS algorithm is used for finding the maximum of this function, but you probably could ignore this fact and live a happy life without that knowledge.