# Do you do linear regression in logistic regression?

I just studied single & multiple linear regression 2 days ago now I'm reading about logistics regression and I want to implement it from scratch, I just want to know If my understanding about logistic regression is correct.

You see I was actually quite confused about logistic regression, In my understanding in logistics regression we still use the equation of linear regression

$$\hat{y} = \hat\beta_0+\hat\beta_1x_1+\cdots+\hat\beta_nx_n$$

so what I think is happening here is:

• first find the best-fit-line, and for that to happen we need to find the beta coefficient just like multiple linear regression
• after finding the $$\hat\beta$$ coefficients, we plug them in the sigmoid function along with our independent variables to get the prediction, $$P(x) = \frac{1}{1+e^{-(b_0+b_1x_1+...+b_nx_n)}}$$ and then that's it.

Is that how it's done? or was that wrong? or was there some steps we need to do first or after?

• There is a lot more to logistic regression than what you state. There are quite a few more steps and the steps you have stated are not correct as you have them. There are many references on logistic regression, both online and in print. It would be easier to help you if you cited the reference you are following and explained which equations from the reference you are trying to implement. Commented Jun 22, 2021 at 6:11
• Also, it would surely help for you to try out one of the readily available logistic software tools first before trying to write you own version yourself. Then you would at least have an appreciation for the data format and output from logistic regression. Commented Jun 22, 2021 at 6:12

The simplest way of likening logistic regression to standard linear regression is using the latent variable interpretation. The logistic regression model can be described by considering the observable responses:

$$Y_i = \mathbb{I}(Y_i^* > 0),$$

based on the unobservable latent variables:

$$Y_i^* = \beta_0 + \beta_1 x_{i,1} + \cdots + \beta_m x_{i,m} + \varepsilon_i \quad \quad \quad \varepsilon_1, ... \varepsilon_n \sim \text{IID Logistic}(0, s).$$

As you can see, in this model form the latent response variable follows a linear regression model with an error term that has a logistic distribution. We observe whether the latent response variable is above zero or not, but we do not observe its actual value. For estimation purposes, this manifests in a substantial difference in estimating the coefficients in the model. We do not estimate using OLS estimation, so this is not "just like multiple linear regression". Nevertheless, we do indeed get estimated coefficients and this does get us an estimated function for the conditional probability of the response outcome, so there are certainly similarities.

No, it is not done like this. Quoting my other answer

Logistic regression can be described as a linear combination

$$\eta = \beta_0 + \beta_1 X_1 + ... + \beta_k X_k$$

that is passed through the link function $$g$$:

$$g(E(Y)) = \eta$$

where the link function is a logit function

$$E(Y|X,\beta) = p = \text{logit}^{-1}( \eta )$$

As you can see, the linear predictor $$\eta = \mathbf{X}\boldsymbol{\beta}$$ is not equal to the conditional mean of $$y$$, but you need to transform it first using the inverse of the logit link function $$g^{-1}(\eta)$$. If you just ran the linear regression, you would be ignoring the fact that the transformation of the linear predictor happens.

You can easily verify this yourself, run linear and logistic regression on the same data. If using linear regression would be enough, you should get the same regression parameters. As you can see from the example below, that's not the case.

> lm(vs~mpg+cyl, data=mtcars)

Call:
lm(formula = vs ~ mpg + cyl, data = mtcars)

Coefficients:
(Intercept)          mpg          cyl
2.164638    -0.008217    -0.252454

> glm(vs~mpg+cyl, family=binomial, data=mtcars)

Call:  glm(formula = vs ~ mpg + cyl, family = binomial, data = mtcars)

Coefficients:
(Intercept)          mpg          cyl
15.9714      -0.1633      -2.1482

Degrees of Freedom: 31 Total (i.e. Null);  29 Residual
Null Deviance:      43.86
Residual Deviance: 17.49    AIC: 23.49


Logistic regression is fitted by using an optimization algorithm that maximizes the likelihood function. The likelihood function is defined in terms of Bernoulli distribution:

$$L(\boldsymbol{\beta}|y;\mathbf{X}) = \prod_i\, g^{-1}(\mathbf{X}_i\boldsymbol{\beta})^{y_i} \, (1 - g^{-1}(\mathbf{X}_i\boldsymbol{\beta}))^{1-y_i}$$

Commonly IRLS algorithm is used for finding the maximum of this function, but you probably could ignore this fact and live a happy life without that knowledge.

• I lack the statistical depth of this answer, but my understanding of the question is OP simply asking if we perform linear regression in logistic. The first step (the linear combination part) is where we perform linear regression - off course, there's much more to logistic regression - and I think that's all he needs to know. Commented Jun 23, 2021 at 7:20