I have heard people describe logistic regression as linear regression except as it is deployed for classification. But I have heard the exact same comment about LDA (linear discriminant analysis). Out of logistic regression and LDA, which is closer to what happens in linear regression?
They are both close, but in different ways
- If you run ordinary least-squares regression with a binary class variable as the outcome (label) variable, you get exactly the 2-class case of linear discriminant analysis. So LDA (in the 2-class case) is linear regression run on a classification problem. It's conceptually different from linear regression in that the original derivation of LDA uses assumptions about the distribution of the predictor (feature) variables, which regression does not.
- Logistic regression is a natural generalisation of linear regression to binary data, in which you model the mean of the outcome variable (which is the probability that it is 1 vs 0) using a linear combination of predictors, but with a 'link' function in between so that the probability stays between 0 and 1. Like linear regression, it's a special case of the generalised linear model, and wasn't derived based on assumptions about the distributions of predictor variables.
So, LDA is computationally just linear regression as applied to a classification problem, but it's quite different as a model; logistic regression is closer as a model, but less similar computationally.