Since Logistic Regression is a statistical classification model dealing with categorical dependent variables, why isn't it called Logistic Classification? Shouldn't the "Regression" name be reserved to models dealing with continuous dependent variables?
Logistic regression is emphatically not a classification algorithm on its own. It is only a classification algorithm in combination with a decision rule that makes dichotomous the predicted probabilities of the outcome. Logistic regression is a regression model because it estimates the probability of class membership as a (transformation of a) multilinear function of the features.
Frank Harrell has posted a number of answers on this website enumerating the pitfalls of regarding logistic regression as a classification algorithm. Among them:
- Classification is a decision. To make an optimal decision, you need to asses a utility function, which implies that you need to account for the uncertainty in the outcome, i.e. a probability.
- The costs of misclassification are not uniform across all units.
- Don't use cutoffs.
- Use proper scoring rules.
- The problem is actually risk estimation, not classification.
If I recall correctly, he once pointed me to his book on regression strategies for more elaboration on these (and more!) points, but I can't seem to find that particular post.
Abstractly, regression is the problem of calculating a conditional expectation $E[Y|X=x]$. The form taken by this expectation is different depending on the assumptions of how the data were generated:
- Assuming (Y|X=x) to be normally distributed yields with classical linear regression.
- Assuming a Poisson distribution yields Poisson regression.
- Assuming a Bernoulli distribution yields logistic regression.
The term "regression" has also been used more generally than this, including approaches like quantile regression, which estimates a given quantile of $(Y|X=x)$.