I read in a machine learning book that parameters of linear regression can be estimated (among other methods) by gradient descent, while parameters of logistic regression are usually estimated by maximum likelihood estimation.
Is it possible to explain to a novice (me) why we need different methods for linear/logistic regression. a.k.a why not MLE for linear regression and why not gradient descent for logistic regression?
You're confusing apples with oranges. That's ok, because they are both delicious.
Maximum likelihood estimation is about what you minimize, gradient descent is about how you minimize it.
Why not MLE for linear regression?
In fact, linear regression is solved with maximum likelihood estimation. The standard "minimize the sum of squared errors" method is exactly mathematically equivalent to maximum likelihood estimation using a conditional normal distribution.
Why not gradient descent for logistic regression?
You can totally solve logistic regression by minimizing the likelihood function using gradient descent. It's a great exercise in fact, and I'd recommend everyone do it at least once.
Gradient descent is not the standard method though. That prize goes to iteratively re-weighted least squares / Newton's method, which is an enhancement to gradient descent that takes into account the second derivative as well. This method just turns out to have much better properties than gradient descent, but is trickier to understand and implement.
