Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
I think I understand the process of determining a Maximum Likelihood Estimator as being similar to the machine learning process of Gradient Descent for Linear Regression in that GD for LR results in a set of parameters that minimizes Least Squares Error and results in the best fitting regression line.
Finding the MLE results in determining a set of parameters that also most closely fits the data.
Is this analogy appropriate?
In general I would say that is correct.
As we get in the weeds of the math (and depending on what parameterization of Gradient Descent you're using) there are of course differences. One is of course that MLE requires a distribution to be differentiable, and generally you get into problems if there is more than one local minimum or maximum (in statistical terms we measure this by determining if the Hessian Matrix is semi-positive definite, but I won't get into that here).
In general, you could say that many machine learning and statistical modeling methods in some way use a form of the function:
$$argmin(f(Y))=X\beta$$
$X$ here is our data, $\beta$ is our parameters or model (in the case of say, a NN this is the whole network). For something like clustering f(Y) may be some combination of within and with-out group variance, while with regression this should look pretty familiar.
Your analogy is basically correct, although a bit loose; for you say that "Finding the MLE results in determining a set of parameters that also most closely fits the data.", and this requires a notion of what it means to "closely fit the data".
When you choose a set of estimates using MLE you are essentially picking from a family of distributions the one which is closest (in Kullback-Leibler sense) to the empirical distribution of the data.
