0
$\begingroup$

I think I understand the process of determining a Maximum Likelihood Estimator as being similar to the machine learning process of Gradient Descent for Linear Regression in that GD for LR results in a set of parameters that minimizes Least Squares Error and results in the best fitting regression line.

Finding the MLE results in determining a set of parameters that also most closely fits the data.

Is this analogy appropriate?

$\endgroup$
1
  • $\begingroup$ Indeed, there is a direct relationship between the MLE for linear regression and MSE loss. We can even formalize this concept in terms of probability and generalize it to all sorts of models beyond ordinary linear regression. stats.stackexchange.com/questions/378274/… $\endgroup$
    – Sycorax
    Commented Dec 16, 2020 at 16:10

2 Answers 2

1
$\begingroup$

In general I would say that is correct.

As we get in the weeds of the math (and depending on what parameterization of Gradient Descent you're using) there are of course differences. One is of course that MLE requires a distribution to be differentiable, and generally you get into problems if there is more than one local minimum or maximum (in statistical terms we measure this by determining if the Hessian Matrix is semi-positive definite, but I won't get into that here).

In general, you could say that many machine learning and statistical modeling methods in some way use a form of the function:

$$argmin(f(Y))=X\beta$$

$X$ here is our data, $\beta$ is our parameters or model (in the case of say, a NN this is the whole network). For something like clustering f(Y) may be some combination of within and with-out group variance, while with regression this should look pretty familiar.

$\endgroup$
2
$\begingroup$

Your analogy is basically correct, although a bit loose; for you say that "Finding the MLE results in determining a set of parameters that also most closely fits the data.", and this requires a notion of what it means to "closely fit the data".

When you choose a set of estimates using MLE you are essentially picking from a family of distributions the one which is closest (in Kullback-Leibler sense) to the empirical distribution of the data.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.