4
$\begingroup$

I've seen some places(such as these lecture notes from ETH Zurich) where they simply declare MLE=ERM, but so far I haven't been able to find any good explanations (or, actually, any explanations at all) of when the two are equivalent and when they might not be. I'm assuming since the two have separate definitions that they must not be equivalent in all cases, but am I mistaken in that assumption?

$\endgroup$
3
$\begingroup$

ERM is equivalent to MLE when the risk is defined as the negative of the likelihood. Other than that I think the differences are subtle and differ from author to author. I'd expect that usually we would only call a method MLE if it optimizes some likelihood function by selecting $\eta(x) = \mathbb{P}(Y | X=x)$ while empirical risk minimization (in the context of classification) is usually framed as selecting a function which maps into $\{0, 1\}$.

$\endgroup$
0
$\begingroup$

ERM is concerned with risk that are expressible as functions. You can just as well use the negative log-likelihood as the risk function. In those cases, MLE is equal to ERM.

Now, ERM does not limits itself to that particular class of risk functions. A very successful example of ERM in machine learning is the SVM. The hinge loss in SVMs is not a negative log-likelihood (even though there are SVM reformulations which define negative log-likelihoods). Instead it's a discriminative loss, a convex approximation to the 0-1 loss (accuracy).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.