0
$\begingroup$

Can you add regularization terms to any likelihood function you're trying to maximize? (e.g. L2/Tikhonov, Lasso terms)

I'm used to seeing this done with simple quadratic loss functions (e.g. for Tikhonov regularization, $Loss=(Y-\beta X)^T (Y-\beta X) + \alpha ||\beta||^2$), but can this be done during MLE? (even when MLE may not be equivalent to least squares minimization, because of non-Gaussian residuals)

The kind of likelihood function I'm applying this too looks a bit like this (but it could be even more complicated).

$\endgroup$
7
  • 3
    $\begingroup$ Are you asking if penalized likelihoods exist? The function certainly exists. Are you asking if the minimizers exist? The squared Euclidean norm is strictly convex, so it will if the (negative log) likelihood is convex. Are you asking about statistical properties of the minimizer? If so, the properties you care about should be clarified I think $\endgroup$
    – user551504
    Mar 9, 2022 at 0:31
  • $\begingroup$ Yes, it is. But more specifically, I'm asking whether or not you can add any (commonly used) regularization term, and still have a statistically justified loss/fitness function. $\endgroup$
    – Mich55
    Mar 9, 2022 at 14:14
  • 1
    $\begingroup$ Thanks, what does statistically justified mean here? $\endgroup$
    – user551504
    Mar 9, 2022 at 14:56
  • 1
    $\begingroup$ they won't be unbiased, most MLEs are not unbiased. They will be consistent if the tuning parameter $\alpha$ of the penalty approach $0$ quickly enough. $\endgroup$
    – user551504
    Mar 9, 2022 at 17:05
  • 1
    $\begingroup$ Yes, the big idea of regularization like this is that some additional finite-sample bias is incurred in order to reduce the finite-sample variance enough to hopefully make it worth it $\endgroup$
    – user551504
    Mar 10, 2022 at 16:08

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.