4
$\begingroup$

I am referring to the family of estimation techniques like MLEs, least-squares, etc., that an l2 penalizer/regularizer can be added to. I'm not interested in NHST, but just estimation (say, of some causal effect or association).

The way I see it is that adding a penalizer term does cause a bias (though MLEs are often already biased...), but there are more gains:

  1. the estimator is still consistent,
  2. the estimator has lower variance,
  3. the estimator can deal with co-linearity and separation problems,
  4. allows some expression of prior knowledge¹

Of course, adding too large of a penalizer will significantly bias results, but a practitioner should know a sensible value (and probably decided on beforehand).

What am I missing? Why should I not always added a small penalizer to my MLE models? Are my confidence intervals (I can't really call them confidence intervals anymore...) drastically broken if I do add a penalizer?

¹ Without going full Bayesian, adding a small penalizer tells the model "yeaaaa, 1e18 is not a likely value".

Improve this question
$\endgroup$
5
  • 1
    $\begingroup$ Point #2 is incorrect, consider for example the MLE of the mean of a Normal distribution, which is already minimum variance in the univariate case. $\endgroup$
    – jbowman
    Commented Apr 16, 2019 at 21:51
  • $\begingroup$ With respect to the last point, if I observe a bunch of failure rate data and all my time to fails are less than two years, I don't need a penalizer to tell the MLE estimation routine that 15 years MTBF is not a likely value. You seem to be way overstating the benefits of penalization... why do you think confidence intervals for non-penalized estimates might be "drastically broken" but wouldn't be if you added a penalty function? Why do you think a practitioner would have any idea what a sensible value for a penalty term would be? I sure wouldn't, and I've been doing this for years... $\endgroup$
    – jbowman
    Commented Apr 16, 2019 at 22:26
  • $\begingroup$ I must have not been clear, but I meant that adding a penalizer could lead to broken/misrepresentative CIs. A sensible value for a penalizer: 1e-5 - not too big to strongly influence results, but also big enough to discourage unrealistic effect sizes. (Also, Bayesian practitioners come up with sensible prior values all the time, do you also think they are walking on thin ice?) $\endgroup$
    – Cam.Davidson.Pilon
    Commented Apr 16, 2019 at 22:45
  • 1
    $\begingroup$ The magnitude of the penalty term has to be tied to the problem scale; if I divide the target variable by 100, all my coefficients in, say, a linear regression, will shrink by 100 as well, so the penalty had better too. That's why you can't come up with off-the-shelf penalties. Also, consider a Lasso regression with 100+ regressors; guessing a good penalty for something like that isn't likely to be much more feasible than guessing the coefficients themselves. Why guess at what penalty is when you can use cross-validation or a reasonable approximation thereto to select a good value? $\endgroup$
    – jbowman
    Commented Apr 17, 2019 at 0:57
  • $\begingroup$ I understand your point, but one can still reasonably guess an approx value. However, I think your suggestion of CV is totally valid! Let's use that. $\endgroup$
    – Cam.Davidson.Pilon
    Commented Apr 17, 2019 at 1:43

1 Answer 1

Reset to default
2
$\begingroup$

I personally am a big fan of regularization. Here are a few arguments against it:

  1. It's more complicated. There is a reason why we first teach OLS and only later regularization. Even the rationale for it, the bias-variance tradeoff, is very unintuitive to non-statisticians.

    Note that "more complicated" has three consequences: it means the model is harder to set up, harder to run (I agree with jbowman that regularizers are not at all trivial to configure), and harder to maintain. Maintenance matters sometimes. For instance, I am involved with building a pretty big software solution which has been and will be expanded over the years and will run for years or decades. Every new functionality has to tie in to existing functionality, and given the interconnections, maintenance complexity increases superlinearly.

  2. It increases runtime, especially if you need to cross-validate to calibrate your regularization. For instance, our software runs millions of models every day. Performance is important, because even if you can parallelize, additional cores do come with a cost. (And incidentally, yes, we do use regularization.)

In the end, it comes down to a tradeoff. Sometimes the benefits are worth the costs, sometimes they aren't. It all depends on your actual problem.

Improve this answer
$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.