I am referring to the family of estimation techniques like MLEs, least-squares, etc., that an l2 penalizer/regularizer can be added to. I'm not interested in NHST, but just estimation (say, of some causal effect or association).
The way I see it is that adding a penalizer term does cause a bias (though MLEs are often already biased...), but there are more gains:
- the estimator is still consistent,
- the estimator has lower variance,
- the estimator can deal with co-linearity and separation problems,
- allows some expression of prior knowledge¹
Of course, adding too large of a penalizer will significantly bias results, but a practitioner should know a sensible value (and probably decided on beforehand).
What am I missing? Why should I not always added a small penalizer to my MLE models? Are my confidence intervals (I can't really call them confidence intervals anymore...) drastically broken if I do add a penalizer?
¹ Without going full Bayesian, adding a small penalizer tells the model "yeaaaa, 1e18 is not a likely value".