Should MLE estimation always be using penalizers? I am referring to the family of estimation techniques like MLEs, least-squares, etc., that an l2 penalizer/regularizer can be added to. I'm not interested in NHST, but just estimation (say, of some causal effect or association).
The way I see it is that adding a penalizer term does cause a bias (though MLEs are often already biased...), but there are more gains:


*

*the estimator is still consistent, 

*the estimator has lower variance, 

*the estimator can deal with co-linearity and separation problems, 

*allows some expression of prior knowledge¹


Of course, adding too large of a penalizer will significantly bias results, but a practitioner should know a sensible value (and probably decided on beforehand). 
What am I missing? Why should I not always added a small penalizer to my MLE models? Are my confidence intervals (I can't really call them confidence intervals anymore...) drastically broken if I do add a penalizer? 
¹ Without going full Bayesian, adding a small penalizer tells the model "yeaaaa, 1e18 is not a likely value". 
 A: I personally am a big fan of regularization. Here are a few arguments against it:

*

*It's more complicated. There is a reason why we first teach OLS and only later regularization. Even the rationale for it, the bias-variance tradeoff, is very unintuitive to non-statisticians.
Note that "more complicated" has three consequences: it means the model is harder to set up, harder to run (I agree with jbowman that regularizers are not at all trivial to configure), and harder to maintain. Maintenance matters sometimes. For instance, I am involved with building a pretty big software solution which has been and will be expanded over the years and will run for years or decades. Every new functionality has to tie in to existing functionality, and given the interconnections, maintenance complexity increases superlinearly.


*It increases runtime, especially if you need to cross-validate to calibrate your regularization. For instance, our software runs millions of models every day. Performance is important, because even if you can parallelize, additional cores do come with a cost. (And incidentally, yes, we do use regularization.)
In the end, it comes down to a tradeoff. Sometimes the benefits are worth the costs, sometimes they aren't. It all depends on your actual problem.
