Do the concepts of regularization and overfitting apply in probability distribution fitting? I'm from a machine learning background and trying to brush up on my stats. I've read a few papers on estimation theory. In a nutshell, estimation theory is: given some probability model, we want to approximate the parameters for this model. 
In machine learning, everyone is scared of overfitting and such. Why isn't overfitting brought up more in classical estimation theory? Do classical statisticians not care about overfitting?
 A: Talking of "estimation" rather than "prediction" already frames the problem as having to do with an assumed parametric model, as @dsaxton says. But you might choose between, say, an exponential & a Weibull model for some univariate observations according to which you think will better predict future observations based on cross-validation or Akaike's information criterion—& justly describe the Weibull, with its additional parameter, as overfitting when you think it will predict worse. Even so, the sample size would have to be quite small before any deleterious effect of fitting the couple of extra parameters at most that differentiate typical brand-name distributions became worrisome.
You can think of multiple regression as fitting a many-parameter probability distribution for the response conditional on the predictors. Here too there's a distinction between the case where you're primarily concerned with estimation of a pre-defined set of parameters, which are perhaps of interest in themselves, & the case where you're primarily concerned with predictive performance, & happy to reduce the parameter set or penalize estimates to improve it.
