I'm from a machine learning background and trying to brush up on my stats. I've read a few papers on estimation theory. In a nutshell, estimation theory is: given some probability model, we want to approximate the parameters for this model.

In machine learning, everyone is scared of overfitting and such. Why isn't overfitting brought up more in classical estimation theory? Do classical statisticians not care about overfitting?

  • 2
    $\begingroup$ I think you already have the answer, which is that in the classical theory of estimation a parametric model is often assumed, and so the question of overfitting is not relevant. It isn't that statisticians don't care about overfitting, it's just that this is more of an applied concern that comes up when we don't know the underlying data generating process. $\endgroup$ – dsaxton Mar 14 '16 at 18:14
  • $\begingroup$ I see. I just wanted to confirm. Estimation theory is essentially an unsupervised process. $\endgroup$ – user46925 Mar 14 '16 at 18:17
  • $\begingroup$ How do researchers determine whether their model is good? $\endgroup$ – user46925 Mar 14 '16 at 18:28
  • $\begingroup$ Wait - why can't we just hold out a testing partition of the sample and test whether the distribution is a good fit? $\endgroup$ – user46925 Mar 14 '16 at 18:31
  • 1
    $\begingroup$ Is that a new question? $\endgroup$ – Glen_b -Reinstate Monica Mar 15 '16 at 0:16

Talking of "estimation" rather than "prediction" already frames the problem as having to do with an assumed parametric model, as @dsaxton says. But you might choose between, say, an exponential & a Weibull model for some univariate observations according to which you think will better predict future observations based on cross-validation or Akaike's information criterion—& justly describe the Weibull, with its additional parameter, as overfitting when you think it will predict worse. Even so, the sample size would have to be quite small before any deleterious effect of fitting the couple of extra parameters at most that differentiate typical brand-name distributions became worrisome.

You can think of multiple regression as fitting a many-parameter probability distribution for the response conditional on the predictors. Here too there's a distinction between the case where you're primarily concerned with estimation of a pre-defined set of parameters, which are perhaps of interest in themselves, & the case where you're primarily concerned with predictive performance, & happy to reduce the parameter set or penalize estimates to improve it.

  • $\begingroup$ Thanks this helps. I come from an ML background and this is super nice! $\endgroup$ – user46925 Mar 22 '16 at 10:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy