25
$\begingroup$

Frequentist statistics for me is synonymous for trying to make decision that are good for all possible samples. I.e., a frequentist decision rule $\delta$ should always try to minimize the frequentist risk, which depends on a loss function $L$ and the true state of nature $\theta_0$:

$$R_\mathrm{freq}=\mathbb{E}_{\theta_0}(L(\theta_0,\delta(Y))$$

How is maximum likelihood estimation connected to the frequentist risk? Given that it is the most used point estimation technique used by frequentists there must be some connection. As far as I know, maximum likelihood estimation is older than the concept of frequentist risk but still there must be some connection why else would so many people claim that it is a frequentist technique?

The closest connection that I have found is that

"For parametric models that satisfy weak regularity conditions, the maximum likelihood estimator is approximately minimax" Wassermann 2006, p. 201"

The accepted answer either links maximum likelihood point estimation stronger to the frequentist risk or provides an alternative formal definition of frequentist inference that shows that MLE is a frequentist inference technique.

Improve this question
$\endgroup$
4
  • 8
    $\begingroup$ ML pays no attention to risk at all! That, in fact, is part of the frequentist decision-theoretic criticism of ML. I suspect this question may be difficult to answer because it implicitly uses "Frequentist" in two incompatible senses--one is decision-theoretic, referring to a loss function, and the other implicitly refers to not assuming a prior distribution. $\endgroup$
    – whuber ♦
    Commented Jan 14, 2016 at 15:35
  • $\begingroup$ @whuber ML pays attention to the risk. In fact it is minimization under logarithmic loss under an improper uniform prior. $\endgroup$
    – Cagdas Ozgenc
    Commented Jan 14, 2016 at 17:30
  • 5
    $\begingroup$ @Cagdas I believe that's not usually the risk for a decision-maker: it merely exhibits ML as if it were minimizing the risk if logarithmic loss were the risk that mattered to them. Appealing to an "improper uniform prior" is decidedly non-frequentist, by the way! $\endgroup$
    – whuber ♦
    Commented Jan 14, 2016 at 18:15
  • 1
    $\begingroup$ @whuber Bayesian estimation procedures are also using accumulated log-loss. Only after that the decision maker risk is applied. If we are talking about optimizing the decision maker risk directly (not via a log-loss stepping stone), then frequentist procedures are more famous on that respect, i.e. OLS. $\endgroup$
    – Cagdas Ozgenc
    Commented Jan 15, 2016 at 6:23

2 Answers 2

Reset to default
21
+50
$\begingroup$

You apply a relatively narrow definition of frequentism and MLE - if we are a bit more generous and define

  • Frequentism: goal of consistency, (asymptotic) optimality, unbiasedness, and controlled error rates under repeated sampling, independent of the true parameters

  • MLE = point estimate + confidence intervals (CIs)

then it seems pretty clear that MLE satisfies all frequentist ideals. In particular, CIs in MLE, as p-values, control the error rate under repeated sampling, and do not give the 95% probability region for the true parameter value, as many people think - hence they are through and through frequentist.

Not all of these ideas were already present in Fisher's foundational 1922 paper "On the mathematical foundations of theoretical statistics", but the idea of optimality and unbiasedness is, and Neyman latter added the idea of constructing CIs with fixed error rates. Efron, 2013, "A 250-year argument: Belief, behavior, and the bootstrap", summarizes in his very readable history of the Bayesian/Frequentist debate:

The frequentist bandwagon really got rolling in the early 1900s. Ronald Fisher developed the maximum likelihood theory of optimal estimation, showing the best possible behavior for an estimate, and Jerzy Neyman did the same for confidence intervals and tests. Fisher’s and Neyman’s procedures were an almost perfect fit to the scientific needs and the computational limits of twentieth century science, casting Bayesianism into a shadow existence.

Regarding your more narrow definition - I mildly disagree with your premise that minimization of frequentist risk (FR) is the main criterion to decide if a method follows frequentist philosophy. I would say the fact that minimizing FR is a desirable property follows from frequentist philosophy, rather than preceding it. Hence, a decision rule / estimator does not have to minimize FR to be frequentist, and minimizing FR is also does not necessarily say that a method is frequentist, but a frequentist would in doubt prefer minimization of FR.

If we look at MLE specifically: Fisher showed that MLE is asymptotically optimal (broadly equivalent to minimizing FR), and that was certainly one reason for promoting MLE. However, he was aware that optimality did not hold for finite sample size. Still, he was happy with this estimator due to other desirable properties such as consistency, asymptotic normality, invariance under parameter transformations, and let's not forget: ease to calculate. Invariance in particular is stressed abundantly in the 1922 paper - from my reading, I would say maintaining invariance under parameter transformation, and the ability to get rid of the priors in general, were one of his main motivations in choosing MLE. If you want to understand his reasoning better, I really recommend the 1922 paper, it's beautifully written and he explains his reasoning very well.

Improve this answer
$\endgroup$
2
  • 2
    $\begingroup$ Could I summarize your answer as maximum likelihood point estimation is most often used in conjunction with CIs or as part of a hypothesis test (e.g. a likelihood ration test), therefore, it is a frequentist technique? If this is the case, I think this is a valid answer, however not the one that I was hoping for. I was aiming for a formal argument why maximum likelihood estimation can be considered a frequentist point estimation technique. If this requires another formal definition of frequentist inference this is fine too. $\endgroup$
    – Julian Karch
    Commented Jan 15, 2016 at 12:11
  • 1
    $\begingroup$ I generally think of MLE as a framework that includes Fisher's point estimates together with Neyman's CIs - this is how it's taught in class, and due to the arguments above, I would maintain it is frequentist to the bone. I wonder how much sense it makes to discuss if MLE alone is a frequentist estimator, without the context of how and why it is used. If you want Fisher's reasons, I really recommend the 1922 paper - I would say the reasons he states are frequentist, although this word did not exist back then. I have extended my comment in that regard. $\endgroup$
    – Florian Hartig
    Commented Jan 16, 2016 at 14:32
1
$\begingroup$

Basically, for two reasons:

  • Maximum likelihood is a point-wise estimate of the model parameters. We Bayesians like posterior distributions.
  • Maximum likelihood assumes no prior distribution, We Bayesians need our priors, it could be informative or uninformative, but it need to exists
Improve this answer
$\endgroup$
4
  • 9
    $\begingroup$ +1 I just wish to point out that you implicitly appear to equate "frequentist" with "non-Bayesian" in this answer. The language of "We Bayesians" also suggests that "Bayesian" refers to some kind of personal characteristic or tribe membership--almost as if you were a kind of Eskimo--rather than a set of techniques and interpretations. $\endgroup$
    – whuber ♦
    Commented Jan 16, 2016 at 21:02
  • 4
    $\begingroup$ An the other hand MLE can easily be derived as a Bayesian technique. It is simply the MAP estimate for any statistical model using a uniform prior. $\endgroup$
    – Julian Karch
    Commented Jan 17, 2016 at 14:58
  • 3
    $\begingroup$ MAP is also a point-wise estimation, and is frowned upon by "True Bayesians" $\endgroup$
    – Uri Goren
    Commented Jan 17, 2016 at 15:12
  • $\begingroup$ I would say that ML is neither frequentist (since it doesn't give exact frequentist coverage) nor Bayesian (since it doesn't use Bayes' theorem). But it can be seen as a zeroth order approximation to both frequentist and Bayesian methods. "Frequentist" refers to properties of a prediction, and "Bayesian" refers to methodology, so one might expect some methods to be both, and, indeed, some are. For example, using the t distribution to predict future samples from a normal can be see as either frequentist or Bayesian. $\endgroup$
    – Stephen Jewson
    Commented Aug 21, 2023 at 16:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.