17
$\begingroup$

With a flat prior, the ML (frequentist -- maximum likelihood) and the MAP (Bayesian -- maximum a posteriori) estimators coincide.

More generally, however, I'm talking about point estimators derived as the optimisers of some loss function. I.e.

$$ \hat x(\,. ) = \text{argmin} \; \mathbb{E} \left( L(X-\hat x(y)) \; | \; y \right) \qquad \; \,\text{ (Bayesian) }$$ $$ \hat x(\,. ) = \text{argmin} \; \mathbb{E} \left( L(x-\hat x(Y)) \; | \; x \right) \qquad \text{(Frequentist)}$$

where $\mathbb{E}$ is the expectation operator, $L$ is the loss function (minimised at zero), $\hat x(y) $ is the estimator, given the data $y$, of the parameter $x$, and random variables are denoted with uppercase letters.

Does anybody know any conditions on $L$, the pdf of $x$ and $y$, imposed linearity and/or unbiasedness, where the estimators will coincide?

Edit

As noted in comments, an impartiality requirement such as unbiasedness is required to render the Frequentist problem meaningful. Flat priors may also be a commonality.

Besides the general discussions provided by some of the answers, the question is really also about providing actual examples. I think an important one comes from linear regression:

  • the OLS, $\mathbf{\hat{x}} = (\mathbf{D}'\mathbf{D})^{-1}\mathbf{D}'\mathbf{y}$ is the BLUE (Gauss-Markov theorem), i.e. it minimises the frequentist MSE among linear-unbiased estimators.
  • if $(X,Y)$ is Gaussian and the prior is flat, $\mathbf{\hat{x}} = (\mathbf{D}'\mathbf{D})^{-1}\mathbf{D}'\mathbf{y}$ is the "posterior" mean minimises the Bayesian mean loss for any convex loss function.

Here, $\mathbf{D}$ seems to be known as data/design matrix in the frequentist/Bayesian lingo, respectively.

$\endgroup$
  • $\begingroup$ I assume that you want the answer to assume a flat prior? Otherwise of course there is no way that the estimates could be reasonable expected to be the same in interesting general cases. $\endgroup$ – user56834 Nov 13 '17 at 5:04
  • 2
    $\begingroup$ It is not a simple question to answer in the generality you pose it, but it is currently a really hot research topic, see for instance Judith Rousseau's work in this area: ceremade.dauphine.fr/~rousseau/publi.html $\endgroup$ – Jeremias K Nov 17 '17 at 11:45
  • $\begingroup$ @JeremiasK, maybe you can explain something about that in an answer? $\endgroup$ – user56834 Nov 19 '17 at 7:40
  • 1
    $\begingroup$ @Programmer2134 I would if I felt comfortable enough with the material, but I don't. I know that what they do is deriving a Bayesian counterpart of a CLT, with certain 'posterior concentration rates' that tell you how fast the parameter posterior concentrates on a point in your parameter space as you increase the sample size, and then you basically end up finding frequentist-type consistency guarantees for your Bayesian estimators. $\endgroup$ – Jeremias K Nov 19 '17 at 15:28
7
+100
$\begingroup$

The question is interesting but somewhat hopeless unless the notion of frequentist estimator is made precise. It is definitely not the one set in the question $$\hat x(\,. ) = \text{argmin} \; \mathbb{E} \left( L(x,\hat x(Y)) \; | \; x \right)$$ since the answer to the minimisation is $\hat{x}(y)=x$ for all $y$'s as pointed out in Programmer2134's answer. The fundamental issue is that there is no single frequentist estimator for an estimation problem, without introducing supplementary constraints or classes of estimators. Without those, all Bayes estimators are also frequentist estimators.

As pointed out in the comments, unbiasedness may be such a constraint, in which case Bayes estimators are excluded. But this frequentist notion clashes with other frequentist notions such as

  1. admissibility, since the James-Stein phenomenon demonstrated that unbiased estimators may be inadmissible (depending on the loss function and on the dimension of the problem);
  2. invariance under reparameterisation, since unbiasedness does not keep under transforms.

Plus unbiasedness only applies to a restricted class of estimation problems. By this, I mean that the class of unbiased estimators of a certain parameter $\theta$ or of a transform $h(\theta)$ is most of the time empty.

Speaking of admissibility, another frequentist notion, there exist settings for which the only admissible estimators are Bayes estimators and conversely. This type of settings relates to the complete class theorems established by Abraham Wald in the 1950's. (The same applies to the best invariant estimators which are Bayes under the appropriate right Haar measure.)

$\endgroup$
  • 1
    $\begingroup$ Are there other canonical ways of restricting the class of estimators so that the minimization problem is well defined and not degenerate (other than requiring unbiasedness), which are closer to Bayesian one? $\endgroup$ – user56834 Nov 16 '17 at 11:14
3
$\begingroup$

In general, frequentist and Bayesian estimators do not coincide, unless you use a degenerate flat prior. The main reason is this: Frequentist estimators often strive to be unbiased. For example, frequentists often try to find the minimum variance unbiased estimator (http://en.wikipedia.org/wiki/Minimum-variance_unbiased_estimator). Meanwhile, all non-degenerate Bayes estimators are biased (in the frequentist sense of bias). See, for example, http://www.stat.washington.edu/~hoff/courses/581/LectureNotes/bayes.pdf, Theorem 5.

To summarize: Most of the popular frequentist estimators strive to be unbiased, while all Bayes estimators are biased. Thus, Bayes and frequentist estimators rarely coincide.

$\endgroup$
  • 5
    $\begingroup$ I wonder about the correctness of these assertions, given that "most of the popular frequentist estimators" are ML and they tend to be biased (depending on the parameterization). Moreover, a good frequentist is deeply concerned about loss and admissibility; a key part of this theory recognizes that admissible procedures come from Bayes procedures, whence--at least in that broad sense--the very heart of frequentist theory relies on Bayes estimators! I might be persuaded to your point of view if you could be clearer about "often," "most," and "rarely," and back that up with evidence. $\endgroup$ – whuber Apr 25 '13 at 19:39
  • $\begingroup$ @whuber Good point - my answer was perhaps a bit simplistic. Real frequentists tend to use biased procedures (e.g. L1 or L2 penalized regression), or may even use formally Bayesian procedures. However, I think unbiased estimators are the starting point for most frequentist analysis. For example, the first meaty chapter of Theory of Point Estimation by Lehmann & Casella (one of the standard texts on frequentist estimation) is all about unbiasedness. $\endgroup$ – Stefan Wager Apr 25 '13 at 20:31
  • 5
    $\begingroup$ Well, OK (+1). But I find your last argument amusing: after all, a book has to start somewhere and usually that starting point is chosen for its simplicity and accessibility, not for its practical importance. By the same reasoning you could claim that most modern mathematics is primarily concerned with logic and set theory, because those often form the first chapter in many math textbooks! A better reflection of statistical practice might be the last half or so of Lehmann & Casella--take a look at what's discussed there :-). $\endgroup$ – whuber Apr 25 '13 at 22:09
  • $\begingroup$ "unless you use a degenerate flat prior". Well this is an interesting special case to think about, isn't it? $\endgroup$ – user56834 Nov 13 '17 at 5:05
  • $\begingroup$ Also, his question is about whether they would theoretically coincide under certain conditions, not whether the estimators that are used in practice coincide. $\endgroup$ – user56834 Nov 13 '17 at 5:07
3
$\begingroup$

This is not a full answer, but while these two $\text{argmin}$'s look very similar, they are fundamentally different in a way: the Bayesian one minimizes the expression with respect to a single value (that is, the value of $\hat x(y)$, depending on $y$).

But the Frequentist one has to minimize the loss function with respect to a single value for every value that $x$ could take, without knowing $x$. This is because the minimum of the function $f(x,\hat x)=E(L(x-\hat x(Y))|x)$ depends on $x$, even though we have to minimize it without knowing $x$. (note that if we would simply minimize $f(x, \hat x)$ w.r.t. $\hat x$, we would simply get the minimizing value of $\hat x = x$.) The Frequentist problem is therefore undefined. I am not sure whether it is even possible to make it well-defined.

$\endgroup$
  • 1
    $\begingroup$ Good points. I think you're right about the frequentist problem. The way to render it well-posed is to restrict the class of estimators. From Lehmann & Casella: "So far, we have been concerned with finding estimators which minimize the risk R(θ,δ) at every value of θ. This was possible only by restricting the class of estimators to be considered by an impartiality requirement such as unbiasedness or equivariance." $\endgroup$ – Patrick Nov 14 '17 at 10:34
1
$\begingroup$

There may exist no answer to this question.

An alternative could be to ask for methods to determine the two estimates efficiently for any problem at hand. The Bayesian methods are pretty close to this ideal. However, even though minimax methods could be used to determine the frequentist point estimate, in general, the application of the minimax method remains difficult, and tends not to be used in practice.

An other alternative would be to rephrase the question as to the conditions under which Bayesian and frequentist estimators provide “consistent” results and try to identify methods to efficiently calculate those estimators. Here "consistent" is taken to imply that Bayesian and frequentist estimators are derived from a common theory and that the same criterion of optimality is used for both estimators. This is very different from trying to oppose Bayesian and frequentist statistics, and may render the above question superfluous. One possible approach is to aim, both for the frequentist case and the Bayesian case, at decision sets that minimize the loss for a given size, i.e., as proposed by

Schafer, Chad M, and Philip B Stark. "Constructing confidence regions of optimal expected size." Journal of the American Statistical Association 104.487 (2009): 1080-1089.

It turns out that this is possible - both for the frequentist and the Bayesian case - by including by preference observations and parameters with large pointwise mutual information. The decision sets will not be identical, since the question being asked is different:

  • Independent of what is the true parameter, limit the risk of making wrong decisions (the frequentist view)
  • Given some observations, limit the risk of including wrong parameters into the decision set (Bayesian view)

However the sets will overlap largely and become identical in some situations, if flat priors are used. The idea is discussed in more detail together with an efficient impementation in

Bartels, Christian (2015): Generic and consistent confidence and credible regions. figshare. https://doi.org/10.6084/m9.figshare.1528163

For informative priors, the decision sets deviate more (as is commonly known and was pointed out in the question and in answers above). However within the consistent framework, one obtains frequentist tests, that guarantee the desired frequentist coverage, but take into account prior knowledge.

Bartels, Christian (2017): Using prior knowledge in frequentist tests. figshare. https://doi.org/10.6084/m9.figshare.4819597

The proposed methods still lack an efficient implementation of marginaization.

$\endgroup$
  • $\begingroup$ Could you elaborate in your question more specifically when they would be "consistent"? $\endgroup$ – user56834 Nov 19 '17 at 17:34
  • $\begingroup$ @Programmer2134. Thanks, tried to clarify in the answer. $\endgroup$ – user36160 Nov 20 '17 at 6:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.