Under which conditions do Bayesian and frequentist point estimators coincide?

Question

With a flat prior, the ML (frequentist -- maximum likelihood) and the MAP (Bayesian -- maximum a posteriori) estimators coincide.

More generally, however, I'm talking about point estimators derived as the optimisers of some loss function. I.e.

$$ \hat x(\,. ) = \text{argmin} \; \mathbb{E} \left( L(X-\hat x(y)) \; | \; y \right) \qquad \; \,\text{ (Bayesian) }$$ $$ \hat x(\,. ) = \text{argmin} \; \mathbb{E} \left( L(x-\hat x(Y)) \; | \; x \right) \qquad \text{(Frequentist)}$$

where $\mathbb{E}$ is the expectation operator, $L$ is the loss function (minimised at zero), $\hat x(y) $ is the estimator, given the data $y$, of the parameter $x$, and random variables are denoted with uppercase letters.

Does anybody know any conditions on $L$, the pdf of $x$ and $y$, imposed linearity and/or unbiasedness, where the estimators will coincide?

Edit

As noted in comments, an impartiality requirement such as unbiasedness is required to render the Frequentist problem meaningful. Flat priors may also be a commonality.

Besides the general discussions provided by some of the answers, the question is really also about providing actual examples. I think an important one comes from linear regression:

• the OLS, $\mathbf{\hat{x}} = (\mathbf{D}'\mathbf{D})^{-1}\mathbf{D}'\mathbf{y}$ is the BLUE (Gauss-Markov theorem), i.e. it minimises the frequentist MSE among linear-unbiased estimators.
• if $(X,Y)$ is Gaussian and the prior is flat, $\mathbf{\hat{x}} = (\mathbf{D}'\mathbf{D})^{-1}\mathbf{D}'\mathbf{y}$ is the "posterior" mean minimises the Bayesian mean loss for any convex loss function.

Here, $\mathbf{D}$ seems to be known as data/design matrix in the frequentist/Bayesian lingo, respectively.

Answer 1

The question is interesting but somewhat hopeless unless the notion of frequentist estimator is made precise. It is definitely not the one set in the question $$\hat x(\,. ) = \text{argmin} \; \mathbb{E} \left( L(x,\hat x(Y)) \; | \; x \right)$$ since the answer to the minimisation is $\hat{x}(y)=x$ for all $y$'s as pointed out in Programmer2134's answer. The fundamental issue is that there is no single frequentist estimator for an estimation problem, without introducing supplementary constraints or classes of estimators. Without those, all Bayes estimators are also frequentist estimators.

As pointed out in the comments, unbiasedness may be such a constraint, in which case Bayes estimators are excluded. But this frequentist notion clashes with other frequentist notions such as

1. admissibility, since the James-Stein phenomenon demonstrated that unbiased estimators may be inadmissible (depending on the loss function and on the dimension of the problem);
2. invariance under reparameterisation, since unbiasedness does not keep under transforms.

Plus unbiasedness only applies to a restricted class of estimation problems. By this, I mean that the class of unbiased estimators of a certain parameter $\theta$ or of a transform $h(\theta)$ is most of the time empty.

Speaking of admissibility, another frequentist notion, there exist settings for which the only admissible estimators are Bayes estimators and conversely. This type of settings relates to the complete class theorems established by Abraham Wald in the 1950's. (The same applies to the best invariant estimators which are Bayes under the appropriate right Haar measure.)

Answer 2

This is not a full answer, but while these two $\text{argmin}$'s look very similar, they are fundamentally different in a way: the Bayesian one minimizes the expression with respect to a single value (that is, the value of $\hat x(y)$, depending on $y$).

But the Frequentist one has to minimize the loss function with respect to a single value for every value that $x$ could take, without knowing $x$. This is because the minimum of the function $f(x,\hat x)=E(L(x-\hat x(Y))|x)$ depends on $x$, even though we have to minimize it without knowing $x$. (note that if we would simply minimize $f(x, \hat x)$ w.r.t. $\hat x$, we would simply get the minimizing value of $\hat x = x$.) The Frequentist problem is therefore undefined. I am not sure whether it is even possible to make it well-defined.

Answer 3

In general, frequentist and Bayesian estimators do not coincide, unless you use a degenerate flat prior. The main reason is this: Frequentist estimators often strive to be unbiased. For example, frequentists often try to find the minimum variance unbiased estimator (http://en.wikipedia.org/wiki/Minimum-variance_unbiased_estimator). Meanwhile, all non-degenerate Bayes estimators are biased (in the frequentist sense of bias). See, for example, http://www.stat.washington.edu/~hoff/courses/581/LectureNotes/bayes.pdf, Theorem 5.

To summarize: Most of the popular frequentist estimators strive to be unbiased, while all Bayes estimators are biased. Thus, Bayes and frequentist estimators rarely coincide.

Answer 4

There may exist no answer to this question.

An alternative could be to ask for methods to determine the two estimates efficiently for any problem at hand. The Bayesian methods are pretty close to this ideal. However, even though minimax methods could be used to determine the frequentist point estimate, in general, the application of the minimax method remains difficult, and tends not to be used in practice.

An other alternative would be to rephrase the question as to the conditions under which Bayesian and frequentist estimators provide “consistent” results and try to identify methods to efficiently calculate those estimators. Here "consistent" is taken to imply that Bayesian and frequentist estimators are derived from a common theory and that the same criterion of optimality is used for both estimators. This is very different from trying to oppose Bayesian and frequentist statistics, and may render the above question superfluous. One possible approach is to aim, both for the frequentist case and the Bayesian case, at decision sets that minimize the loss for a given size, i.e., as proposed by

Schafer, Chad M, and Philip B Stark. "Constructing confidence regions of optimal expected size." Journal of the American Statistical Association 104.487 (2009): 1080-1089.

It turns out that this is possible - both for the frequentist and the Bayesian case - by including by preference observations and parameters with large pointwise mutual information. The decision sets will not be identical, since the question being asked is different:

• Independent of what is the true parameter, limit the risk of making wrong decisions (the frequentist view)
• Given some observations, limit the risk of including wrong parameters into the decision set (Bayesian view)

However the sets will overlap largely and become identical in some situations, if flat priors are used. The idea is discussed in more detail together with an efficient impementation in

Bartels, Christian (2015): Generic and consistent confidence and credible regions. figshare. https://doi.org/10.6084/m9.figshare.1528163

For informative priors, the decision sets deviate more (as is commonly known and was pointed out in the question and in answers above). However within the consistent framework, one obtains frequentist tests, that guarantee the desired frequentist coverage, but take into account prior knowledge.

Bartels, Christian (2017): Using prior knowledge in frequentist tests. figshare. https://doi.org/10.6084/m9.figshare.4819597

The proposed methods still lack an efficient implementation of marginaization.