Frequentist vs. Bayesian bias-variance decomposition

Question

Iv'e read the answer to this related question and still have some issues.

Suppose that given some data $X$, we want an estimator $\hat{\theta}$ for some parameter $\theta$. A common approach is to minimize the expected $\ell_{2}$ loss$$\hat{\theta}_{\ell_{2}}=\underset{\hat{\theta}}{\arg\min}\left\langle \left(\hat{\theta}-\theta\right)^{2}\right\rangle $$

This expression admits the famous bias-variance decomposition. Assuming a Bayesian setting, and that the expectation is taken over the posterior distribution $p\left(\theta\mid X\right)$, this looks like

$$\left\langle \left(\hat{\theta}-\theta\right)^{2}\right\rangle _{p\left(\theta\mid X\right)}=\hat{\theta}^{2}-2\hat{\theta}\left\langle \theta\right\rangle +\left\langle \theta^{2}\right\rangle =\hat{\theta}^{2}-2\hat{\theta}\left\langle \theta\right\rangle +\left\langle \theta^{2}\right\rangle +\left\langle \theta\right\rangle ^{2}-\left\langle \theta\right\rangle ^{2}=\overbrace{\left(\hat{\theta}-\left\langle \theta\right\rangle \right)^{2}}^{b\left(\theta\right)^{2}}+\overbrace{\left(\left\langle \theta^{2}\right\rangle -\left\langle \theta\right\rangle ^{2}\right)}^{Var\left(p\left(\theta\mid X\right)\right)}$$ Which means that the optimal estimator is $$\hat{\theta}_{\ell_{2}}=\left\langle \theta\right\rangle _{p\left(\theta\mid x\right)}$$ However, we can also view $\theta$ as fixed, and the expectation over sampling $X\sim\theta$, i.e. the distribution $p\left(\hat{\theta}\left(X\right)\mid\theta\right)$, in which case the decomposition looks like $$\left\langle \left(\hat{\theta}-\theta\right)^{2}\right\rangle _{p\left(\hat{\theta}\left(X\right)\mid\theta\right)}=\overbrace{\left(\left\langle \hat{\theta}\right\rangle -\theta\right)^{2}}^{b\left(\hat{\theta}\right)^{2}}+\overbrace{\left(\left\langle \hat{\theta}^{2}\right\rangle -\left\langle \hat{\theta}\right\rangle ^{2}\right)}^{Var\left(\hat{\theta}\left(X\right)\right)}$$

Questions:

1. Is this analysis correct? If so, which one is the more "mainstream" approach, and what are the practical differences between these two views?

2. Is there a closed-form expression for the minimal MSE estimator in the frequentist setting?

Answer 1

1. Yes, your analysis is correct. For $l_2$, the optimal estimator in the Bayesian setting will always the posterior mean $\left\langle \theta\right\rangle _{p\left(\theta\mid x\right)}$. In the case of absolute loss we would have the posterior median and in the discrete case of constant loss we would have the posterior mode. Now, for which one is more mainstream, my gut feeling tells me that the frequentist one is still more popular as it is usually taught first and it's very intuitive. Nevertheless, I believe that probably this gap is narrower than it used to be. The practical differences, now this is a wide topic. The first thing to know is that the more data you have and/or the more vague your prior is then the results of the analyses converge. Now, the practical differences start to emerge when you depart from the 2 things I mentioned before. Especially for the prior. The main advantage and at the same time drawback of the Bayesian analysis is when you start to include your personal beliefs and you make your prior less vague. In that case you may end up with very subjective/questionable/unreliable results or sometimes much more accurate ones! For example, here is how a Bayesian approach (via an informative prior) helped the location of the a plane crash. My suggestion would be to know and understand both and pick the one that suits best the problem.

2. For a closed form of the Frequentist Minimial MSE, I'm afraid that there is not one for all distributions because of the nature of the problem. What I mean is that, under mild regularity conditions, the variance will always be higher or equal than the Cramér-Rao Lower Bound which makes it similar to the Uncertainty principle in physics: the more you try to lower one quantity the other one increases. So, what we usually try to do is to fix one quantity (the bias to 0 usually) and we try to minimise the variance. That said, the only thing that I can think of which is similar to a "closed form" (roughly speaking) is a theorem that says that: There is an estimator $\hat{g}(\textbf{X})$ of $g(\theta)$ Which attains the Cramér-Rao Lower Bound if and only if we can write $U(\textbf{X}) = \frac{\partial \log_{p_{\textbf{x}}}(\textbf{X}\vert\theta)}{\partial \theta}$ in the form $U(\textbf{X}) = \alpha(\theta)g(\hat{\theta}) +\beta(\theta)$ where $\alpha$ and $\beta$ are arbitrary functions of $\theta$.