5
$\begingroup$

Let's suppose that I have a quantity/parameter $\theta\in\Theta$ that I would like to make inferences about. I am able to draw data, say $X = (X_1,\ldots,X_n)$ I.I.D. from a distribution $P(x;\theta)$. I have a good handle on my data likelihood since it relies on trusted physics models.

I'm also willing to admit that $\theta$ is a random variable, but I have no good reason to assume that I know what a good prior distribution is - imagine that $\theta$ is very high dimensional and complex, and little is really known about its statistics - wild guesses have been tried but they all seem suspicious.

So, for lack of a good prior, I'd like to consider purely likelihood-based inference procedures, just to get started - i.e. I'll take a frequentist/likelihoodist approach to my inference - maximum likelihood estimators, confidence intervals, Fisher information, etc. So, suppose I've selected an estimation rule $\hat{\theta} = \delta(X)$.

Given this estimator, I can test its performance by choosing different priors $\pi(\theta)$, then evaluating something like a Bayes risk that depends on both the prior and the decision rule:

$$ E_\pi[L(\theta,\hat{\theta})] = \int_{\Theta}L(\theta,\hat{\theta})d\pi(\theta) $$Of course, as written, this Bayes risk is actually a random variable, since $\hat{\theta} = \delta(X)$ and $X$ is random; to deal with this I could either take the average over $X\vert \theta$ using the likelihood (fixed number of samples), or I could take a large-sample limit. At any rate, my definition of risk thus depends on both the prior and the decision rule, and the decision rule was formed independently of the prior used to test it. Thus on one hand I'd like to try forming an estimator independently of any prior, then I'd like to evaluate the resulting procedure using priors.

My question is: does this procedure make sense, and if so does it have a precedent/name in the statistics literature? I'm more of a math/physics person so I'm less familiar with the stats literature.

$\endgroup$
2
$\begingroup$

Using the quantity $$\mathbb{E}_\pi[L(\theta,\hat{\theta})] = \int_{\Theta}L(\theta,\hat{\theta})\text{d}\pi(\theta)$$ to evaluate the prior $\pi$ is not meaningful as such, but there may be interesting directions in this question:

  1. The "best" prior is the Dirac mass at $\hat{\theta}$
  2. The integral should be against $\pi(\theta|X)$ since $X$ has been observed
  3. If one wants to integrate $X$ out the equality $$\int_\mathcal{X}\int_{\Theta}L(\theta,\hat{\theta}(x))\text{d}\pi(\theta|x)\text{d}m(x)=\int_{\Theta}\int_{\mathcal{X}}L(\theta,\hat{\theta}(x))\text{d}f(x|\theta)\text{d}\pi(\theta)$$ (Fubini's theorem) shows the order does not matter.
  4. For the integrated risk above, the question gets more interesting.

For instance, in the normal mean N$(\theta,1)$ case, the integrated risks associated with conjugate priors on $\theta$ (and prior mean zero) look like \begin{align*}\mathbb{E}_m[\text{var}(\theta|X)]+\mathbb{E}_m[(\mathbb{E}(\theta|X)-X)^2]&=\dfrac{b}{1+b}+\mathbb{E}_m\left[\left(\dfrac{X}{1+b}\right)^2\right]\\&=\dfrac{b}{1+b}+\dfrac{1}{(1+b)^2}(1+b)\\ &=\dfrac{b}{1+b}+\dfrac{1}{(1+b)}=1\\\end{align*} which is obvious if $X$ is first integrated out since $$\mathbb{E}[(\theta-X)^2]=\mathbb{E}^\pi[\mathbb{E}^X[(\theta-X)^2]]=\mathbb{E}^\pi[1]=1$$ This result means these priors cannot be compared with respect to this measure.

$\endgroup$
  • 1
    $\begingroup$ I suppose I was imagining using $\Bbb{E}_\pi[L(\theta,\hat{\theta})]$ to evaluate the combination of estimator ($\hat{\theta}$) and prior, i.e. not evaluating the prior itself per se, but seeing how well an estimator constructed in some way (using a different prior or "no" prior) does against different choices of prior. I'm starting to think that what I'm really interested in is the issue of sensitivity and prior misspecification $\endgroup$ – icurays1 Aug 16 '17 at 15:38
  • $\begingroup$ In some settings, like the Normal mean one, there are Bayes estimates equal to $\hat{\theta}(x)$. $\endgroup$ – Xi'an Aug 16 '17 at 15:40
  • 1
    $\begingroup$ Just out of curiosity, does the integrated Bayes risk $\Bbb{E}_\pi[\Bbb{E}_{x\vert\theta} [ L(\theta,\hat{\theta})]]$ have an "official" name in the statistics literature? I'd like to call it the correct thing in my dissertation. $\endgroup$ – icurays1 Aug 20 '17 at 16:10
  • $\begingroup$ In my book and others, it is called the Bayes risk rather than the integrated risk. When conditioning upon $X$, it is called the posterior loss. $\endgroup$ – Xi'an Aug 20 '17 at 17:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.