3
$\begingroup$

I am revisiting some self-study assignment questions in elementary theoretical statistics that I previously had difficulty with. I would appreciate some clarity on a few points in the following argument for computing a Bayes estimator by minimising posterior risk.

Context.

Given independent and identically distributed $X_1 \dots, X_n \sim N(\theta, 1)$, and a prior $\pi(\theta)$, find the Bayes estimator for $\tau = e^{\theta} + 1$ under the loss function $$L(\tau, \hat{\tau}) = \frac{(\hat{\tau} - \tau)^2}{\tau}$$ where $\hat{\tau}$ is some estimator.

My attempt.

Denoting the observed data $X_1 = x_1 \dots, X_n = x_n$ as $x^n$, and using the fact that the Bayes estimator $\hat{\tau}_B$ minimises the posterior risk $r(\hat{\tau} | x^n)$ we can be lazy statisticians and do

$$\hat{\tau}_B = \min_{\hat{\tau}} r(\hat{\tau} | x^n) = \min_{\hat{\tau}} \mathbb{E}_{p(\theta | x^n)}[L(\tau(\theta), \hat{\tau})] = \min_{\hat{\tau}} \int L(\tau(\theta), \hat{\tau}) p(\theta | x^n) d\theta$$

Computing derivatives with respect to $\hat{\tau}$ yields

\begin{align*} \frac{d}{d\hat{\tau}} \int \frac{(\hat{\tau} - \tau)^2}{\tau} p(\theta | x^n) \space d\theta &= \int \frac{\partial}{\partial \hat{\tau}} \frac{(\hat{\tau} - \tau)^2}{\tau} p(\theta | x^n) \space d\theta \tag{*}\\ &= 2 \int \left( \frac{\hat{\tau} - \tau}{\tau} \right) p(\theta | x^n) \space d \theta \\ &= 2 \left( \hat{\tau} \int \frac{1}{\tau} p(\theta | x^n) \space d\theta - \int p(\theta | x^n) \space d\theta \right) \end{align*}

Assuming that the posterior distribution $p(\theta | x^n)$ is appropriately normalised, setting the above to 0 and solving for $\hat{\tau}$ yields

$$\hat{\tau}_B = \frac{1}{\int [1 / \tau(\theta)] \cdot p(\theta | x^n) \space d \theta}$$

Queries.

Assuming that the routine manipulations I have carried out are not erroneous, my queries concerning $(*)$ above are:

1. The use of a total derivative $\frac{d}{d \hat{\tau}}$ instead of a partial derivative for minimisation purposes. I reasoned that $\mathbb{E}_{p(\theta | x^n)}[L(\tau(\theta), \hat{\tau})]$ can only freely vary in $\hat{\tau}$, and hence this was appropriate (instead of a partial derivative). Is that a valid assessment?

2. When the total derivative moves into the integration, and the order of limits and integration is interchanged thereby becoming the integral of a partial derivative $\int \frac{\partial}{\partial \hat{\tau}} \dots$, am I correct in understanding that an informal reason for the partial derivative without recourse to technicality is due to the fact that we are doing $\int \frac{\partial}{\partial \hat{\tau}} g(\tau(\theta), \hat{\tau}) \space d\theta$, and that the reason for the partial as opposed to total derivative is due to the additional $\tau(\theta)$ argument in $g$?

3. Does the assumption that one can globally minimise the posterior risk by setting the total derivative to 0, without any further investigation of higher-order derivatives, amount to the assumption that the posterior risk is convex? If that is the case, given that the loss $L(\tau(\theta), \hat{\tau})$ is explicitly specified, and the likelihood $p(x^n | \theta)$ is also specified within the posterior $p(\theta | x^n)$, would this depend on the unspecified functional form of the prior $\pi(\theta)$?

$\endgroup$
7
  • 1
    $\begingroup$ The easier route is to notice that the loss amounts to changing the prior into $\pi(\tau)/\tau$. $\endgroup$
    – Xi'an
    Apr 26, 2021 at 15:14
  • 1
    $\begingroup$ Bayes estimator of $\tau(\theta)$ should be (see stats.stackexchange.com/q/358241/119261) $$\frac{E\left[\frac1{\theta}\tau(\theta)\mid \boldsymbol X\right]}{E\left[\frac1{\theta}\mid \boldsymbol X\right]}$$ $\endgroup$ Apr 26, 2021 at 15:39
  • $\begingroup$ @StubbornAtom. Thank you for indexing the appropriate post - I would not have found it without your assistance as I didn't even know how to articulate what I was looking for. Having worked through the post, I've now edited my question to include workings. But I'm unable to reconcile them with the solution you've stated in the comments. $\endgroup$
    – microhaus
    Apr 26, 2021 at 23:40
  • $\begingroup$ @Xi'an. I spent at least an hour or so trying to make sense of your hint (which still remains cryptic to me). In context of what I know, I was trying to work out whether it was alluding to either of i) minimising the Bayes risk $B_{\pi}(\hat{\tau})$ with respect to a prior $\pi(\tau) / \tau$ or ii) minimising a new posterior risk under squared error loss whereby the weights have been absorbed and renormalised to form the new posterior similar to what StubbornAtom linked to? Are there pedagogical reasons for going down the 'easier' route than dealing with the queries I have specified? $\endgroup$
    – microhaus
    Apr 26, 2021 at 23:58
  • 1
    $\begingroup$ @StubbornAtom. I understand the part that I have a weighted square error loss $\frac{(\hat{\tau} - \tau)^2}{\tau}$ with weight $\frac{1}{\tau(\theta)}$. What I'm unable to understand is why applying the results of the thread does not yield the following Bayes estimator for $\tau(\theta)$: $$\frac{\mathbb{E} \left[\frac{1}{\tau(\theta)} \cdot \tau(\theta) \space | \space \mathbf{X} \right]}{\mathbb{E} \left[\frac{1}{\tau(\theta)} \space | \space \mathbf{X} \right]}$$ rather than what you've specified. $\endgroup$
    – microhaus
    Apr 27, 2021 at 12:08

1 Answer 1

0
$\begingroup$

Solution.

Using the results in here linked by StubbornAtom.

The loss function you have specified can be interpreted as a generalisation of squared error loss, known as weighted squared error loss. That is,

$$L(\tau, \hat{\tau}) = \frac{1}{\tau} \cdot (\hat{\tau} - \tau)^2,$$

where $w(\theta) = 1 / \tau(\theta)$ is a weight, and $L'(\tau, \hat{\tau}) = (\hat{\tau} - \tau)^2$ is squared error loss.

In this case, the Bayes estimator $\hat{\tau}_B$ minimises posterior risk $r'(\hat{\tau} | x^n)$ under squared error loss $L'(\tau, \hat{\tau})$. Where the weight $1 / \tau({\theta})$ has been absorbed into the original posterior $p(\theta | x^n)$ with renormalisation to form a new posterior $p'(\theta | x^n)$:

$$\hat{\tau}_B = \min_{\hat{\tau}} r'(\hat{\tau} | x^n) = \min_{\hat{\tau}} \int (\hat{\tau} - \tau)^2 \cdot p'(\theta | x^n) \space d\theta.$$

Using the result that the Bayes estimator under squared error loss is mean of the posterior $r'$, and the lazy statistician rule, we have that

$$\hat{\tau}_B = \mathbb{E}_{p'(\theta | x^n)}[\tau(\theta) | X^n = x^n].$$

Rewriting the right hand side in terms of our original posterior $p$, the solution is:

\begin{align*}\hat{\tau}_B &= \int \tau(\theta) \left( \frac{[1 / \tau(\theta)] \cdot p(\theta | x^n)}{\int [1 / \tau(\theta')] \cdot p(\theta' | x^n) \space d \theta'} \right) \space d \theta \\ &= \frac{\int p(\theta | x^n) \space d \theta}{\int [(1 / \tau(\theta')] \cdot p(\theta' | x^n) \space d \theta'} \\ &= \frac{1}{\int [(1 / \tau(\theta')] \cdot p(\theta' | x^n) \space d \theta'}. \end{align*}

Where in the 2nd line the normalisation constant has been factored out, and in the 3rd line it has been assumed that $p(\theta | x^n)$ has been appropriately normalised.

The above result is an instance of the more general solution (in this context) that

$$\hat{\tau}_B = \frac{\mathbb{E}_{p(\theta | x^n)}[w(\theta) \tau(\theta) | X^n = x^n]}{\mathbb{E}_{p(\theta | x^n)}[w(\theta) | X^n = x^n]}.$$

A statement of this can be found in Corollary 2.5.2. of The Bayesian Choice by Robert (2003).


Whilst you have in a sense raised some appropriate concerns about the validity of interchanging the order of differentiation and integration, also known in intermediate statistics reference textbooks as "differentiating under an integral sign", there are broader issues at stake when you "compute the derivative of the posterior risk with respect to an estimator".

On differentiating under the integral sign.

The technicality of differentiation under an integral sign amounts to inquiring, "under what conditions it is valid to interchange a limit and an integral?" This amounts to an appeal to Lebesgue's dominated convergence theorem. Without knowledge of formal tools in analysis and measure theory, this is difficult to treat, so the best one can hope to do in this situation is to supply some more easily verifiable conditions/corollaries of the theorem that should in principle allow one to assess if differentiating under the integral sign is appropriate.

The relevant reference you require is Section 2.4: Differentiating under an integral sign in Statistical Inference by Casella and Berger (2002):

1. In the case that range of definite integration $[a, b]$ is such that $a$ and $b$ are constants not depending on $\theta$, and $f(x, \theta)$ is differentiable with respect to $\theta$, then a special case of Leibniz rule means that

$$\frac{d}{d \theta} \int^b_a f(x, \theta) \space dx = \int^b_a \frac{\partial}{\partial \theta} f(x, \theta) \space dx.$$

2. In the case that the range of integration $[a(\theta), b(\theta)]$ depends on $\theta$ and all of $f(x, \theta)$, $a(\theta)$ and $b(\theta)$ are differentiable with respect to $\theta$, then Leibniz's integral rule states that

\begin{align*} \frac{d}{d \theta} \int^{b(\theta)}_{a(\theta)} f(x, \theta) dx = & \space f(b(\theta), \theta) \frac{d}{d \theta} b(\theta) - f(a(\theta), \theta) \frac{d}{d \theta} a(\theta) \\ & \space + \int^{b(\theta)}_{a(\theta)} \frac{\partial}{\partial \theta} f(x, \theta) \space dx. \end{align*}

3. In the case that the range of definite integration is not finite. Supposing that $f(x, \theta)$ is differentiable and there exists a function $g(x, \theta)$ such that

$$\left | \frac{\partial}{\partial \theta} f(x, \theta) \left. \right |_{\theta = \theta'} \right | \leq g(x, \theta)$$

for all $\theta'$ such that $\lvert \theta' - \theta \rvert \leq \delta_0$. If additionally

$$\int^{\infty}_{- \infty} g(x, \theta) \space dx < \infty$$

then

$$\frac{d}{d \theta} \int^{\infty}_{- \infty} f(x, \theta) \space dx = \int^{\infty}_{- \infty} \frac{\partial}{\partial \theta} f(x, \theta) dx.$$

That is, if $f$ is sufficiently 'smooth', in the sense that you can bound the variability in its partial derivative using a function $g(x, \theta)$ which has a finite integral, then the order of differentiation and integration may be interchanged.

On why these results may not be applicable to your situation in a straightforward way.

The arbitrary estimator $\hat{\tau} = \hat{\tau}(X_1, \dots, X_n)$ is a function of data. The posterior risk $r(\hat{\tau} | x^n)$ which can vary in choice of function $\hat{\tau}$, is a functional. In order to compute derivatives with respect to functions, you need to define what differentiation means in this context, and to establish its properties. Given this additional complication, it is also somewhat unclear how differentiating under an integral sign would work in the context of functional derivatives. Perhaps that is why an alternative approach was suggested.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.