# Are these independent: the sample, randomized rule, and random variable having the prior distribution on the parameter space?

In section 1.3 of Bickel and Doksum's Mathematical Statistics 2006, the risk function of a nonrandomized rule $d$ is the expectation of loss of the rule wrt the random sample. $$R(\theta, d) := E_{X\sim P_\theta} l(\theta, d(X))$$

1. the risk function of a randomized rule $\delta$ (distributed according to $\lambda$) is further taking expectation wrt the randomized rule. $$R(\theta, \delta) := E_{\delta \sim \lambda} E_{X\sim P_\theta} l(\theta, d(X))$$

Q: Are the randomized rule $\delta$ and the sample supposed to be independent? I would like to know if we can exchange the order of the two expectations as: $$R(\theta, \delta) = E_{X\sim P_\theta} E_{\delta \sim \lambda} l(\theta, \delta(X))$$

2. In section 3.3, they studied the relation between minimax rule and Bayes rule in terms of some two-player 0-1 game, where they introduced the nature and a statistician as the two players. The nature chooses a prior distribution $\pi$ over the parameter space $\Theta$, and the statistician chooses a randomized rule. The payoff one player pays to the other is

$$r(\pi, \delta) := E_{\theta \sim \pi} R(\theta, \delta) = E_{\theta \sim \pi} E_{\delta \sim \lambda} E_{X\sim P_\theta} l(\theta, \delta(X))$$

Q: Are $\theta$, the randomized rule $\delta$ and the sample supposed to be independent? I would like to know if we can exchange the order of the three expectations as:

$$r(\pi, \delta) = E_{\delta \sim \lambda} E_{\theta \sim \pi} E_{X\sim P_\theta} l(\theta, \delta(X))$$

3. The reason of my question in 2 is:

They said that given $\delta$, $\arg\max_\pi r(\pi, \delta)$, when exists, can be chosen to the point mass probability measure on $\arg\max_\theta R(\theta, \delta)$. It means that, given a randomized rule $\delta$, the least favorite prior distribution $\pi_\delta$ on $\Theta$, can chosen to be a single $\theta_\delta$, i.e. we can remove the randomization of $\theta$

Q: I would like to know if it is also similarly true that given $\pi$, the Bayes randomized rule $\arg\min_\delta r(\pi, \delta)$, when exists, can be chosen to have the point mass probability measure on $\arg\min_d E_{\theta \sim \pi} E_{X\sim P_\theta} l(\theta, d(X))$?

If yes, it implies a Bayes randomized rule is always a Bayes nonrandomized rule. But I am not sure if this conclusion is correct.

I think that this can be true, if we can exchange the order of $E_{\delta \sim \lambda}$ and $E_{\theta \sim \pi}$ in $r(\pi, \delta)$ as in part 2, which is why I ask if $\theta$ and $\delta$ are independent in part 2.

Thanks.

The relevant pages are here for section 1.3 and here for section 3.3.

• Thanks. @Glen_b. I have been trying to update my post for the past half an hour, but couldn't perhaps due to the site problem. – Tim Dec 1 '14 at 2:20
• Indeed, I started experiencing problems a few minutes after I typed that comment. Whenever you can. – Glen_b Dec 1 '14 at 2:38

1. Are the randomized rule δ and the sample supposed to be independent?

There is no reason to restrict randomised rules in this manner. For instance, randomisation may only occur for some values of the observation/sample.

1. Are θ, the randomized rule δ and the sample supposed to be independent?

In a joint Bayesian model, sample and parameter are dependent since the sampling distribution is the conditional distribution. While the randomised rule cannot depend on the unknown $\theta$ in a functional manner, since it can [functionally] depend on the sample, it possibly does [probabilistically] depend on the unknown $\theta$.

1. a Bayes randomized rule is always a Bayes nonrandomized rule

This is incorrect: A randomised Bayes rule may differ from a non-randomised one and achieve better performances from a risk perspective. For instance, in finite parameter spaces with strictly convex losses, a minimax rule is always a randomised Bayes rule but not necessarily a non-randomised Bayes rule

• On your answer to #3, could you point me in the direction to a proof of this? Thanks. – John Nov 6 '19 at 13:35
• This situation is covered in my book. – Xi'an Nov 7 '19 at 2:35